A system shouldn't just know how to act. It should know when to wait, when to retry, and when to stop.
I've been building Senda, an early prototype for outbound call operations. It processes call events, tracks outcomes, evaluates proposed changes, and requires human approval before applying them.
One problem I spent considerable time on was ambiguous failures.
A timeout doesn't mean a write failed. It may have succeeded. Retrying blindly can create duplicate actions.
Senda handles this by:
Recording the intent in PostgreSQL before making a remote write.
Checking the remote state when the result is ambiguous, rather than retrying immediately.
Stopping for human review when the outcome cannot be established.
Built with Python and PostgreSQL. 306 tests, using a local simulator.
It's still early, with plenty left to build.
How do you handle ambiguous failures in your systems? I'd especially appreciate feedback on the recovery design.
