Why Successful API Calls Can Still Produce Failed Workflows
Every call that ran returned 200 OK. The workflow still failed. Designing for the gaps between requests.
A customer requests a refund. The payment API returns 200 OK. Then the worker crashes before the confirmation email goes out and before the ledger is updated. Every call that ran succeeded. The workflow still failed.
An HTTP 200 only proves that one request crossed a network boundary and got a response. In systems with several steps, every individual call can succeed while the overall workflow quietly leaves data inconsistent or stuck.
Reliable software means designing for the gaps between calls, where timeouts, partial writes, duplicate deliveries, and restarts actually happen.
1. A request is not a workflow
Suppose one HTTP request ingests data, calls an external model, transforms the result, and writes it to storage. Tying all of that to a single synchronous request creates problems right away:
- If the external call times out, the caller can’t tell whether the side effect already happened.
- If the process restarts mid-flight, in-memory progress disappears with no record.
- If the client retries the whole request, it may repeat expensive work or write duplicate records.
The fix is to separate accepting the request from doing the work. Accept it, record it, and process it in the background, saving the workflow’s state at every step. An opaque chain of network calls becomes something you can inspect: a record of which step each job is in and what happened at each one.
Under the hood
In practice, the API durably records or queues the job first, then returns 202 Accepted with a job ID the client can check. Returning 202 before the job is saved means a crash can silently lose it.
A common pattern behind this is the transactional outbox: save a business change and a message to send in the same database transaction, then publish the message separately. That prevents losing a notification after a successful database write. External actions, like issuing a refund, still need idempotency and reconciliation, meaning checking the provider’s records against yours and fixing any mismatch, because they can’t be rolled back with your local database. For longer workflows, durable execution tools such as Temporal or AWS Step Functions save each step’s state for you.
2. Make every step safe to repeat
Every retry tests your data model. If a worker crashes after finishing step two of three, recovery depends on two guarantees:
- Recorded checkpoints. Each step saves its input, its status (pending, running, succeeded, failed), and a pointer to its output before the workflow moves on.
- Idempotent writes. Running a step again with the same idempotency key updates or skips existing records instead of creating duplicates. In other words, doing it twice has the same effect as doing it once.
Under the hood
Store idempotency keys with a unique constraint, and handle repeated requests by returning the existing result rather than running the operation again. Payment APIs such as Stripe support idempotency keys so that retrying a request doesn’t accidentally perform the same operation twice.
Many reliable queue configurations use at-least-once delivery, meaning a message can arrive more than once. Even systems advertising exactly-once processing have limits around external side effects, so design consumers to handle duplicates.
Retry only failures that might succeed later, like timeouts or rate limits, not permanent errors like invalid input. Use exponential backoff plus jitter, cap the attempts, and send persistently failing jobs to a dead-letter queue for investigation, where supported.
Some steps can’t be fixed by retrying, such as a payment captured for an order that then failed. Those need a compensating action, like an automatic refund. This is known as the saga pattern.
3. Monitor whether workflows finish, not just whether requests succeed
Request metrics like status codes and p95 latency can look healthy while a background queue stalls or failed jobs pile up. Watch the workflow itself:
- How many workflows started versus finished within the expected time?
- Where are retries concentrating, and are they recovering or running out?
- Can any failed run be replayed from its last good checkpoint?
Under the hood
Track workflow age, meaning how long the oldest job has been stuck in an unfinished state. Alert on dead-letter queue depth and on the share of workflows that finish within their expected window. Attach one correlation ID to every step, so a single workflow can be traced end to end across services.
A quick checklist
- Record each step’s state before moving to the next.
- Make every retry safe to repeat.
- Alert on workflows that don’t finish, not just requests that fail.
Designing for failure means treating saved state, safe retries, and recovery as part of the architecture from day one, not patches added after an incident.