The State Machine Saga: How to Survive a Job That Fails Halfway Through

Your batch job just moved real money, and then the downstream API timed out. ROLLBACK can't help you now.

By Ayush Pandey | Aug 6, 2026 | 8 min read

Software Architecture Microservices Distributed Systems Software Engineering

TL;DR

You can't wrap a distributed job in one ACID transaction, and some side effects (wires, emails, card charges) can't be rolled back at all.
The State Machine Saga pattern combines two ideas: a persisted state machine (where are we, what's next?) and compensating actions (an undo for every forward step).
You get restartability, graceful failure, and forced idempotency.
Use it whenever a job touches more than one system you can't roll back together, whether it runs for five hours or five seconds.

The 2 AM problem
Why "just wrap it in a transaction" fails
Two patterns, one job
What this buys you
Walking through the payroll example
A sketch in code
Tools you'd actually use
A note on scale
Where to go from here
The takeaway

The 2 AM Problem

It's 2 AM. Your nightly payroll batch job has been running for two hours. It has just pulled money out of a corporate bank account, and then the downstream banking API times out.

What happens next?

If your answer is "the whole thing rolls back," you're thinking in terms of a world that microservices left behind. A single ROLLBACK statement can't undo a wire transfer, un-send an email, or reverse a charge to a credit card.

You need something smarter. That something is the State Machine Saga pattern, and it has quietly become one of the most important architectural tools for anyone running long-lived, distributed batch jobs.

Why "Just Wrap It in a Transaction" Fails

Traditional batch processing loves the monolithic script: open a connection, start a transaction, chew through a few million rows, commit at the end. It's simple, and it works, right up until it doesn't.

Two things break this model in a modern architecture:

#	Problem	Why it hurts
1	You can't hold a lock across a distributed system for hours.	The moment your job touches more than one database or service, a single ACID transaction stops being an option.
2	Some actions can't be rolled back at all.	A `COMMIT`/`ROLLBACK` pair is meaningless once you've called an external payment gateway or fired off a confirmation email. It already happened in the real world.

So instead of pretending we can undo things at the database level, the State Machine Saga pattern accepts that failure is inevitable and designs for it explicitly.

Two Patterns, One Job

The model is really two well-known patterns working in tandem.

1. The State Machine: the orchestrator

Rather than a script running top to bottom, the batch job becomes a series of explicit, named states:

At any given moment, the job is in exactly one state. That state is persisted, written to a database and not just held in memory, so it survives a crash.

The state machine's only job is to answer two questions:

Where are we right now? What happens next, given success or failure?

2. The Saga: the safety net

Because the job's steps might hit different microservices or third-party APIs, there's no way to guarantee atomicity across all of them. The Saga pattern's rule is simple:

Every forward action needs a compensating action.

Forward action	Compensating action
Deduct funds	Refund funds
Reserve inventory	Release inventory
Send confirmation	Send correction or cancellation

Nothing moves forward without an undo button sitting right next to it.

When these two ideas merge, you get a job that knows exactly where it is and exactly how to unwind itself if things go wrong.

The diagram below shows every possible state, including the compensation and stuck branches.

What This Buys You

Checkpointing and restartability

Every state transition gets logged to a persistent store. If the server dies mid-TRANSFORMING, a fresh instance can read the last known state and pick up exactly there, with no re-processing three hours of work.

Graceful degradation

Instead of leaving your system full of half-written records, the Saga's compensating transactions clean up partial writes, issue refunds, and fire off failure notifications automatically.

Forced idempotency

This might be the most underrated benefit. Because a step might be retried after an ambiguous failure (was that timeout because the request never arrived, or because the response never came back?), every step has to be safe to run more than once. That discipline pays off everywhere, not just in this pattern.

An idempotency key lets a service recognize a retried request instead of double-processing it:

Walking Through the Payroll Example

Let's make this concrete with a nightly payroll job spread across three services:

Step	Service	Responsibility
`CALCULATE_PAY`	Service A	Works out taxes and net pay for every employee
`DEDUCT_FUNDS`	Service B	Pulls the total payroll amount from the corporate account
`DISTRIBUTE_FUNDS`	Service C	Deposits money into individual employee accounts

Steps 1 and 2 succeed. Then the banking API backing step 3 goes down.

Here's where the pattern earns its keep. The state machine doesn't just log an error and stop. It flips into a compensation flow, running the undo steps in reverse order:

Compensate step 2: refund the corporate account.
Compensate step 1: void the payroll calculation records.
End state: transition to FAILED_SAFELY and page the on-call engineer.

Nobody's money is stuck in limbo. Nothing is double-counted. The system lands in a known, consistent state. It just isn't the state you originally wanted.

The same shape applies to other domains. Here is an orchestrator dispatching commands to downstream services in a booking flow (seat blocking, payment, seat allocation) and tracking progress in a state store:

In a message-driven setup, the orchestrator talks to each participant through a broker's command and reply channels rather than calling it directly:

A Sketch of What This Looks Like in Code

You don't need a heavyweight framework to see the shape of this. A minimal version might look like:

class PayrollSaga:
    STATES = ["PENDING", "CALCULATE_PAY", "DEDUCT_FUNDS",
              "DISTRIBUTE_FUNDS", "COMPLETED", "FAILED_SAFELY"]

    def __init__(self, job_id, store):
        self.job_id = job_id
        self.store = store  # persistent state store (DB, Redis, etc.)
        self.state = store.load_state(job_id) or "PENDING"

    def advance(self):
        try:
            if self.state == "PENDING":
                self._transition("CALCULATE_PAY", calculate_pay)
            elif self.state == "CALCULATE_PAY":
                self._transition("DEDUCT_FUNDS", deduct_funds)
            elif self.state == "DEDUCT_FUNDS":
                self._transition("DISTRIBUTE_FUNDS", distribute_funds)
            elif self.state == "DISTRIBUTE_FUNDS":
                self._transition("COMPLETED", None)
        except StepFailed:
            self._compensate()

    def _transition(self, next_state, action):
        if action:
            action(self.job_id)  # must be idempotent
        self.state = next_state
        self.store.save_state(self.job_id, self.state)

    def _compensate(self):
        compensations = {
            "DISTRIBUTE_FUNDS": None,        # nothing sent yet, no-op
            "DEDUCT_FUNDS": refund_funds,
            "CALCULATE_PAY": void_pay_records,
        }
        for state in reversed(self.STATES):
            if state == self.state:
                comp = compensations.get(state)
                if comp:
                    comp(self.job_id)
        self.state = "FAILED_SAFELY"
        self.store.save_state(self.job_id, self.state)

The important part isn't the code itself, it's the shape: state is persisted after every transition, and every forward step has a named counterpart ready to run in reverse.

Where This Shows Up in Practice

You rarely build this entirely from scratch. A few tools have effectively become the standard toolkit:

Tool	What it is	Your role
AWS Step Functions	Fully managed state machine service. Define states and transitions declaratively (Amazon States Language).	AWS handles persistence, retries, and error branching.
Temporal	Durable execution engine. Workflows are ordinary code; the runtime checkpoints every step.	A crash mid-workflow just resumes where it left off.
Spring StateMachine	JVM library for modeling states and transitions inside a single application.	Pair it with your own Saga and compensation logic on top.

Here is a Step Functions workflow pausing mid-flow for manual approval. CloudWatch triggers a Lambda, which requests a task token from Step Functions and waits on a human response before resuming:

Each takes a different stance on how much you write versus how much the framework manages, but the underlying contract is the same:

explicit states + persisted transitions + compensating actions for every side effect a database can't roll back.

A Quick Note on Scale

Everything above used a nightly, three-hour payroll job as the running example, but don't let that fool you into thinking this pattern is only for long jobs. Duration isn't the trigger condition. The number of independent systems a job touches is.

A job that finishes in four or five seconds can still need this pattern if it fans out across multiple sources. Take a checkout flow that:

validates a coupon against a promotions service,
reserves stock in an inventory service, and
authorizes a charge against a payment gateway,

all in under a second. If the charge fails, you still need to release the stock reservation and invalidate the coupon hold, in reverse order, before you tell the customer anything.

The compensation logic is identical to the payroll example; it's just compressed into milliseconds instead of hours. The state machine still persists each transition (even if that persistence window is tiny), and each step still needs its compensating counterpart ready to fire.

The real question isn't "how long does this job run?" It's "how many systems does this job touch that I can't roll back together?" If the answer is more than one, the pattern applies, whether the job takes five seconds or five hours.

This is also why observability matters just as much for a five-second saga as a five-hour one. When a single logical operation is smeared across five services, you need a way to follow it end to end.

Distributed tracing carries a single trace context across every service a request touches, breaking the overall trace into spans. That gives you the visibility to debug a saga that failed three services deep:

Where to Go From Here

This post covered the shape of the pattern. A few directions worth exploring next:

Comparing orchestration engines

Beyond AWS Step Functions vs. Temporal, look at Camunda / Zeebe (BPMN-based, often used where business analysts co-own the workflow definition), Netflix Conductor, and Azure Durable Functions. Each makes different trade-offs around how much of the state machine is declarative config versus code, and how visibility and observability tooling is handled.

Designing compensating transactions

This is the part teams underestimate most. Worth studying:

How to handle a compensation that itself fails (compensation of a compensation)
How to make compensations idempotent, just like forward steps
Choosing between a hard undo (fully reverse the effect) and a semantic undo (e.g., issue a refund rather than pretending a charge never happened)

Choreography vs. orchestration

This post assumed a central orchestrator driving everything. The alternative is a choreographed saga, where each service listens for events and decides its own next action with no central coordinator. Orchestration gives you one place to see the whole flow; choreography avoids a single point of coordination but makes the overall flow harder to trace.

Saga logs vs. event sourcing

The state machine's persisted transition log looks a lot like an event-sourced audit trail. It's worth digging into how these two ideas overlap, and whether your state store should just be an event store.

Timeout and retry policy design

Idempotency gets you safe retries, but you still need to decide how long to wait before assuming a step failed, how many retries to attempt, and when to give up and trigger compensation instead of retrying again.

Testing a saga end to end

Because a saga's steps are spread across services and message channels, testing isn't as simple as calling a function and asserting on the return value. Study how teams build automated tests that produce a command, consume it on the other side, and assert the resulting event matches what was expected, effectively replaying the saga's message flow in a test harness.

The Takeaway

The State Machine Saga model is really an admission that distributed systems don't get rollbacks for free, so you have to build them in deliberately.

By breaking a batch job into named states and pairing every action with an explicit undo, you get a system that can crash at 2 AM, come back up, and either finish the job or unwind it cleanly.

No re-running three hours of work
No half-charged credit cards
No silent data corruption

Just a job that always knows where it is, and always knows how to fail safely.

Thanks for reading! If this helped, share it with the teammate who still says "just wrap it in a transaction."

The State Machine Saga: How to Survive a Job That Fails Halfway Through

The State Machine Saga: How to Survive a Job That Fails Halfway Through

TL;DR

Table of Contents

The 2 AM Problem

Why "Just Wrap It in a Transaction" Fails

Two Patterns, One Job

1. The State Machine: the orchestrator

2. The Saga: the safety net

What This Buys You

Checkpointing and restartability

Graceful degradation

Forced idempotency

Walking Through the Payroll Example

A Sketch of What This Looks Like in Code

Where This Shows Up in Practice

A Quick Note on Scale

Where to Go From Here

Comparing orchestration engines

Designing compensating transactions

Choreography vs. orchestration

Saga logs vs. event sourcing

Timeout and retry policy design

Testing a saga end to end

The Takeaway

More Articles