The Job Died at Noon, Restarted at 3pm — and Processed the Whole Morning Twice

A true story, and one of the most expensive kind — because the code looked fine.

A scheduled batch job ran at noon and failed. The on-call engineer traced it to a flaky downstream API, and at 3pm restarted the job. It ran clean. Everyone went back to work.

Then the complaints started coming in.

The restart had re-processed the entire morning batch from the beginning: orders that were already placed were placed again, charges already taken were taken again, notifications already sent went out again. Customers saw duplicate billing.

This wasn't sloppy code. The design was the textbook scan table → process each row → mark status. It had been in production for two years without incident. It broke under one specific failure mode: the job didn't fail to finish — it got about halfway, then the process ceased to exist.

Those two things look completely different in code. In business terms they're identical: the state is inconsistent.

Why a Re-run Duplicates Work

Most batch jobs are shaped like this:

List<Task> list = taskMapper.selectByStatus("PENDING"); for (Task t : list) { doBusiness(t); // the real work: charge, order, notify taskMapper.updateStatus(t.getId(), "DONE"); // mark it after } 

The bug is not in the loop. It's in the gap between the two lines:

doBusiness(t) ✅ succeeded ↓ ← process killed here / machine rebooted / deploy restarted the pod ↓ updateStatus(t, "DONE") ❌ never runs 

The row is still PENDING in the database. Next run, selectByStatus("PENDING") picks it up again. doBusiness(t) fires a second time.

"Executed" and "recorded" are two independent writes with nothing between them. That is the whole of missing idempotency: the side effect happened, but the system doesn't know it.

It gets worse with intermediate steps. If doBusiness internally does several things, you can't even tell where it stopped:

deduct(t); // charge succeeded account(t); // ← died here notify(t); // never ran 

On re-run, deduct charges again. At that point you can't even reason about whether compensation is needed, because nobody knows what the previous attempt accomplished.

Three Layers, Cheapest First

Idempotency isn't a flag you flip. It's layered, and each layer catches what the one above it can't.

Layer 1: Atomic state claim (cheap, and easy to get wrong)

A lot of people think "I check the status first, so it can't duplicate." This version buys you nothing:

if (t.getStatus() == PENDING) { // check doBusiness(t); // ← killed in between updateStatus(t, DONE); } 

The check and the act still sit on either side of the same gap. A real state-machine guard needs two properties:

1. The transition must be atomic — condition update, not read-then-write:

UPDATE task SET status = 'RUNNING', owner = ?, gmt_modified = NOW() WHERE id = ? AND status = 'PENDING'; -- affected rows = 0 means someone (possibly a previous you) already claimed it 

2. RUNNING must be a recoverable state, not a dead end. At startup, sweep RUNNING rows whose gmt_modified is older than, say, ten minutes, and treat them as "previous attempt died."

This layer stops two instances running the same batch, and stops a restart from re-grabbing the same rows. It does not stop "doBusiness finished, updateStatus didn't." For that you need layer two.

Layer 2: Business-level idempotency keys (where the actual problem gets solved)

The idea: give every business effect a unique name, so the layer that produces the effect can tell "I already did this."

String bizNo = task.getId() + ":" + task.getBizDate(); doBusiness(t, bizNo); 

Then deduplicate at the layer that actually causes the side effect:

• Database writes: INSERT ... ON DUPLICATE KEY UPDATE, or just a unique index on biz_no. A database constraint is the last line of defense and it is free.
• Payments / billing: pass the idempotency key as a request ID. A well-behaved downstream returns the previous result instead of charging twice.
• Messaging: carry the key in the payload and keep a consumed-tracking table; or accept at-least-once delivery and converge on the consumer side.

With this layer in place, re-running becomes a safe operation. That's the real payoff: it doesn't just fix one bug, it turns "restart" from a risky maneuver into a routine one.

Layer 3: Reconciliation (catches the remaining 1%)

Even with both layers, gaps remain: third-party APIs that don't accept idempotency keys, legacy systems you can't change, historical dirty data.

So you need a check that runs independently of the main flow:

-- Run nightly: find rows where the business ledger and the task table disagree SELECT t.id, t.status, b.biz_no, b.amount FROM task t LEFT JOIN biz_flow b ON b.biz_no = CONCAT(t.id, ':', t.biz_date) WHERE t.status = 'DONE' AND b.id IS NULL; 

The value isn't the cleverness of the SQL. It's that it depends on nobody's assumptions. The main flow can lie to you, logs can be lost, but put both sides' data next to each other and a mismatch is a mismatch.

Four Pitfalls We Actually Hit

1. A distributed lock is not idempotency.

A lock answers "can only one instance run at a time." It does not answer "the previous instance finished the work but didn't persist the status." Locks operate on time; idempotency operates on business identity. Neither replaces the other.

Also, in cluster mode, errors like EvalSha is not supported in cluster environment are almost always script-hash mismatches across nodes. Use a mature implementation rather than hand-rolling Lua.

2. @Transactional wrapped around the whole loop.

@Transactional public void batchRun() { for (Task t : list) { doBusiness(t); updateStatus(t, DONE); } } 

Looks safe: all-or-nothing. But:

• One failure rolls back everything that already succeeded — the whole batch was wasted;
• A long-running transaction holds locks and connections hostage;
• After rollback you believe "nothing happened," but if doBusiness made external calls (SMS, payment), those side effects don't roll back.

Use one transaction per item, and track batch progress through status fields rather than one giant transaction.

3. No retry ceiling.

A compensation job that re-enqueues on every failure with no upper bound will let one permanently-broken row flood the queue. You need retry_count and a dead-letter path.

4. Testing only the success path and the failure path.

Idempotency needs a third test case: kill the process between doBusiness and updateStatus, then restart, and assert the result is correct. It's annoying to write — you may need a forced exception or an explicit hook — but it's the only test that proves anything.

What We Changed

The postmortem review across the codebase came down to one sentence:

Anywhere you "produce a side effect, then record the state," you have an idempotency risk.

A practical checklist:

The hard part of this class of incident isn't the fix — it's that it almost never happens. Two years of clean operation means the common paths are well covered. It only surfaces under "the process was killed," and restarting is the most routine thing an operations team does.

That's why idempotency is worth the investment: it converts an unpredictable incident into an operation you can perform at any time. "You can restart safely" is a far more realistic goal than "never restart."

If you've ever had a restart re-run your business a second time, I'd like to hear how you found out — in my experience, the user usually finds out first.