The job that ran twice: leases, idempotency keys and the duplicate your schedule will eventually produce

The duplicate never announces itself. It arrives as a second confirmation email that looks like a courtesy, or a second charge that looks like a pricing bug, or a message to a stranger that the log says was sent exactly once.

I run a scheduled assistant: a few dozen actions that wake themselves at fixed times, look at something, and occasionally act outside the sandbox — send a message, submit a form, place an order, cancel a booking. The bug that cost me the most debugging time was not a model failure. It was one action firing twice, twenty minutes apart, on a schedule that ran once a day. Nothing in the code was wrong in the sense a test would catch. Every layer was doing its own normal thing.

If you are building anything on a timer, assume this will happen to you and design for it before it does. This is the design I ended up with: leases, idempotency keys derived from facts rather than runs, fencing tokens, and one extra state that most run logs are missing.

Duplicates are the default

At-least-once is not a property you choose. It is what you get from every piece you assemble:

• A scheduler that retries on a non-zero exit, without knowing whether the work completed before the process died.
• A worker restarted mid-run by a deploy, a crash, or an OOM kill.
• A run that takes longer than its interval, so the next firing starts while the first is still working.
• A timeout in your own client, followed by a retry, where the far side actually received the first request.
• Two copies of the scheduler running for a few minutes during a blue-green deploy.
• A human pressing "run now" while the timer is already due.
• The same action registered twice in a config file under slightly different names, which is the commonest one and the hardest to see.

Exactly-once delivery is a story you tell about logs after the fact. Exactly-once effect is the property you can actually build, and it needs three things: a claim that only one caller can hold, a key that identifies the effect rather than the attempt, and a store that refuses stale writes.

Separate the claim from the effect

The first mistake I made was treating "the run started" as the lock. It is not a lock, because a run that dies cannot release it and a run that is slow cannot defend it.

Make the lock a row, not a process:

CREATE TABLE job_lease ( job_key TEXT PRIMARY KEY, owner TEXT NOT NULL, fencing_token INTEGER NOT NULL, lease_until TEXT NOT NULL, -- UTC ISO created_at TEXT NOT NULL ); 

Claiming is a single statement, and it either inserts or steals an expired lease, atomically:

INSERT INTO job_lease (job_key, owner, fencing_token, lease_until, created_at) VALUES (?1, ?2, (SELECT coalesce(max(fencing_token), 0) + 1 FROM job_lease WHERE job_key = ?1), ?3, ?4) ON CONFLICT(job_key) DO UPDATE SET owner = excluded.owner, fencing_token = excluded.fencing_token, lease_until = excluded.lease_until WHERE job_lease.lease_until < excluded.created_at; 

Two things matter in that statement. The WHERE on the conflict path is the whole claim: a live lease makes the update a no-op, so a second caller gets zero rows changed and goes back to sleep. And fencing_token increments monotonically per job, which is how a stale run gets told it is stale later.

The lease duration has to satisfy two bounds at once: longer than the worst-case run, so you do not steal a lease from a run that is alive; shorter than the interval, so a crashed run does not block the next one. Those bounds conflict for long jobs, which is a signal that the job should be split, not that the lease should be a compromise.

The idempotency key comes from the fact, not the run

A run id is the wrong key. Two attempts of the same job at the same logical moment are the same effect restricted to one; two attempts at different moments are usually different effects. What distinguishes them is what the effect is about.

Derive the key from the thing being acted on, at a specific version:

def effect_key(action_id, target_id, input_digest): raw = f"{action_id}|{target_id}|{input_digest}" return hashlib.sha256(raw.encode()).hexdigest()[:32] 

input_digest is the part worth arguing about. It should cover every field that changes the consequence: the amount, the date, the recipient, the option selected, the body of the message. Not the whole page, and not the run's timestamp. If the underlying fact changed — the price moved, the slot opened — the key changes, and a second effect is correct, not a duplicate. If the fact did not change, a second attempt must collapse into the first.

Then the effect store enforces it:

CREATE TABLE effect_ledger ( effect_key TEXT PRIMARY KEY, job_key TEXT NOT NULL, run_id TEXT NOT NULL, state TEXT NOT NULL, -- claimed | submitted | done | unknown terms_shown TEXT, response TEXT, created_at TEXT NOT NULL, updated_at TEXT NOT NULL ); 

INSERT INTO effect_ledger (effect_key, state, ...) VALUES (?, 'claimed', ...) fails with a unique violation on the second attempt. That violation is a feature: it is how the second attempt learns what is going on. Catch it, read the row, and branch on state instead of guessing.

Fencing: the stale run has to be unable to write

A lease that expires does not stop the run that overran it. The classic failure: job A takes 40 minutes, lease is 10, job B steals at minute 11, both write. Now you have two effects and two logs that each claim to be the only one.

Fix it by making every write conditional on the token:

UPDATE job_lease SET lease_until = ?2 WHERE job_key = ?1 AND fencing_token = ?3; -- 0 rows => you were fenced 

and by carrying the token into the effect store:

INSERT INTO effect_ledger (effect_key, job_key, run_id, fencing_token, state, ...) SELECT ?1, ?2, ?3, ?4, 'submitted', ... WHERE ?4 = (SELECT max(fencing_token) FROM job_lease WHERE job_key = ?2); 

The overrunning run finds zero rows affected, marks itself fenced, and stops. It should not retry — it is no longer the owner of that job. This is the one place where "give up quietly" is correct, and it deserves its own log level so you can see how often it happens.

The state nobody logs: unknown

Most run logs have two outcomes, success and failure. There is a third, and it is the one that produces both duplicates and silent losses:

unknown is not failed. If a timeout happened after the request left your machine, the far side may have acted. Retrying an unknown effect without checking is how you get the second charge; treating it as failed is how you tell yourself nothing happened when it did.

The resolution step has to be specific to the effect: list the orders in the last hour and look for the matching one, read the message thread and see whether your line is there, read back the form's confirmation state. Write that function at the same time as the action, not after the first incident. If an effect has no way to be observed after the fact, that is a design defect, and the honest response is to make the effect observable before automating it.

Overlap: your p99 is the real interval

Measure run duration and compare its tail to the schedule. If a job's p99 is longer than its interval, you do not have a schedule; you have a queue with an uncontrolled arrival rate, and the duplicates are just the visible symptom.

Pick a policy explicitly, because the default is "whatever the scheduler does":

• Coalesce — a due firing while a run is live is dropped, not queued. Right for monitoring and status checks, where only the latest state matters.
• Bounded queue — right for work where each firing matters but the backlog must not grow without limit. Cap depth, alert on saturation.
• Fail loudly — right when a missed firing is a real miss, such as an end-of-day close. Silence here is the worst option.

The one thing not to do is let the platform decide by accident. Cron semantics vary on whether the next firing waits for the previous one, and the difference only shows up under load.

The only test that matters: kill it mid-effect

Unit tests cannot find this class of bug, because the bug is a process death at an inconvenient line. So make the death a test:

1. Start the effect with an injectable delay after the request is sent and before the ledger is updated.
2. Kill the process (SIGKILL, not a clean exit) at that point.
3. Restart the runner without clearing the ledger.
4. Assert: exactly one effect exists on the far side, and the ledger says unknown or done, never claimed forever.
5. Repeat a few hundred times with the kill point moved around.

The number to report is not "tests passed". It is duplicate effects observed (must be zero) and effects resolved from unknown (must be all of them, or each one explained). Run the same harness against a deploy restart, which is the same failure with a friendlier cause.

Approval gates and duplicates are the same bug

If a person approved something, that approval belongs to the effect key, not to the run. A retry that reuses an approval has spent it twice; a retry that asks again has spent the person's patience. Store the approval with the effect_key and mark it consumed on first use — the same unique constraint that stops the duplicate effect stops the duplicate approval. An approval that is still unused after a run is a fact worth seeing, not a detail.

Numbers worth keeping in the log

• Duplicate attempts suppressed — count of unique-violation branches taken. If this is always zero, your dedupe is untested; inject a duplicate on purpose in staging and watch it move.
• Fenced writes — a run that overran its lease and was rejected. Rising numbers mean the lease is too short.
• Claimed, never submitted — reserved and abandoned. A leak in the claim path.
• unknown age — how long an unresolved effect has been sitting. This is the real risk metric, because unknown is the only state that can hide a wrong outcome.
• Run duration p50/p99 versus interval — the overlap forecast.

Checklist

• A lease row per job, claimed in one atomic statement, with a monotonic fencing token.
• Lease longer than worst-case run, shorter than the interval; long jobs split.
• Idempotency key derived from action + target + digest of the consequential inputs, so a changed fact produces a new key and an unchanged fact collapses.
• The far side dedupes, or the effect is observable after the fact; prefer both.
• Every write carries the fencing token and is rejected when stale.
• Four states in the effect ledger, with unknown resolved before any retry.
• An explicit overlap policy, chosen rather than inherited.
• A kill-mid-effect test, run repeatedly, plus the same against a deploy.
• Approvals stored against the effect key and consumed once.
• Duplicate-suppressed and fenced counters on a dashboard, not in a log file nobody opens.

None of this makes a scheduled agent run exactly once. It makes a second run unable to do anything new and unable to pretend it did — which is the only version of "once" that survives a reboot.

If you want the concrete version of this on a phone-side assistant — what it does at a CAPTCHA, how a daily usage limit behaves, what a reset ticket actually restores — I keep two public notes:

• The English guide: https://pickle-invite-code-en.deed.page/ — the app's actual behaviour, including the 4 AM reset and what a reset ticket does.
• The invite link: https://pickle.com/i/RDKXK77G9ZRT

Disclosure: I use Pickle, a phone-side AI assistant app, and I am not its vendor. The links above are mine; if a sign-up completes through the invite link, both sides receive reset tickets, and the app's own notices govern any benefit. This is not a paid placement. The lease and ledger design here is the one I run in my own scheduled agent work, written from duplicates I caused myself.