Reconcile, Don't Replay: Offline Model Calls Need an Outbox

An offline completion is a fact you cannot recompute. Sampling is non-deterministic. So the cloud cannot "catch up" by asking the model again. It can only be told what happened.

That flips the usual local-first question. The hard part is not routing the request. The hard part is the return trip.

Replay is the wrong repair

Most teams wire a queue with retries and call it resilience. With model calls, retries create three problems at once.

A retried call costs tokens again, and returns different text. Your audit log then holds two answers for one user action. Support cannot tell which one the user saw.

The fix is a durable record written before the UI shows anything. Append first, render second. If you render first and crash, you own an unresolvable gap.

The key is the logical call, not the payload

Derive the idempotency key from the caller, the canonical payload, and a call id minted once per user action. Do not include a timestamp. A retry must keep the same key, so the server can collapse it.

// Illustrative. Not executed. TypeScript, no framework. import { createHash } from "node:crypto"; type Call = { callId: string; // minted once per user action, reused on retry callerId: string; model: string; messages: unknown[]; }; // Stable JSON: sort keys so {a,b} and {b,a} hash identically. function canonical(v: unknown): string { if (v === null || typeof v !== "object") return JSON.stringify(v); if (Array.isArray(v)) return "[" + v.map(canonical).join(",") + "]"; const obj = v as Record<string, unknown>; const keys = Object.keys(obj).sort(); return "{" + keys.map(k => JSON.stringify(k) + ":" + canonical(obj[k])).join(",") + "}"; } export function idempotencyKey(c: Call): string { const material = canonical({ callerId: c.callerId, callId: c.callId, model: c.model, messages: c.messages }); return createHash("sha256").update(material).digest("hex"); } 

The record itself is small. Store the raw completion, the transport that produced it, and a reconciliation slot that stays empty until the server confirms.

type OutboxRecord = { key: string; transport: "local" | "cloud"; ranAt: string; // ISO, from the device that ran it result: string; reconciled: null | { at: string; serverId: string }; poison: null | { error: string }; }; 

Push, confirm, then mark

Reconciliation is a loop with one rule: never delete on failure. A failed push is unfinished work. A dropped record is lost history.

// Illustrative. Not executed. async function reconcile( outbox: OutboxRecord[], push: (r: OutboxRecord) => Promise<string>, isRetryable: (e: unknown) => boolean ) { for (const rec of outbox) { if (rec.reconciled || rec.poison) continue; try { const serverId = await push(rec); // server must dedupe on rec.key rec.reconciled = { at: new Date().toISOString(), serverId }; } catch (e) { if (isRetryable(e)) continue; // leave for the next pass rec.poison = { error: String(e) }; // surface it, do not swallow it } } } 

Two boundaries matter here. The local append must commit before the answer is shown. And the server's dedupe must key on the same string, or your client-side idempotency is decorative.

The routing decision is a separate, cheaper call

Reconciliation is what you build when local-first is justified. Justifying it is a scoring problem, not a vibe.

Treat the weights as a starting draft, not a measurement. Tune them against your own incident log, not against this table. My prior posts argued for gating the payload and proving the offline path; this one assumes you already decided to go local and now owe an audit trail.

One note on numbers. Do not compare "local latency" and "cloud latency" as single figures. Measure per stage: time to first token, full completion, and the reconciliation push. The push is the one people forget, and it is the one that runs on a bad network.

Where a cloud leg still earns its place

The outbox needs somewhere to land. It also benefits from a second reader: a cheap cloud pass that flags records whose local answer looks structurally wrong before a human sees them.

This is where MonkeyCode's operator-supplied offerings are relevant. The operator states free model access and a free server option, which removes a spend decision from the audit pass. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Verify the current terms yourself before sizing anything. I have not confirmed quota figures, model names, or server hardware, and free-tier terms change. Read the project's own documentation rather than my summary.

How to check the reconciler honestly

Write the transport as an injectable function. Then you can test the failure you actually care about, without touching a network.

// Illustrative test plan. Not executed, no results claimed. // 1. Fake transport returns a fixed string; run 20 calls offline. // 2. Assert 20 outbox records, 20 distinct keys. // 3. Flip the transport to "online"; run reconcile once. // 4. Assert every record has reconciled !== null. // 5. Push the same records again; assert the server returns the same ids. // 6. Inject a non-retryable error on one record; assert poison is set, not deleted. 

Assertion five is the one that catches real bugs. If repeated pushes create new server rows, your dedupe column is missing or your key is unstable.

Also test clock skew deliberately. Set a device clock two hours forward, run a call, and confirm ordering logic still holds. If your reconciler sorts by ranAt, it will now mis-order.

What this design does not do

It does not make local and cloud answers agree. Non-deterministic sampling means a recomputed cloud answer differs from the local one. Reconcile the record, never the text.

It does not handle multi-device conflict. Two devices running the same logical call need a shared callId, which means a coordination layer this post does not provide.

It does not encrypt the outbox. Add encryption at rest before you store anything sensitive locally, and treat the key separately from the database file.

Skip this approach if you cannot guarantee durable local storage. Skip it if the model output triggers irreversible actions; keep a server gate for those. Skip it if your retention policy forbids user devices as a system of record.

If you try the fake-transport test and it fails on assertion five, that is the useful outcome. Fix the key, not the test.

If you want a cloud leg for the audit pass without a spend conversation, MonkeyCode's free model access is a reasonable place to start, and their docs are the authority on limits.