I build WaistHeart, a small site that turns one portrait into a short vertical video of a heart pose (the TikTok "waist heart" and a few others). A customer pays, we call a video model, and we store the result. That sounds like a queue and a webhook. It took two rounds of code review before launch to get it right, and nearly every bug they found had the same outcome: the customer pays, credits are spent, and no video ever arrives.

Here is what we changed, in case you are building something similar on Workers + D1 + R2.

1. D1 has no interactive transactions, so every state change is a compare-and-swap

db.transaction() on D1 runs your statements one by one. A failure halfway leaves a paid order with nothing behind it. We moved every write that must succeed together into one batch(). Each batch starts with a guard statement that fails the whole batch unless the row still looks the way we read it. Reads come first, decisions are made in JS, and then one batch writes the new state and the credit change together.

The trap: a guard whose WHERE matches no rows passes. We had a "restore credits to the original grant" step guarded per grant row. If the grant row had been deleted, the guard passed and the job reported "credits returned" when nothing was returned. Anchor guards on a row that always exists, and put an EXISTS (...) check inside.

Also, D1 allows at most 100 bound parameters per statement. A guard built as one big OR over "every grant this order touched" will eventually fail with "too many SQL variables". Keep the parameter count per statement constant.

2. The provider callback is a hint, not the truth

The video API calls us back when a job finishes. We treat that callback only as "go check now": the body is ignored, and the status comes from a request signed with our API key. A one-minute Cron Trigger does the same check as a fallback, so a lost callback costs at most a minute.

3. Two workers will process the same job unless something stops them

The callback and the cron sweep can run at the same moment. Without protection, both submit, both poll, both download a 50 MB video, and one of the results is thrown away. Worse, a retried submit under the same idempotency key can trigger a second paid generation.

The fix is a short lease stored on the job row: leaseUntil plus a random leaseId, taken with a compare-and-swap before a submit or a download. Two details mattered:

  • Use a random lease id, not a timestamp. Two workers in the same millisecond wrote byte-identical rows, and both passed the CAS.
  • Compute the expiry from the real clock. Our sweep passed its start-of-run now down to every job, so a job handled four minutes into the run would get a one-minute lease.

Plain status polls don't take a lease. A duplicate GET is harmless, and it saves a write per poll.

4. Don't hold the video in memory twice

Our first version collected the response in chunks and then concatenated them into a second buffer: two copies of up to 64 MB each, inside a cron handler. Now, when a Content-Length is present, the body streams straight into the R2 PUT. On Workers that means a FixedLengthStream.

One more trap: if the CDN sends Content-Encoding: gzip, Content-Length is the compressed size and fetch hands you the decoded bytes. An exact-length check then fails on every retry. If the response has any encoding other than identity, treat its length as unknown.

5. "Out of credits" on the provider side is not the customer's failure

In the first version, an HTTP 402 from the provider (our own account out of credits) sent the job to a review state that nothing retried. Now it stays queued and retries every 15 minutes, and after 24 hours it goes to an admin review list. Under the first version, topping up the provider account wouldn't have brought those jobs back.

6. After payment, the photo must be immutable

The checkout form could be posted again with the same request key. It uploaded the new photo to the same storage key before the compare-and-swap noticed the order was already paid. A customer who paid in one tab and re-uploaded in another could have had the wrong photo sent to the model. Each upload now goes to a new versioned key. The record points at that key only inside the CAS, and a losing upload is deleted.

7. "Feature flag on, but config missing" must not fall back silently

In the first version, if live generation was switched on but the API key or storage signing was misconfigured, paid orders quietly took the old mock path and were marked done. Now that combination throws, logs loudly, and leaves the order unfulfilled. The sweep picks it up once the config is fixed.

What I'd do again

  • Write the failure scenario first ("customer paid, credits gone, no video"), then look for the code path that gets there.
  • Get an independent review that doesn't see the first round's conclusions. Our second reviewer found the photo-swap race the first fix had left open.

If you want to see what the pipeline produces, the waist heart pose guide shows frames from one of our sample videos. Questions about the D1 patterns are welcome in the comments.

Disclosure: I build WaistHeart.