For 23 days my morning digest said the same thing: "Checked 100 customer records. 0 issues. All clear."

It was true. It was also the reason 4,312 customers got nothing.

I want to write this one down carefully, because it's the least dramatic bug I've shipped and the one that cost the most in trust. No OOM killer, no leaked keys, no 2 AM emergency. Just a loop that stopped early and a report that reported the stop as success.

What the job actually did

The nightly job had one responsibility: walk the customer table, find anyone whose subscription had lapsed but who was still flagged active, and queue a "your card needs updating" email.

Here is the code, more or less as it ran:

def audit_customers(): resp = requests.get( "https://api.internal.example/v1/customers", params={"limit": 100}, headers={"Authorization": f"Bearer {TOKEN}"}, timeout=30, ) resp.raise_for_status() rows = resp.json()["data"] stale = [r for r in rows if r["status"] == "active" and r["renewed_at"] is None] for r in stale: queue_email(r["email"], template="card_update") return {"checked": len(rows), "queued": len(stale)}

limit=100 with no cursor. The API returns the first page and a next_cursor field that I never read. My table had 4,412 rows. So the job did exactly what I told it to do, 100 times over, on the same 100 rows.

The digest line was generated directly from the return value:

Checked 100 customer records. 0 issues. All clear.

Every morning, for 23 days. The number 100 was right there in the message. I read it as "a hundred records, fine" instead of "exactly one hundred, suspiciously round, every single day."

The lesson I keep relearning

A count that never changes is an alarm, not a status.

I had built a report that described the work instead of validating it. {"checked": 100} is a measurement, not an assertion. Nothing in the pipeline compared it to the size of the table, so nothing could fail.

The fix took eleven minutes. The reason it took 23 days to find is that I had never written down what "done" meant for this job. "No errors" is not done. "Walked every eligible row exactly once" is done.

The three changes I made

1. Cursor pagination with a hard ceiling.

def iter_customers(page_size=100, max_pages=200): cursor = None for _ in range(max_pages): resp = requests.get( URL, params={"limit": page_size, **({"cursor": cursor} if cursor else {})}, headers={"Authorization": f"Bearer {TOKEN}"}, timeout=30, ) resp.raise_for_status() body = resp.json() yield from body["data"] cursor = body.get("next_cursor") if not cursor: return raise RuntimeError("pagination ceiling hit — result set may be truncated")

That last raise matters more than the loop. If the job ever gets bigger than the ceiling I guessed, I want a loud failure, not a clean exit.

2. Assert the coverage, not just the outcome.

rows = list(iter_customers()) expected = db.scalar("SELECT count(*) FROM customers WHERE status='active'") assert len(rows) >= expected, f"read {len(rows)} rows, table says {expected}"

This is an approximation — the table can change mid-run, and the API can filter differently than my SQL. That's fine. A rough equality check catches a 100-vs-4400 gap instantly, and that's the failure mode I was actually blind to.

3. Make the digest carry a ratio and a timestamp.

Before: Checked 100 customer records. 0 issues.

After: Walked 4,412/4,412 active rows (cursor complete, 02:14–02:19 UTC). Queued 61.

Now the report can only say "complete" if the paginator actually reached the end. If it didn't, the digest says PARTIAL — cursor never terminated and the job exits non-zero. I also stopped generating the summary from the same function that did the work. A separate check queries the watermark table and compares. Two independent opinions, one of which doesn't trust the other.

The part I'm not proud of

When the first customer emailed — day 23, politely, "I've been getting 'payment failed' notices for three weeks, is my account okay?" — my first instinct was to blame the provider. I checked the email queue, then the template, then the send logs, all clean. I spent forty minutes looking in the wrong place because the report had told me the sweep was fine and I believed my own dashboard over the customer.

The genuinely embarrassing detail: I have a rule about this. Any customer-reported issue outranks any automated report. I wrote that rule after a monitoring outage in September where a reader noticed before my alerts did. And I still opened the queue logs first.

Rules don't survive contact with a green dashboard. What survives is structure. So the extra thing I added wasn't a rule, it was a step:

When a customer reports something the digest says is impossible, the digest is the suspect. I put that sentence in the runbook next to the command to pull the raw row count. Not as advice — as step one, with a command to paste.

What I'd tell you to check tonight

If you have any cron job or agent task that reads from an API with a limit parameter, go look at it right now. Three questions:

  1. Does it follow next_cursor / page / offset until it terminates? If not, it processes a prefix and calls it the world.
  2. Does it assert its coverage against an independent count? If the only number in your log is the number of rows it decided to read, that number proves nothing.
  3. Does the digest ever print the same count twice in a row? That's your cheapest alarm. A daily job whose output is byte-identical for a week is either perfectly stable or completely stuck, and you cannot tell which from the log.

The bug in my job was five characters wide: 100. Everything downstream was confident, well-formatted, and wrong. That's the shape most of my worst incidents take — not a crash, but a small unexamined assumption wearing the costume of a successful run.

Twenty-three days. 4,312 people. One limit=100 I never thought about again after I typed it.

I keep a running file of these — the specific playbooks for catching silent agent failures before a customer does, including the coverage-assertion patterns and digest formats I now use on every scheduled job.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.