Tuesday, 10:12 UTC. Finance posts in the billing channel: the usage totals Stripe received for Sunday don't match the warehouse. Stripe is lower by 1.9%. Nobody changed anything on Sunday.
the scramble
Billing is mine, so I get tagged. Finance has the numbers: 18,420 metered API events exist in our warehouse for Sunday November 1 that never made it into a Stripe usage record. They belong to 212 customers. The total is small per customer, a few dollars each, and large enough in aggregate that finance won't let it ride.
First theory: Stripe dropped the events. We report usage through the usage-records API, and I have been burned by retries before. I pull our outbound request log for Sunday. Every call we made returned 200, and every call we made carries a quantity that matches what we meant to send. Stripe agrees with what we told it. The problem is upstream of the API.
Second theory: ingestion lost events. If events never reached the warehouse they couldn't be billed, but they are in the warehouse. 18,420 rows, all with correct customer IDs and timestamps. Kafka consumer lag on Sunday never rose above 4 seconds.
the hunt
The rollup is a Python worker. Every hour at :30 it sums the previous 60 minutes of events per customer and pushes one usage record each. I group the missing events by timestamp. They are not scattered. Every one of them falls between 04:30 and 05:30 UTC, a clean 60-minute block, which on the east coast is 00:30 to 01:30 EDT on the night daylight saving ended.
That narrows it to one missing run. I check whether the worker was down: pod restarts for the week are zero, and it logged heartbeats the whole night. Next I check whether it ran and failed: no exceptions, no retry entries. I grep for the rollup's own log line for that night:
$ grep "usage rollup" worker.log | grep "2026-11-01" 2026-11-01 00:30:00-04:00 usage rollup window=\[03:30Z, 04:30Z) customers=1311 2026-11-01 01:30:00-05:00 usage rollup window=\[05:30Z, 06:30Z) customers=1298 2026-11-01 02:30:00-05:00 usage rollup window=\[06:30Z, 07:30Z) customers=1275 The first line is the 00:30 EDT run, covering 03:30Z to 04:30Z. The second is the run labeled 01:30 with a -05:00 offset, covering 05:30Z to 06:30Z. Between those windows sits 04:30Z to 05:30Z. Nothing covers it. There was no 01:30 EDT run. The scheduler fired at 04:30Z, then next at 06:30Z. Two hours apart.
The worker uses APScheduler 3.9.1 with a cron trigger in the business timezone, because the original request was "run at half past every hour, Eastern, like the finance team thinks about it":
scheduler = BackgroundScheduler(timezone=pytz.timezone("America/New\_York")) scheduler.add\_job(rollup\_previous\_hour, CronTrigger.from\_crontab("30 \* \* \* \*")) I suspect the trigger, but a suspicion about a library isn't a root cause. I reproduce it with the pinned version, no worker involved, just asking the trigger for its next fire times across the boundary:
tz = pytz.timezone("America/New\_York") t = CronTrigger.from\_crontab("30 \* \* \* \*", timezone=tz) now = tz.localize(datetime(2026, 11, 1, 0, 0)) prev = None for \_ in range(3): nxt = t.get\_next\_fire\_time(prev, now) print(nxt, nxt.astimezone(pytz.utc).strftime("%H:%M UTC")) prev = now = nxt # 2026-11-01 00:30:00-04:00 04:30 UTC # 2026-11-01 01:30:00-05:00 06:30 UTC # 2026-11-01 02:30:00-05:00 07:30 UTC Same output as the production log. On the night clocks go back, the wall-clock time 01:30 happens twice, once in EDT and once in EST. The trigger resolves "01:30" to a single instant, the second one at 06:30 UTC, and never fires at the first one, 05:30 UTC. One hour of the schedule is gone.
the find
It took two mistakes to lose that hour. The first is the schedule: a cron expression evaluated in a timezone with DST has no guarantee of one fire per hour. In spring the same trigger has the opposite problem, a wall-clock time that never exists. The second is in our own job: each run aggregated "the 60 minutes ending now", so it had no memory of what it had already covered. When one run disappeared, the window it would have processed became a hole, and the next run happily started from its own fire time.
Nothing in the system could notice the hole. Every run that happened succeeded. A run that never happens throws no error, which is exactly why nothing paged.
the fix
Two changes. First, the scheduler runs in UTC. The business-facing label stays in the report, but the trigger doesn't depend on any civil calendar:
scheduler = BackgroundScheduler(timezone=pytz.utc) scheduler.add\_job(rollup\_since\_watermark, CronTrigger.from\_crontab("30 \* \* \* \*")) Second, the job tracks a watermark instead of assuming its own window. It reads the end of the last window it completed, aggregates from there to the current half-hour boundary, and advances the watermark in the same transaction as the usage rows it writes:
def rollup\_since\_watermark(now=None): now = (now or datetime.now(timezone.utc)).replace(minute=30, second=0, microsecond=0) with db.begin() as tx: start = tx.scalar( "SELECT window\_end FROM usage\_rollup\_watermark FOR UPDATE" ) if start >= now: return rows = tx.execute(AGGREGATE\_SQL, {"start": start, "end": now}) enqueue\_usage\_records(tx, rows, window=(start, now)) tx.execute( "UPDATE usage\_rollup\_watermark SET window\_end = :end", {"end": now} ) With this version a missed fire or a pod that was down for forty minutes resolves the same way: the next run covers everything since the last completed window. The Stripe usage records carry the window start as an idempotency key, so an overlapping retry can't double-report.
For the November 1 gap I ran the new function once by hand with the watermark set back to 04:30Z. It produced 212 usage records for 18,420 events and finance's reconciliation closed to zero difference. The customers got the usage on their next invoice, with a line explaining the delay.
the aftermath
Impact: 60 minutes of usage unbilled for 2 days, 212 customers, about 1.9% of that day's metered total. Nobody was overbilled. That is the only reason this stayed a reconciliation ticket instead of an incident review, and I don't love how much luck was in that.
What changed: the scheduler runs in UTC and local time only shows up in report labels. Each run covers the time since the last success. A new alert fires when the watermark is more than 90 minutes behind, which would have paged at 07:00 UTC that Sunday. And every scheduled job in the runbook now has one line answering "what happens if this run doesn't fire."
I also went through the other jobs on that scheduler. Two more were scheduled in Eastern time. One was a nightly cleanup, which doesn't care. The other was a daily 02:30 report, and in March it would have landed on a time that doesn't exist. It now runs in UTC too.

