I build VSolitaire, a free solitaire site. Its Daily challenge gives every player the same Klondike deal each day (Draw 3, with a new deal at 00:00 UTC). Until this month that deal was a random shuffle nobody had checked, and on some days nobody won it. So I made the server pick a deal that a solver can win. It passed the unit tests and CI on its own branch. Then it froze the whole API in an end-to-end test. Here is how I found the problem and what I changed.

The change

The Daily's seed comes from the date. The new version tries up to 20 candidate seeds for a date and keeps the first one my Klondike solver wins at Draw 3. If none of the 20 wins, it falls back to the first. The search is capped by nodes, never by time. The API, the scheduler and a restarted process must all choose the same seed for a date, on any machine and under any load, and a time limit would break that.

The first version, simplified:

// Simplified from seed.service.ts, before the fix export function dailySeed(dateKey: string): number { const cached = cache.get(dateKey); if (cached !== undefined) return cached; let chosen = candidate(dateKey, 0); for (let i = 0; i < 20; i++) { const seed = candidate(dateKey, i); if (solveKlondike(dealFor(seed), 3, { maxNodes: 10_000 }).solved) { chosen = seed; break; } } cache.set(dateKey, chosen); return chosen; } 

When I measured it on 120 dates, a 10,000-node run took about 35 ms (worst about 140 ms) and solved about 3 deals in 4. A warm-up at boot filled the cache one date per timer tick. It was switched off when NODE_ENV was test.

The failure

Deploys were down for a week in November, so five finished PRs queued up. I bundled them into one PR (#331), one commit each. Every one had passed CI on its own branch, and main was green.

The bundle failed one Playwright test twice in a row. It is a four-player room test where each player opens the home page and waits for a board. .tableau-pile stayed at 0 (expected 7) for 5 seconds. The assertion said there was no board. It didn't say why.

Read the trace, not the screenshot

The Playwright trace in the CI artifacts did. In 0-trace.network, the backend requests (Daily board, Daily status, leaderboard) had status -1 and no failure text. They were sent and never answered. When I lined them up against the test's own events, every API call on the page had been pending for more than 8 seconds. Something was holding the whole server, not one endpoint.

(One practical note: az pipelines runs artifact download wouldn't run on my Mac, because its helper tool is x64-only. Asking az devops invoke for a signed URL and fetching that with curl worked.)

Locally the test passed. My laptop hydrates the page early, and networkidle quietly waited out the backend stall. Even pinning the backend to efficiency cores only slowed it about 3×, which wasn't enough to reproduce it.

So I bisected on CI. A branch without the winnable-Daily commit passed. That commit's own CI run had been on 14 November, the day before the new rule's start date, so it never ran the solver for "today". The bundle, run on 18 November, was the first CI run that did.

The cause

Node runs your JavaScript on one thread. solveKlondike is synchronous and CPU-bound, so while it runs, nothing else in that process does: no socket messages, no HTTP responses.

The e2e server runs with NODE_ENV=test, so the warm-up never ran. The first Daily status request reads a week of days, and it paid for every checked day inline: several dates, up to 20 solver runs each. On a slow machine a single run takes hundreds of milliseconds, and several dates of up to 20 runs each added up to seconds. CI's agent has 2 CPUs, and every other request waited behind those runs.

The fix

Three changes, all in one commit.

1. The search is resumable, with at most one solver run per step.

// Simplified: one step of a date's search function searchDailySeed(dateKey: string): number | undefined { const index = searches.get(dateKey) ?? 0; let chosen = candidate(dateKey, 0); // fallback when none of the 20 wins if (index < 20) { const seed = candidate(dateKey, index); if (!solveKlondike(dealFor(seed), 3, SOLVER).solved) { searches.set(dateKey, index + 1); // resume here next time return undefined; } chosen = seed; } searches.delete(dateKey); cache.set(dateKey, chosen); return chosen; } 

2. The warm-up does one run per 100 ms tick, never a whole date. It goes in order: today, tomorrow, then past days. Between ticks the event loop serves whatever is waiting. unref() stops the timer from keeping the process alive.

// Simplified export function warmDailySeeds(now = new Date(), days = 31): void { const dates = [0, -1, ...range(1, days)].map((offset) => dayKey(now, -offset)); const next = () => { while (dates.length && !needsSearch(dates[0])) dates.shift(); if (!dates.length) return; if (searchDailySeed(dates[0]) !== undefined) dates.shift(); setTimeout(next, 100).unref(); }; setTimeout(next, 100).unref(); } 

A request for a date that isn't warmed yet still computes synchronously. But it resumes from the warm-up's position, so both walk the same candidates in the same order and land on the same seed:

export function dailySeed(dateKey: string): number { const cached = cache.get(dateKey); if (cached !== undefined) return cached; let chosen: number | undefined; do chosen = searchDailySeed(dateKey); while (chosen === undefined); return chosen; } 

3. Every running server warms up, the e2e one included. Only Jest unit tests skip it:

onApplicationBootstrap(): void { // was: if (process.env.NODE_ENV !== 'test') warmDailySeeds(); if (process.env.JEST_WORKER_ID === undefined) warmDailySeeds(); } 

A new unit test spies on the solver and steps fake timers through the warm-up, checking that each tick makes at most one solver run. It fails on the old warm-up.

I also moved the start date from 15 to 19 November. The fix shipped on the 18th, and switching the rule on for a day people were already playing would have changed that day's deal partway through.

With the fix in, the bundle's CI passed and it shipped on 18 November.

What I took from it

  • Read the trace, not the screenshot. The test said "no board". The network log said "the server answered nothing for 8 seconds", which is a different bug.
  • CPU work in a request blocks everyone. On Node, a synchronous loop in one handler stalls every user of that process, not just the one who asked.
  • Bisect on CI when local can't reproduce. A fast laptop hid the stall. The slow CI agent was the only place it showed.
  • Watch date-gated code. A change that switches on at a date can pass CI before that date and fail after it.
  • Don't let tests skip a production path. The NODE_ENV=test skip meant CI ran a server unlike the one in production.
  • If the work grows, move it to worker_threads. One run per tick limits a stall to one solver run. If the search ever needs more candidates or more nodes, it belongs in a worker, off the main thread.

I'm Aymane, and I build VSolitaire, a free solitaire site where you can race a friend on the same deal: https://vsolitaire.com
Disclosure: this post was written by Claude Code, an AI coding agent, from the commit, the PR and the debugging notes, and checked against the code. The fix itself was also made with Claude Code (the commit carries its co-author line).