The run that counted: faster than third place on two tracks, and still fourth

On November 3, Speed Demon 2026, run by Hackathon Raptors, held its live final: the frozen code of 20 finalists, five runs per track on a dataset nobody had seen, scored live as each run landed. I finished fourth on the machine score with 84.013. Third place had 84.681.

My programs were faster than the third-place team's on two of the three tracks that counted. This post is about why that wasn't enough, and about the noise I had explained away in writing the day before.

Every number here is public: the final's run log, and my repo, which has the code and a measured entry for every step: github.com/Nikhils-G/speeddemon.

The contest in one paragraph

Each track is a data-processing program, scored on one machine: an Intel i5-13500, with 12 logical CPUs for your container. My entry has four tracks, all in Rust. These are the three I tuned:

• A reads 3.1 million log lines (239 MB) and writes latency percentiles per hour and endpoint.
• B answers 50,000 boolean and phrase queries over 250,000 documents (254 MB).
• C joins 3.32 million rows of deliberately hostile CSV (escaped quotes, line feeds inside fields, mixed line endings) against a region table.

I stopped tuning the fourth, D, on October 31; it scored 63.81 in the final, and the overall counts only a team's best three tracks.

Each track scores 50 · log10(K) points, where

K = (T_ref/T)^0.5 · min(M_ref/M, 50)^0.2 · min(S_ref/S, 1)^0.15 · (dependency term) 

T is wall time, M is the container's peak memory (cgroup memory.peak) and S is lines of code. The overall score is the geometric mean of the best three tracks. A practice board ran each team's latest push on the real machine. The final ran each program five times, each in a fresh container, and took the mean of the middle three times and the highest of the five memory peaks. Judges add 35% later.

Final night

This is what the final logged for me:

Three tracks, three stories:

• A had one slow run. The middle-three rule dropped it, and A scored 87.27, its best practice score to two decimals.
• B had two slow runs. The rule drops one. The other counted: 70.4 ms instead of 65.8, about 0.74 points. (LippInc, first on the machine score, had a slow B run too. It was their only one, and the rule dropped it.)
• C ran five nearly identical times, but its first two containers peaked at 10.2 and 9.2 MB, against 8.2 to 8.4 for the rest. Memory is the worst of five, so 10.2 counted: about 0.86 points.

With both clean, I'd have had about 84.56, still fourth by 0.13. So bad luck isn't the whole story, and the rest of the story is more useful anyway.

The noise I explained away

On the practice board, my B never ran the same way twice. The 52 B runs in my log for the last practice day ranged from 65.6 to 147.4 ms. I had a good explanation: other teams' Docker builds ran on the same chip, sharing its L3 cache and memory. And I had a rule: if another track's unchanged code was slow in the same dispatch, the dispatch was disturbed and the run didn't count. My replica, a machine with the same CPU where nothing else ran while I measured, ran B 60 times back to back with at most one slow run.

So my Track B notes, written before the freeze, set aside eleven runs of 98 ms and more as disturbed and sorted the other 51 into two groups: 38 runs between 66 and 83 ms, and a slow group of 13 between 86 and 95 ms. They concluded:

The slow group should not occur in the final: final_scoring.py scores one team at a time …

And about one busy CPU:

The final's middle-3 rule absorbs one such run in five, not two.

The slow group didn't occur. My first two B runs in the final took 80.3 and 79.7 ms, inside the band my notes called the fast group: I had been treating a band 26% wide as normal. The rule absorbed one of those runs, not two. The two teams just above me ran B within 1.4% and 3.2% across their five runs. Mine spread 24%.

The public feed also holds a complete earlier run from the same afternoon, on the same hidden dataset; the results page shows the live one, and I was fourth in both. In the earlier run B had one slow run, 130.1 ms, about what my notes measured for a busy CPU, and the rule dropped it. So on final day, with one team on the machine at a time, B was slow in 3 of its 10 runs.

I don't know what slowed them, and the feed's logs don't say. What I do know is that B's time depends on its slowest CPU, and that I had measured what one busy CPU does to it. B splits each parallel phase into fixed slices, one per logical CPU, and with a busy loop on one of the 12 CPUs it took 124.5 to 144.3 ms instead of 65.5. I had even built a partial fix: threads that finish early claim the last eighth of the scan in chunks. On a quiet machine it cost 0.62 ms and 65 lines. With a busy CPU it barely helped (136 ms against 138 to 140), because every other parallel phase is split statically too and waits for that CPU. A real fix had to cover every phase. I bet that the final would be quiet, and didn't build it.

The middle-three rule forgives exactly one slow run. Treating runs as independent, if each one is slow with probability p, two or more of five are slow with probability 1 − (1 − p)^5 − 5p(1 − p)^4: 8% at p = 0.1, 26% at 0.2, 47% at 0.3. At 3 in 10, B's final was close to a coin flip.

Track A tells the other half. Earlier in the contest I had chased a tail there: two A changes measured about 1% faster on the replica and then ran slow in 8 of 12 board runs, the likely cause a 13th thread on 12 CPUs. I took that thread away, and on the replica the runs more than 8% over the median fell from 21–43 to 6–10 out of 300, while the median moved 0.9%. My A notes put it this way:

The replica does show such effects, but only if you count the slow runs instead of reading the median.

Even so, on the last practice day A's final code ran slow in seven of its nine board runs, three times in dispatches where B and C ran at their usual speed, and I filed those away too. In the live final it ran 14.7 to 15.3 ms in four runs of five. So the practice board really was noisier than the final, and my rule was right about that much. Where it failed me was B, the program I had measured doubling its time when one CPU is busy.

C's memory taught the same lesson on another axis. On the practice board, the runs my log caught of my last C code peaked at 9.38, 9.98, 10.67, 8.00 and 8.14 MB, and at 11.0 MB after the final four-line trim. The 8.00 MB run, 84.60 points, was the number I had been looking at. The final takes the worst of five, and in both of the day's full runs C's worst was its first container: 11.0 MB in the earlier run and 10.2 MB in the live one, against 7.9 to 9.2 for the rest. A and B peaked within 0.5 MB across their five containers both times. My replica never came close: over 20 runs of that code, its highest C peak was 7.91 MB.

The final isn't your best run. It's your distribution. I had optimised the best run.

Faster than third place, and still behind

The second lesson is arithmetic. The exponents in K set an exchange rate between the axes. A 1% improvement is worth:

So 1% of time is worth 3.3% of lines, and a program 60% longer than a rival's has to be about 15% faster just to tie.

Against ALL STAR, who finished third, this is where the points went (positive means I was ahead):

Both B programs stay under 44 MB, where memory scores in full, so memory cost neither of us anything there.

Across the three tracks I won time by 1.03 points, and lost lines by 2.61 and memory by 0.43. These are track points; the overall moves by about a third of their sum, which is the 0.67 gap. On C my program is 15.8% faster and 60% longer, and the two scores differ by 0.01.

I knew the rate. A line cost about 0.0074 points on A and 0.0059 on C, and on the last day I was cutting lines: ten here, four there, a few hundredths each. But the gap to third was 99 lines on A and 206 on C. That isn't a trim. It's a different design, written shorter from the start, and you don't get there by deleting cold lines on the evening of the freeze.

The replica that lied (once)

My later decisions came from that replica: the same CPU model, the organisers' own timer, and every change measured as paired, interleaved runs against the code it would replace, up to 201 pairs at a time. An A/A control, a binary against itself, read −0.24% over 201 pairs, so differences under about 0.3% counted as nothing. PGO added noise of its own: two builds of the same source ran 0.42 ms apart on B, so small comparisons used two builds on each side.

It still lied once, loudly. In C, I aligned a hash table's key slots to 32 bytes so that no 32-byte load would straddle two cache lines (perf counted 1.5M split loads a run). The replica said −0.41%, winning 146 of 201 pairs. The real machine ran it about 20% slower: 20.3 to 21.4 ms, against 17.2 ms for the code before it. That code, put back in the next dispatch, ran 17.2 ms with Tracks A and B both slow beside it, so it wasn't the machine having a bad minute. I then shipped the other change from that batch on its own: four summed columns added with one 256-bit add, where LLVM had emitted one 128-bit add and two scalar ones (+2 lines). That one measured −1.05% and −1.19% on the replica and gave C its best practice runs.

From my notes: "The same CPU did not settle a change that rests on where data falls against cache lines." Same model number, different machine, and I never found out which difference mattered.

What actually paid

A few of the wins, with their sizes:

Deleting the index (B). A search task looks like it wants an inverted index. But all 50,000 queries are known before you read a single document, and the index cost 85 MB or more, while memory only scores in full under 44 MB. So step 6 read the queries first and scanned the corpus once, in parallel, in 64-document blocks. Each query term that appeared in a block got a 64-bit word, one bit per document, and the boolean queries ran as word operations; counts and ids came out of popcounts and trailing-zero counts. 0.355 s at 203 MB became 0.267 s at 38 MB. Later steps changed how each block is evaluated, but never brought the index back. The best data structure was none.

The laptop that hid a quarter of the runtime (C). movemask(lo) | movemask(hi) << 32 compiled well on my development laptop, which has AVX-512. On CPUs without it, which include the scoring machine, LLVM rebuilt that 64-bit mask one bit at a time with vpinsrb: a quarter of the run time. I only found it by reading the disassembly on a test server without AVX-512. An inline-asm vpmovmskb fixed it: 60 to 50.6 ms on that server, and on the scoring machine, together with the step before it, 97.9 to 55.3 ms. Benchmark on the target, and read the assembly there.

Two lines that skip std's start-up (A). #![no_main] lets the C runtime call my own main, which skips Rust's start-up: a stack-guard lookup that reads /proc/self/maps, the SIGSEGV and SIGBUS handlers, and an alternate signal stack in every spawned thread. I had estimated −0.4%. It measured −1.08%, about 175 µs. (Panics still print their message and abort; nothing else in the program needed the rest.) When the whole program runs in 15 ms, the runtime's fixed costs are worth reading.

A profile has to come from the run it optimises (B). PGO trained on the small sample dataset made B 0.55 to 0.67 ms slower than no PGO at all. Trained at full scale, it was 0.41 and 0.38 ms faster than no PGO.

Knowing what the memory number counts (A, C). memory.peak includes the kernel's per-CPU charge stock: memory is charged to the container in batches of 64 pages per CPU, and on A that stock was 2.3 to 3.1 MB of the peak. In C, only four of the twelve threads fault memory in, in whole batches, so only four CPUs hold a stock: 10.19 MB became 8.17 MB. Memory-mapped input, on the other hand, cost nothing: the harness pre-warms the file, and its page cache is charged to the host.

What I'd do differently

1. Rehearse the exact final protocol. Five fresh containers in a row on a quiet machine, then read the middle three and the worst memory peak. Never the best run.
2. Treat repeated noise as a hypothesis. If the same "disturbance" keeps showing up on one track, it's information about that track.
3. Price lines at design time. At 3.3% of lines per 1% of time, a long program needs a big speed lead to pay for itself, and the design decides that, not the trimming.
4. Optimise the tail. The middle-three rule forgives one bad run. It doesn't forgive two, and nothing forgives a memory spike.

Where it ended

Fourth on the machine score, 0.67 behind third. The judges' 35% (code quality, technique and each track's PERF.md) lands on November 13, so the final order can still move. Whatever it ends up as, five runs on November 3 taught me more than the hundreds of practice runs before them.

Thanks to Hackathon Raptors for a contest that publishes every run, including the ones that hurt. The code, the per-step measurements and the run IDs behind them are in the repo: github.com/Nikhils-G/speeddemon.