Day 3 of MLH Global Hack Week: Builders Week. I loaded every benchmark run from Build Buddy into a Tiger Data hypertable to see what the numbers look like over time.
The setup
- A PostgreSQL table for each result: run time, model, sample, error code, build passed, fixed, seconds
- Converted to a hypertable with
create_hypertable('build_results', by_range('run_at', INTERVAL '1 day')) - 24 real rows from 4 runs, which landed in 2 daily chunks
- Queried with
time_bucketfor fix rate and speed
The SQL is in the repo: db/tiger-hypertable.sql
The "hardest" error
| Error | Runs | Avg s | Fastest | Slowest |
|---|---|---|---|---|
| CS0029 | 4 | 15.0 | 7.6 | 23.1 |
| CS0161 | 4 | 11.6 | 10.4 | 12.9 |
| CS0246 | 4 | 8.3 | 6.9 | 9.6 |
| CS0103 | 4 | 6.9 | 6.1 | 7.8 |
| CS1061 | 4 | 6.8 | 5.2 | 8.3 |
| CS1002 | 4 | 5.5 | 4.8 | 6.1 |
CS0029 looked like the hardest error to fix. It isn't. It's the first sample in every run, alphabetically, so it pays for loading the model into memory:
- Cold runs: 21.7 s and 23.1 s
- Warm runs: 7.7 s and 7.6 s
The same thing shows up per run. Tonight's first run averaged 11.2 s per error; the next two, with the model still warm, averaged about 7 s.
One run would never have shown this. Comparing runs over time did.
Builds aren't the whole story (again)
Tonight's first run fixed 6/6 by my current definition: the patched project builds. But reading the answers:
- the CS0161 fix added two extra
Console.WriteLinecalls, so the output changed - the CS1002 explanation said the semicolon "wasn't required," the opposite of the real error
- the CS1061 explanation invented an extension method in
System.Globalization
Next
- Record cold start separately, so timing compares like with like. With the cold starts removed, CS0161 is the slowest error even when the model is warm: 10.4 to 12.9 s on every run.
- Behaviour checks: a fix only counts if it builds and produces the expected output.
