An ensemble is a claim about independence. You put k models in a room, they vote, and errors unique to each member cancel out. That only works if members actually disagree. If they all came from the same handful of prompts, voting is theatre.
Our platform has 3,784 model records and 46,807 graded calls over 16,787 fixtures. On paper that is a deep committee. I measured how deep with three estimators: stance concentration, pairwise contradiction, and Fleiss' kappa converted to effective ensemble size.
On the 2,722 fixtures with five or more voters, the average ballot held 10.34 models and 1.46 effective opinions. A majority vote over those ballots scored 56.95% — slightly worse than the 57.00% average of the members it aggregated.
The data
Everything below comes from one append-only table. The relevant columns:
| column | meaning |
|---|---|
match_id / model_id | fixture key, model record |
training_word_id | prompt lineage that model was built from |
prediction_direction | home, draw, away, home/draw, away/draw |
direction_arity | single (70.18%) or double (29.76%) |
confidence | 1–10 integer, not a probability |
hit / actual_result | settled once, home / draw / away |
Two properties matter. First, 29.76% of calls are two-outcome leans, so "agreement" is not string equality — two members can differ in label yet stay logically compatible. Second, training_word_id exposes the pedigree behind each ballot, where the collapse originates.
Estimator 1: how concentrated is a ballot?
If ten members split 9/1/0, the committee has one opinion and one dissenter. The Herfindahl index over stance shares gives a "number of effective stances" directly:
WITH per_match AS ( SELECT match_id, count(*) AS n, count(*) FILTER (WHERE prediction_direction='home') AS h, count(*) FILTER (WHERE prediction_direction='draw') AS d, count(*) FILTER (WHERE prediction_direction='away') AS a, count(*) FILTER (WHERE prediction_direction='home/draw') AS hd, count(*) FILTER (WHERE prediction_direction='away/draw') AS ad FROM ai_predictions WHERE hit IS NOT NULL GROUP BY 1 ), scored AS ( SELECT *, (h*h + d*d + a*a + hd*hd + ad*ad)::numeric / (n*n) AS hhi, greatest(h,d,a,hd,ad)::numeric / n AS top_share FROM per_match WHERE n >= 5 ) SELECT count(*) AS fixtures, round(avg(n),2) AS mean_ballot, round(avg(top_share),4) AS mean_top_share, round(avg(hhi),4) AS mean_hhi, round(avg(1.0/hhi),3) AS effective_stances, round(100.0*count(*) FILTER (WHERE top_share=1)/count(*),2) AS pct_unanimous, round(100.0*count(*) FILTER (WHERE h>0 AND a>0)/count(*),2) AS pct_home_and_away FROM scored; Read the last two columns together. A third of ballots are unanimous, and only 3.99% of fixtures have at least one member on each side of the home/away divide. The 358-call maximum ballot we once recorded is not a debate; it is a chorus. Mean top share is 0.8119 and mean HHI 0.7486, so 1/hhi = 1.455 effective stances against a 10.34-member roster.
Estimator 2: are disagreements real or cosmetic?
Label identity is too strict. home and home/draw are compatible; home and away are not. So classify each pair by outcome set, not string:
WITH ballots AS ( SELECT match_id, model_id, prediction_direction AS dir FROM ai_predictions WHERE hit IS NOT NULL AND match_id IN ( SELECT match_id FROM ai_predictions WHERE hit IS NOT NULL GROUP BY 1 HAVING count(*) BETWEEN 2 AND 40) ), pairs AS ( SELECT a.dir AS d1, b.dir AS d2 FROM ballots a JOIN ballots b ON a.match_id = b.match_id AND a.model_id < b.model_id ) SELECT count(*) AS pairs, round(100.0 * count(*) FILTER (WHERE d1 = d2) / count(*), 2) AS identical_pct, round(100.0 * count(*) FILTER (WHERE NOT ( (d1 IN ('home','home/draw') AND d2 IN ('home','home/draw')) OR (d1 IN ('draw','home/draw','away/draw') AND d2 IN ('draw','home/draw','away/draw')) OR (d1 IN ('away','away/draw') AND d2 IN ('away','away/draw')) )) / count(*), 2) AS contradictory_pct FROM pairs; Across 115,084 member pairs: 71.66% carry the identical label, 28.34% differ, and 24.93% are irreconcilable — one member says the home side wins, another says it does not. A quarter of all pairs are in direct logical conflict, yet ballot concentration is 0.81. Both are true: disagreement lives in the tail, and the tail almost never decides the vote.
Estimator 3: kappa to effective sample size
Concentration is a per-fixture view. Fleiss' kappa corrects observed agreement for what chance alone would produce given the label marginals — here home 19,072, away/draw 11,116, away 10,249, home/draw 6,010, draw 360, across 46,807 rows.
from math import comb marginals = {"home": 19072, "away/draw": 11116, "away": 10249, "home/draw": 6010, "draw": 360} total = sum(marginals.values()) p_exp = sum((v / total) ** 2 for v in marginals.values()) # 0.2869 p_obs = 0.7166 # identical-label rate kappa = (p_obs - p_exp) / (1 - p_exp) # 0.6026 n = 10.34 # mean ballot size n_eff = n / (1 + (n - 1) * kappa) # 1.56 print(round(p_exp, 4), round(kappa, 4), round(n_eff, 3)) # What a genuinely independent committee of 57% members would score p = 0.5700 for k in (1, 5, 11, 21): acc = sum(comb(k, i) * p**i * (1 - p)**(k - i) for i in range(k // 2 + 1, k + 1)) print(k, round(acc * 100, 2)) 0.2869 0.6026 1.56 1 57.00 5 62.95 11 68.34 21 74.30 Two independent estimators land in the same place: Herfindahl says 1.455 effective stances, kappa says 1.56 effective members. Eleven independent 57% members would score 68.34%. Ours score 56.95%. The theoretical gain is 11.4 points of accuracy; the realised gain is negative 0.05.
Where the collapse comes from: the pedigree column
The ballot average is 10.34 models but only 4.10 distinct training_word_id values — 0.439 lineages per model, and on 94.31% of multi-model fixtures there are strictly fewer lineages than models. Models are not independent individuals; they are re-parameterisations of a few prompts, so their errors are shared by construction. That suggests a fairer vote: one ballot per lineage, not per model. Take each lineage's modal stance, then vote across lineages.
WITH base AS ( SELECT match_id, max(actual_result) AS res FROM ai_predictions WHERE hit IS NOT NULL AND actual_result IS NOT NULL GROUP BY 1 HAVING count(*) >= 5 ), rows AS ( SELECT p.match_id, p.training_word_id AS tw, p.prediction_direction AS dir FROM ai_predictions p JOIN base b USING (match_id) ), lineage AS ( SELECT match_id, tw, mode() WITHIN GROUP (ORDER BY dir) AS stance FROM rows GROUP BY 1, 2 ), tallied AS ( SELECT match_id, count(*) AS lineages, count(*) FILTER (WHERE stance='home') AS h, count(*) FILTER (WHERE stance='draw') AS d, count(*) FILTER (WHERE stance='away') AS a, count(*) FILTER (WHERE stance='home/draw') AS hd, count(*) FILTER (WHERE stance='away/draw') AS ad FROM lineage GROUP BY 1 ), voted AS ( SELECT t.*, b.res, h + 0.5*hd AS hv, d + 0.5*hd + 0.5*ad AS dv, a + 0.5*ad AS av FROM tallied t JOIN base b USING (match_id) ), picked AS ( SELECT *, CASE WHEN hv > dv AND hv > av THEN 'home' WHEN dv > hv AND dv > av THEN 'draw' WHEN av > hv AND av > dv THEN 'away' END AS pick FROM voted ) SELECT count(*) AS fixtures, round(avg(lineages), 2) AS mean_lineages, round(100.0 * count(pick) / count(*), 2) AS pct_decisive, round(100.0 * count(*) FILTER (WHERE pick = res) / count(pick), 2) AS accuracy_on_decisive FROM picked; fixtures 2722 | mean_lineages 4.10 | pct_decisive 72.44 | accuracy_on_decisive 57.35 Model-level voting on the same fixtures: 78.18% decisive, 56.95% accurate. Lineage-level: 72.44% decisive, 57.35% accurate. Deduplicating the ballot buys 0.40 points and costs 5.74 points of decisiveness, because a smaller, more honest electorate ties more often.
Consensus tells you the fixture is easy, not that you are right
One more diagnostic, because "agreement predicts accuracy" is the assumption teams lean on hardest. Split fixtures by unanimity and look at the average accuracy of individual members in each group:
| ballot | share of fixtures | mean member accuracy |
|---|---|---|
| unanimous | 33.43% | 65.36% |
| contested (top share < 0.70) | 800 fixtures | 50.14% |
Unanimity does correlate with being right — but just as strongly with the fixture being easy. On the contested set, where a committee would add value, member accuracy is 50.14% and the disagreements do not resolve. The aggregate is brutal: committee 56.95% against members 57.00%. Aggregation over clones is not free; it is a small tax.
Practical rules I now apply
- Report effective size, not roster size.
1/hhior the kappa conversion belongs next to every "we have N models" claim. Ours reads 1.46 out of 10.34. - Deduplicate by lineage before voting. Weight each member
1 / members_in_its_lineage, otherwise you sample one prompt many times. - Track the contradiction rate, not just disagreement. 28.34% of pairs differ; 24.93% are strictly incompatible. The gap is the cosmetic part.
- Never read consensus as confidence. Log the top share with every aggregated call so you can split unanimity from easiness later.
- Publish the negative result. "Committee is 0.05 points worse than its members" is the sentence that stops someone shipping an ensemble as a feature.
Limits
Observational statistics on our own settled history, not a controlled experiment. Ballot composition is not random — high-profile fixtures attract more models, so the ≥5-model subset over-represents them. training_word_id approximates pedigree; two models on one word can diverge through temperature and version drift. Condorcet figures assume independence and are arithmetic ceilings, not forecasts; the correlated-vote simulation at kappa 0.6–0.81 reproduces only the flat 56.8–57.3% band we observe — evidence about mechanism, not proof. Every number is recomputable from ai_predictions with the queries above.
An ensemble's value is a property of its pedigree graph, not its member count — and the graph is measurable in about forty lines of SQL.
Analytics and research content. Not betting advice. 18+
本内容仅供数据分析与研究参考,不构成任何决策建议。
