Breaking 0.669 to Reach Top 5 in Kaggle's "Name That LLM"

How we uncovered the public "decoy camouflage", introduced a high-diversity DeBERTa-v3 sequence head, and recalibrated Sinkhorn optimal transport to push beyond the 0.672 plateau.

1. The Challenge: Who Wrote That Passage?

In Kaggle’s Name That LLM competition, participants are tasked with attributing short text passages to one of 18 generative language models spanning six distinct model families:

• Anthropic: Claude Fable 5, Opus 4.6, Opus 4.8, Opus 5, Sonnet 5
• Google: Gemini 3.1 Pro Preview, 3.6 Flash, 3.7 Flash, 3.8 Flash
• Zhipu AI: GLM-5.3, GLM-5.3 Flash
• OpenAI: GPT-5.5, GPT-5.6 Luna, GPT-5.6 Sol, GPT-5.6 Terra, GPT-6 Astra
• Moonshot AI: Kimi K3
• Alibaba: Qwen 3.8 Max

The metric is Balanced Accuracy Score (macro-average recall per class). The key trap: the test set features unseen prompts, meaning standard NLP classifiers that memorize topic semantics overfit catastrophically.

flowchart LR A[Unseen Test Passage] --> B[4-View LinearSVC\nTrain + Pool] A --> C[Multi-Backbone LoRA24\nQwen14B / Qwen8B / Llama8B] A --> D[DeBERTa-v3 Sequence Head\n18-Class Classifier] B & C & D --> E[Logit Blend] E --> F[ATLAS Likelihood Ratio\nContested Rows Only] F --> G[Soft Sinkhorn Balancing\nalpha=0.30, rho=0.35] G --> H[Deterministic Typography\nHard Guardrails] H --> I[Final Parquet Submission]

2. Deconstructing the 0.669 Plateau

After deploying early iterations of the public ensemble, our model hit a wall at 0.669 Balanced Accuracy. Inspecting recent public lineage code (from ozertuu, Alex Chilton, mercurydust, and synthreaper) revealed crucial secrets:

A. The "Decoy Camouflage" Parameters

Top public notebooks were deliberately seeded with slightly de-tuned hyperparameters to prevent forks from overtaking the original authors:

• OURS_SVC_W = 0.85 (Sub-optimal; true optimum is 1.2)
• SVC_W = 12.0 (Caused excessive n-gram noise; optimal is 10.0)
• ATLAS_MARGIN = 0.36 and ATLAS_W = 0.22 (Over-corrected rows where the ensemble was already confident)
• SINKHORN = 0.42 with SINKHORN_TARGET_MIX = 0.50 (Violently forced predictions to the training prior)

B. The Hidden Test Distribution (The Small Cluster Trap)

While the training set contains ~150 prompt groups with 18 passages each, the hidden test set contains only 1 to 3 passages per prompt. Offline cross-validation that evaluated combinatorial Hungarian bipartite matching across full 18-passage groups inflated CV by +8.5%, but tanked the public Leaderboard by -0.015.

Evaluating out-of-fold on randomly sampled $k=2$ passages per prompt group yielded an $r = 0.91$ Pearson correlation with the live Kaggle Leaderboard!

3. The 4 Upgrades Driving the Top 5 Solution

1. High-Diversity Sequence Classifier (deb_v3_cls)

Most ensembles rely purely on causal autoregressive LLMs (Qwen3 and Llama-3.1). We mounted nuridudak/ntl-two-stage-encoders and integrated a fine-tuned DeBERTa-v3 sequence classification head with 18 output logits (w: 2.2):

if spec.get('type') == 'hf_cls': _ctok = AutoTokenizer.from_pretrained(spec['path']) _cls = AutoModelForSequenceClassification.from_pretrained(spec['path'], dtype=torch.float32).to('cuda:0').eval() # Batch-padded inference with label alignment  _zz = _cls_logits(test.text.values) LP[spec['name']] = log_softmax(_zz) * spec.get('w', 1.0) 

2. De-camouflaged & Damped Sinkhorn Balancing

We shifted from aggressive balancing ($\alpha=0.42$) to damped soft balancing ($\alpha=0.30$, $\rho=0.35$):
$$\mathbf{t} = (1 - \rho)\mathbf{p}{\text{pred}} + \rho \mathbf{p}{\text{prior}}$$
This prevents model collapse across rare classes without distorting high-confidence author attribution.

3. Precision-Targeted ATLAS Likelihood Verification

ATLAS (Ko et al.) evaluates source-conditioned token likelihoods. We restricted ATLAS evaluation strictly to rows with contested top-2 margins ($\Delta < 0.32$) at weight $0.20$, preserving confident predictions while breaking deadlocks on subtle sibling pairs.

4. Deterministic Typography Invariant Guardrails

While Bayesian evidence softens markers, certain tokenization and normalization invariants are mathematically impossible for specific providers:

• Curly Apostrophe (’): NEVER produced by Claude or GLM ($P = 0$).
• Curly Double Quotes (“): Strictly absent from Claude and GLM.
• Spaced Em-Dash (—): Produced only by Claude Opus 4.6, Opus 5, GLM-5.3, and GLM-5.3 Flash.
• Standard Em-Dash (—): Absent from Gemini 3.1 Pro, 3.6 Flash, and 3.8 Flash.

Directly applying hard masking ($-\infty$ penalty) on final decision logits ensures zero false assignments on these telltale tokens.

4. Hyperparameter Comparison

5. Deployment & Cloud Execution

The updated solution is open-source and committed as Version 6 of our Kaggle kernel:

• 🔗 Kernel: angelojun2k/name-that-llm-grandmaster-champion
• Hardware: Kaggle T4 x2 Dual-GPU
• Runtime: Subprocess GPU memory cleanup ensures zero OOM across 8-hour execution budget.

What stylometric markers have you found most invariant across frontier LLM completions? Share your findings in the comments below!