Coding agents don't read a repo the way you do. You skim past uv.lock without thinking; a grep, a glob or a careless cat from an agent doesn't. I wanted to know how much of a typical repo is text a model pays for but can't learn anything from, so I built a small CLI to count it and ran it on six well-known projects.
The numbers
| Repo | Total tokens | Flagged | Biggest single source |
|---|---|---|---|
| langchain | 9.82M | 55% | 22 uv.lock files, 5.11M tokens |
| ollama | 9.26M | 51% | integration/testdata/shakespeare.txt, 1.33M tokens |
| excalidraw | 4.50M | 54% | a wasm wrapper and a font source file, ~590k |
| fastapi | 8.47M | 43% | one drawio SVG, 631k tokens |
| shadcn/ui | 7.79M | 38% | 22 lockfiles, 813k tokens |
| express | 196k | ~0% | nothing worth flagging |
Shallow clones, scanned 12 and 13 Nov 2026.
These are whole-tree totals, so read them as an upper bound. No agent loads an entire repo in one go. But agents do search, glob and open files they don't need, and every file above is a place where that goes badly.
What stood out
LangChain's lockfiles outweigh its code. It's a monorepo and every package carries its own uv.lock. Together they come to 5.11M tokens. Everything an agent could actually use comes to 4.45M.
ollama keeps the complete works of Shakespeare as test data. 1.33M tokens, more than a whole 1M-token context window. A perfectly reasonable test fixture, and a terrible thing for an agent to open.
fastapi's biggest file is a diagram. A single drawio SVG is 631k tokens of XML path data.
express is clean. Small, hand-written repos are fine. The problem is rarely your code. It's three or four files nobody thinks about.
How it decides what counts as waste
There are 14 categories: lockfiles, build output, minified bundles, generated code, snapshots, bulk data, duplicates, vendored dependencies, binaries and so on.
The rules are deliberately conservative. Folders like packages/, lib/ and bin/ are not flagged, because monorepos and CLIs keep real source there. .d.ts files are kept too, since type declarations are good context. Telling you to hide your own code from your agent would be worse than missing a lockfile.
Token counts come from a rule-based BPE approximation: no vocabulary file, no network. Fitted against gpt-tokenizer (o200k_base) over 435 files, the median per-file error is 6.1% and the corpus-wide bias is 0.0%. The calibration script is in the repo if you want to rerun it. Claude's tokenizer is different, so treat Claude numbers as estimates of an estimate.
Try it on your repo
npx github:geraltbhai/ctxray No install, no dependencies, no network calls, MIT. It prints the breakdown and prices a full read on current Claude, GPT and Gemini models. ctxray init then writes an .agentignore built from what it actually found, plus an AGENTS.md scaffold with your repo's real shape filled in.
There's a paid tier for the HTML report, PR diffs and CI checks. Everything in this post is the free part.
If it flags something on your repo that it shouldn't, drop the path in the comments. I'm treating false positives as bugs.
