A one-shot script that classifies a forum archive feels done the moment the catalog shows up. It is not done. It has not met next Sunday.
The idea is less glamorous than a new video site, which is how you know it will still matter on Monday. An LLM label is a pure function of the text, the prompt, and the model. If none of those changed, you already bought the answer. Write it down. The second run should spend almost nothing.
This week's exhibit is a catalog of YouTube videos people on Hacker News have recommended since 2007. The interesting object is the pipeline, and it splits into two stages that should never share a function.
Stage one is a parser. Walk a comment, pull out a YouTube URL or video id, and store the hit next to the comment id. That step is deterministic. Run it every night. It will give you the same rows it gave you yesterday, which is the highest compliment a pipeline can receive.
Stage two is the judgment. A link is not a recommendation. One person is praising a conference talk. Another is pasting the same URL to complain about the codec. A third comment is spam with a video id glued to the end. You want those separated, and you probably want a topic so the catalog is browsable. That call is the whole budget.
On the Hacker News thread, the author reported about 504 million input tokens and 7.6 million output tokens for the project, more than 90 percent of them spent labeling comments. Building and deploying the app was on the order of a million tokens. Sit with that ratio before you schedule anything. The interface is a view over a table. The table is what you paid for.
Keep the table. One row per judgment:
- the comment id
- a hash of the comment body
- a prompt version
- the model name
- the label
- when you labeled it
The hash matters because comments get edited, and an edit is a new input. The prompt version is an integer you increment when the instructions change. A lookup key of comment id, content hash, prompt version, and model turns a rerun into a read. You are not asking the model whether it still agrees with itself.
Beside the table, store a watermark: the newest item id you have fully processed. Each run fetches only newer items, extracts links, and calls the model only when that key is missing. Then it moves the watermark forward. The first run walks the archive. Every later run is a delta. If the front page adds forty comments overnight, you pay for forty judgments, not for 2007.
The common failure is a script with an honest name like rebuild.py. It downloads the corpus and labels every row, on the theory that the provider's prompt cache will catch the repeat. It will not, not for long. Those caches expire. They miss when you change a single line of the system prompt. They also do not promise the same output twice, so a comment from 2009 can grow a new topic on Thursday and the catalog flickers for no editorial reason. Cached input does not make output tokens free. You have built a small business whose only product is paying a model to re-read "this talk is great."
There is a quieter version of the same bug. You change the prompt, forget to bump the version, and celebrate a perfect hit rate. You are serving last month's policy under this month's filename. Stale labels are fine when you chose them and can see the migration. They are a defect when the cache hid them.
Tomorrow, add the table before you add the schedule. At the end of every run, print two counts: comments seen, and comments sent to the model. After the first backfill those numbers should diverge, hard. If they stay equal, the watermark is a comment in the code, and you are still running the demo.
This post was generated automatically from public Hacker News data.
