I run Stockinsky, a small site that writes paid AI research reports on listed companies. Most companies will never get a paid report, but people still search for them by name. So I added a free page for every stock: ten years of financials, a red-flag checklist, dividend history and the latest close.

There are now about 2,000 China A-share pages, plus around 530 Hong Kong and 530 U.S. ones. Here is how they are built, and the one incident that shaped the design.

The rule: a page view never touches an upstream API

The first version fetched data when someone opened a page and cached the result. That works fine for people. It does not work for crawlers. A search engine bot can open hundreds of stock pages in a few minutes, and every cache miss turns into several requests to free market-data endpoints.

Then I made it worse myself. I ran a bulk measurement script on the production server, and the quote provider started rate-limiting the server's IP. Live quotes on the site broke for real users. The site and my batch job shared one IP, so the batch job took the site down with it.

After that I set one rule: the server only reads files. All fetching happens on my own machine.

Snapshots, one JSON file per stock

A local script builds one JSON file per stock:

data/stock-snapshot/A/688981.json data/stock-snapshot/HK/... data/stock-snapshot/US/...

Each file holds:

  • finance: ten years of revenue, net profit, ROE, gross margin, operating cash flow;
  • risk: a slimmed-down red-flag checklist (goodwill, cash and debt both high, share pledges, audit opinion, regulator inquiries...), each with the value, the rule and the source;
  • dividends: cash dividend per share for the last ten years;
  • quote: the close, change and market cap on the day the snapshot was built, with that date shown on the page.

The script then syncs the files to the server, which renders pages from disk. A crawler can open every page on the site and not a single upstream request goes out.

Being polite to free endpoints

The data sources are free public endpoints, so the build is deliberately slow:

  • Concurrency 2 by default, capped at 3. Each stock needs about 8–10 requests and takes roughly 12 seconds, so 1,000 stocks take about an hour and a half.
  • Shared data is fetched once per run. Quotes for the whole market come from about 100 batch requests, and the dividend table from about 20, instead of one call per stock.
  • Resumable. Snapshots newer than 7 days are skipped, so an interrupted run just starts again. One stock failing is logged and does not stop the others.

Slow is fine here. These numbers change once a quarter, not once a minute.

Thin pages get noindex, not a 404

Free endpoints time out, and some stocks come back with gaps. Some are suspended for months. Instead of hiding those pages, I gate what goes into the sitemap:

// a page enters the sitemap only if it passes indexable(): // enough data, not long suspended, snapshot not too old. // Pages that fail still open normally, but carry noindex.

Visitors who land on any stock still get whatever data exists. Search engines are only asked to index the pages that are actually useful.

A timezone bug worth sharing

Snapshot dates were cut from toISOString(), which is UTC. The market data is in Beijing time (UTC+8), so a snapshot built before 8 a.m. local time was labelled with the previous day. Nothing crashed. The page just showed the wrong "as of" date. Format dates in the market's own timezone, not UTC.

Why no LLM on these pages

The site already uses LLMs for paid reports, so this was a deliberate choice:

  • Cost. Generating text for 3,000 stocks every week would be paid by the site, for pages that earn nothing directly.
  • Trust. Numbers that come straight from filings, with the source next to them, beat a fluent paragraph that is probably right.

So the free pages are purely mechanical: the same inputs always give the same page. The model only runs when a user asks for a full report.

What I would do again

  1. Separate the machine that fetches data from the machine that serves pages, even for a tiny site.
  2. Treat crawlers as a load pattern you design for, not an afterthought.
  3. Let missing data lower indexability instead of breaking the page.

Here is an example page: SMIC (688981). The free pages for China A-shares are in Chinese for now; the English site covers U.S. and Hong Kong stocks.

I built Stockinsky. The data comes from public disclosures and may lag or contain errors; company filings are the final word. Nothing here is investment advice.