I wanted to know what people actually type before a trip to China. Not what I assumed, and not what travel blogs cover — what goes into the box.
Google does not sell that data cheaply. But it does hand out its autocomplete endpoint for free, and that endpoint returns the phrases real people typed, ordered by how often they type them.
So I scraped my own question into it.
The harvest
Three layers, about 2,560 requests:
# 1. seed phrases SEEDS = ['china', 'china travel', 'china visa', 'china hotel', 'china sim card', ...] # 2. question prefixes — this is where intent lives PREFIXES = ['how to', 'how much', 'can i', 'do i need', 'best', 'cheapest', ...] # 3. alphabet soup on the shortest results for seed in top_50_shortest: for ch in 'abcdefghijklmnopqrstuvwxyz0123456789': yield seed + ' ' + ch def suggest(q): url = f'https://suggestqueries.google.com/complete/search?client=firefox&hl=en&gl=us&q={quote(q)}' return json.loads(requests.get(url).text)[1] Result: 10,102 phrases, zero API failures, nine minutes on five workers.
Three kinds of garbage
Autocomplete is a firehose of homonyms. All three of these cost me a pass:
1. Brand collisions. china one returns the US restaurant chain China One — cities, addresses, opening hours. Also china wok, china buffet, china garden.
2. Echo. Feed how to china and the endpoint obligingly returns how to china visa — a phrase nobody says, generated by my own malformed seed. Real users write how to get a china visa.
3. Substring hits. The worst one, because it silently corrupts your grouping. I filtered travel relevance with rain, and china approves brain implant sailed through: brain. Same trick with hot inside photo.
# wrong r'rain|hot|bus|eat' # right r'\brain\b|\bhot\b|\bbus\b|\beat\b' Word boundaries. Always.
Clustering: 6,938 → 5,394
After filtering for travel relevance: 6,938 phrases. Now group them.
The naive approach — exact string dedupe — gets you nowhere, because these are the same question:
china visa cost china visa how much how much is china visa how much does china visa cost So: tokenise, drop stopwords, and cluster by Jaccard similarity on the token sets.
def toks(s): return frozenset(w for w in re.findall(r"[a-z0-9']+", s.lower()) if w not in STOP and len(w) > 1) def jaccard(a, b): return len(a & b) / len(a | b) # union-find over pairs above threshold for a, b in candidate_pairs: if jaccard(toks(a), toks(b)) >= 0.7: uf.union(a, b) Candidate generation matters: comparing all 6,938² pairs is 48 million comparisons. An inverted index on the tokens (and skipping ultra-common tokens like visa that would produce enormous posting lists) cuts it to about 950,000 pairs, which runs in under a second.
Output: 5,394 distinct questions. Average 1.29 phrasings each.
The distribution is the finding
The biggest single cluster was 80 different phrasings of "what should I do with ten days in China". Second: 65 ways of asking whether you can visit without a visa. Third: 55 ways of asking what a high-speed train costs.
Then by theme:
transport 1,183 clusters money 693 visas 590 planning 486 connectivity 380 timing 245 food 242 hotels 232 What is missing is the interesting part. No cluster of meaningful size for "hidden gems", "photo spots", "local secrets". The demand is overwhelmingly logistics — how to get there, how to pay, whether the hotel will take a passport.
What I did with it
I stopped writing pages from intuition and started writing them from clusters. Every page now carries the list of exact phrasings it targets, printed on the page:
What this entry answers
china hotel for foreigners · china hotel not for foreigners · how to book hotel in china for foreigners
That is not keyword stuffing — it answers the reader's actual question ("does this page cover my case?") and it makes the coverage measurable. 5,394 clusters, 2,235 answered: 41%.
The rest is a to-do list that writes itself.
The dataset and the pages are at sinostay — the answers are at the wiki. The scraper is a hundred lines of Python and one regex you will get wrong the first time.
