Freeze a prompt set and a list of countries, collect every prompt several times in each country from ChatGPT's web interface, and store the raw JSON, the rendered HTML, the timestamp and the model ChatGPT used with every answer. A single answer per prompt is a sample of one from a system that changes between runs and between weeks.
- The model under your dataset changes. In our own monitoring of a fixed prompt set,
gpt-5-6produced 92% to 99.8% of ChatGPT answers each week in September. From the week of 12 October,gpt-6produced 86% to 97%. Themodelfield is how we saw the switch. - Part of the answers carry no model name. Between 0.2% and 66% of answers each week had
model: null, which marks OpenAI's mobile-web response format. A dataset that drops those rows drops a share that moves by week. - The size in the title costs one entry-level plan. 500 prompts in 3 countries with 5 samples each is 7,500 answers. At 5 credits per async answer that is 37,500 credits, the $30 Lite plan.
The steps:
- Write the prompt set, give each prompt an ID, and freeze it as version 1.
- Pick the countries, and write each prompt in the language of its country.
- Decide the samples per prompt and country, and spread them over days.
- Submit the tasks in async batches of up to 500, each with a webhook URL and an
idempotencyKeythat encodes prompt, country, sample and wave. - Store every webhook body as it arrives, and download the HTML within 24 hours.
- Analyse with
model, country and collection time as variables.
How do I build a dataset of ChatGPT answers for research?
Treat each answer as one observation of a cell: prompt × country × wave × sample.
| Decision | Recommendation | Why |
|---|---|---|
| Prompt set | A fixed list with IDs and a version number | You can compare waves only on the same prompts |
| Countries | Only the ones your question needs; check GET /v1/countries?model=chatgpt | Supported countries change; each country multiplies the cost |
| Language | The language of the country | The country sets the proxy location, and the prompt sets the answer language |
| Samples per cell | 3 to 5 in one wave, more if your measure is rare | Answers and their sources vary between runs |
| Waves | Repeat the whole set on a schedule, for example weekly | The model and the sources change between weeks |
| Web search | Decide, and keep it the same | ChatGPT answers differently with and without search |
For scale, a 2026 preprint ran each of 80 questions 20 times on two ChatGPT versions. A Hamburg study of more than 24,000 German news answers over five weeks found that source diversity changed from week to week, as The Decoder reported.
Why sample each prompt more than once?
ChatGPT's answer to one prompt is a draw from a distribution. The searches ChatGPT runs for a prompt vary in length and structure between runs, and the Hamburg study saw the cited sources shift from week to week. In our daily monitoring, a brand that ChatGPT named for a prompt on one day was named again the next day 75% of the time at the median. If your measure is a share (how often a brand is named, how often a domain is cited), you need several draws per cell to estimate it, and a confidence interval to report it.
Spread the samples. Five samples in one minute measure run-to-run noise; five samples on five days also catch day-to-day drift, at the same cost.
Why log the model and the timestamp?
The web app does not let you pin a model. ChatGPT picks one for each request, and the choice changes over time. Our monitoring logged these shares of ChatGPT answers per week, on the same prompt set:
| Weeks starting | Main value of model | Share of answers per week | null per week |
|---|---|---|---|
| 10 to 31 August 2026 | gpt-5-6, with some gpt-5-5 and gpt-5-5-mini | 26% to 81% gpt-5-6 | 16% to 66% |
| 7 to 28 September | gpt-5-6 | 92% to 99.8% | 0.2% to 1.3% |
| 5 October | gpt-5-6 and gpt-6 | 55% and 43% | 1.5% |
| 12 October to 9 November | gpt-6 | 86% to 97% | 3.1% to 14.1% |
The mini model appeared on 2.5% to 4% of answers in three August weeks and on none from the week of 31 August. A study that ran from August into October would compare three models without knowing it, unless it logged the field.
In the API docs, null marks answers that OpenAI served in its mobile-web format, which also has no searchQueries and a shorter sources list. Keep those rows and code null as its own value. The request flag legacy: true asks for the legacy interface, which returns the model name, on a best-effort basis.
For time, keep task.createdAt (when you submitted) and your own receipt time. task.latencyMs is the processing time and leaves out the queue wait, so neither field alone is the exact moment ChatGPT answered.
How do I collect the answers in batches?
The async batch endpoint takes up to 500 tasks per request and calls your webhook once for each finished task. Its parameters are in the batch endpoint reference. This submits one wave:
import itertools import os import requests PROMPTS = {"p001": "What is the best CRM for a small business?"} # frozen set COUNTRIES = ["US", "GB", "IE"] SAMPLES = 5 WAVE = "2026-11-16" tasks = [ { "taskType": "CHATGPT", "idempotencyKey": f"v1-{pid}-{country}-{WAVE}-s{s}", "webhook": {"url": "https://example.org/webhook"}, "payload": {"prompt": text, "country": country, "include": {"html": True}}, } for (pid, text), country, s in itertools.product(PROMPTS.items(), COUNTRIES, range(SAMPLES)) ] for i in range(0, len(tasks), 500): r = requests.post( "https://api.cloro.dev/v1/async/task/batch", headers={"Authorization": f"Bearer {os.environ['API_KEY']}"}, json=tasks[i : i + 500], timeout=60, ) data = r.json() print(data["summary"]) for res in data["results"]: if not res["success"]: print(res["index"], res["error"]["code"]) The idempotencyKey must be unique across your account, so a resubmitted task returns 409 and is not charged twice. It also comes back in every webhook as task.idempotencyKey, so each answer maps to its cell without a lookup table.
The receiver stores the raw body under the task ID. The service makes up to 5 delivery attempts, so the same task can arrive twice, and writing to <task.id>.json makes the second copy harmless:
import json import pathlib from flask import Flask, request app = Flask(__name__) RAW = pathlib.Path("raw") @app.post("/webhook") def receive(): body = request.get_data() # raw bytes, also needed for signature checks task = json.loads(body)["task"] (RAW / f"{task['id']}.json").write_bytes(body) return "", 200 Two clocks run out after about 24 hours: finished tasks are deleted from the API, and the response.html link expires. A second job should fetch each HTML file and store it beside the JSON on the day it arrives. The queue holds up to 100,000 waiting tasks per organization.
What should each stored record contain?
| Field | Source | Use in analysis |
|---|---|---|
| Prompt ID, country, wave, sample | Your idempotencyKey | The cell |
task.id, task.createdAt, receipt time | Webhook body, your server | Deduplication and time |
response.model | Webhook body | Model as a variable; null as its own value |
response.text, response.sources, response.citationPills | Webhook body | The answer and what it cited |
credits.creditsCharged | Webhook body | Cost per cell |
| HTML file | response.html, downloaded | Evidence of what the page showed |
| The full raw JSON | Webhook body | Re-parsing when your coding scheme changes |
A field you ignore today, such as entities or footnote on each source, can become a variable in the next paper, so keep the raw JSON.
How much does a research dataset cost?
An async ChatGPT answer costs 5 credits. Query fan-out (include.searchQueries) adds 2, and a US state adds 2. HTML and Markdown are free, and failed tasks cost nothing. Credits expire at the end of each monthly cycle, so a long study pays per month of collection.
| Dataset | Answers | Credits | Plan that covers one month | Plan price per answer |
|---|---|---|---|---|
| Pilot: 50 prompts × 1 country × 2 samples | 100 | 500 | Free ($0, 500 credits) | $0 |
| 500 prompts × 3 countries × 5 samples | 7,500 | 37,500 | Lite ($30, 37,500 credits) | $0.004 |
| The same with query fan-out (7 credits) | 7,500 | 52,500 | Hobby ($100, 250,000 credits) | $0.013 |
| 500 × 3 × 5, weekly for 4 weeks | 30,000 | 150,000 | Hobby ($100, 250,000 credits) | $0.0033 |
| 1,000 prompts × 5 countries × 10 samples | 50,000 | 250,000 | Hobby ($100, 250,000 credits) | $0.002 |
The 500 × 3 × 5 design uses all of the Lite credits, with none left for a rerun. Lite allows 10 concurrent tasks and Hobby 20, which sets how fast a wave finishes.
What do the terms and research ethics require?
OpenAI's Terms of Use (effective 1 January 2026) list "Automatically or programmatically extract data or Output" among the things users may not do. Your ethics board or legal team should see that clause and your collection method before the study starts.
The route in this post uses logged-out sessions, with no account of any kind, and reads only new answers to your own prompts. It does not reach other users' conversations. If your prompts come from real people, for example from a survey, they are research data about those people, and the usual consent rules apply to them.
In the paper, report the collection route, the logged-out state, the countries, the dates of each wave, and the share of each model value, null included, so that others can compare their data with yours.
How was this checked?
The model shares come from our own monitoring of ChatGPT on a fixed prompt set, 10 August to 15 November 2026, as shares of successful answers per week. The endpoint paths, fields, limits and credit prices come from the API docs, and the plan prices from the published plan table, on 16 November 2026. This post is published by cloro, which sells the API used in the code above.