Synthetic users work for one job: ranking and stress-testing options you've already written, in categories the model knows well. The best study so far got AI consumers to about 90% of real panels' own consistency on purchase intent. They don't work as a stand-in for real buyers. They're too agreeable and too shallow, and they only predict what people say.

That's the verdict from the primary sources below. If you're a founder deciding whether to trust an AI "customer" before you spend money or time, this page tells you what was actually measured, where it falls apart, and how to use it without fooling yourself.

Run your positioning past simulated buyers with RightPositioning →

If you want the basics of how a simulation run works first, read synthetic buyer simulation explained.

How do synthetic users work?

A synthetic user is a large language model told to answer as a particular person. The tool writes a persona (a job title, age, income, location, sometimes a personality profile or notes from real interviews), asks your question, and records the answer. Then it does that dozens or hundreds of times with different personas and adds the answers up.

The differences between tools come down to three choices:

  1. What the persona is built from. A one-line demographic prompt, a richer profile, or real data like interview transcripts and survey answers. The research below shows this matters a lot.
  2. How the answer is collected. Asking the model for a number ("rate 1 to 5") or asking it to react in its own words and scoring the text afterwards.
  3. One voice or a group. Separate interviews, a survey, or several personas discussing the offer together.

What did the research actually find?

Four sources are worth your time. Each tested something different, so read the numbers for what they measure.

Study What it tested Headline result What it doesn't show
Maier et al., PyMC Labs and Colgate-Palmolive (arXiv, 2025) GPT-4o and Gemini 2.0 Flash as consumers on 57 personal care concept surveys, 9,300 real respondents About 90% of human test-retest reliability on ranking concepts Actual purchases, or categories outside personal care
Brand, Israeli and Ngwe, "Using GPT for Market Research" GPT as a random shopper: demand curves, income effects, willingness to pay WTP for fluoride in toothpaste of $3.40 vs $3.27 from a human study New features without prior human data
Rosala and Moran, Nielsen Norman Group (2024) Synthetic Users and ChatGPT re-running three real interview studies Answers "too shallow to be useful", over-positive, no priorities Survey-style ranking tasks
Park et al., Stanford and Google (arXiv) AI twins of 1,052 real people built from two-hour interviews 72.8% accuracy on survey items vs 79.5% for people repeating their own answers Buyers you haven't interviewed

The PyMC Labs / Colgate-Palmolive paper: purchase intent

This is the paper people quote when they say synthetic users work. The authors, from PyMC Labs and Colgate-Palmolive, took 57 concept surveys "supplied by a leading personal care corporation", with 9,300 real responses, and asked LLM personas the same purchase-intent question (arXiv 2510.08338).

The finding that matters is about method. When they asked the model for a 1 to 5 rating directly, it mostly answered "3" and rarely used 1 or 5, while real people mostly picked 4 and 5. The distribution match was poor: a KS similarity of 0.26 for GPT-4o (full paper). So they had the model answer in words, then scored those words against reference statements by meaning. They call this semantic similarity rating (SSR). With SSR, GPT-4o reached 90% "correlation attainment", meaning its ranking of the 57 concepts agreed with humans about 90% as well as one half of the human panel agreed with the other half. Distribution similarity rose to 0.88.

Two more results are easy to miss. Without demographic personas, the model's average purchase intent still matched humans, but its ability to tell good concepts from bad ones fell to 50%, against 92% with personas. And it reproduced age and income patterns but not gender or region.

Brand, Israeli and Ngwe: willingness to pay

The Harvard and Microsoft team prompted GPT as "a customer randomly selected while shopping" and repeated each prompt hundreds of times (slides, Israeli, May 2024). Demand sloped down as price rose, income mattered, and a conjoint study put willingness to pay for fluoride in toothpaste at $3.40, close to the $3.27 a human study found. The 10,800 responses cost $3 and took 35 minutes.

The same slides flag that GPT "gravitates toward the first option presented". And the later version of the paper adds a hard limit: the model needs fine-tuning on prior human results to predict willingness to pay for new features, and fine-tuning "may not improve LLMs ability to extrapolate to unrelated product categories" (MSI working paper, 2025).

Nielsen Norman Group: interviews

NN/g ran the commercial tool Synthetic Users and ChatGPT against three studies it had already done with real people (NN/g, 2024). Real learners admitted they'd quit online courses. Synthetic ones said "Yes, I completed all the courses I mentioned." Real learners mostly avoided discussion forums. Synthetic ones said forums mattered. Synthetic users produced long lists of needs with no sense of which ones counted, and liked concepts because the model tends to please whoever's asking.

NN/g's 2025 follow-up reviewed three more academic studies and landed in the same place: AI-simulated users can fill data gaps and forecast population-level trends, but should "complement, not replace, human-centered research" (NN/g, 2025).

Stanford and Google: twins built from real interviews

Park et al. interviewed 1,052 people for two hours each and built an AI version of every one. Built from interviews plus surveys, the twins matched their person's General Social Survey answers 72.8% of the time. The people themselves only matched their own answers two weeks later 79.5% of the time. Twins built from demographics alone managed 60.4%, and the authors say demographic-prompt simulations "often misjudge individuals and distort sample-level patterns."

Where do synthetic users break?

Every source above names its own failure points. Here they are, matched to what you'd see.

Where it breaks What you'd see Source
New categories Confident answers about a product type the model has little data on SSR "will not conjure valid consumer preferences" where training knowledge is sparse (Maier et al.); fine-tuning may not carry to unrelated categories (Brand et al.)
Stated intent vs behaviour High "would buy" scores that never turn into sign-ups Every result above is compared with survey answers, not purchases; SSR "cannot fully capture the real-world contingencies of purchasing behavior" (Maier et al.)
Agreeable answers Personas like your idea, claim good habits, accept your framing NN/g's course-completion and concept findings (NN/g); assistants tend to tell users what they want to hear (Sharma et al., Anthropic)
Shallow personas Every buyer sounds alike, no ranking of what matters Demographic-only personas dropped concept discrimination to 50% (Maier et al.) and scored 60.4% vs 72.8% for interview-built twins (Park et al.)
Forcing a number Every rating lands on 3 Direct 1 to 5 ratings collapsed to the middle (Maier et al.)
Groups agreeing too fast A simulated discussion where everyone ends up nodding Role-playing agent groups converged much more than real human groups (DEBATE benchmark, NeurIPS 2026)
Prompt side effects Changing the price also changes what the persona assumes about quality Changing one treatment shifts other attributes in simulated subjects (Gui and Toubia)
Order effects The first option you list wins GPT "gravitates toward the first option presented" (Israeli slides)
Niche groups Patchy, generic answers for specialist buyers NN/g advises against synthetic users for niche or specialised groups (NN/g)

Notice the pattern. Synthetic users do best on aggregate questions ("which of these concepts will people prefer?") in categories with lots of public data. They do worst on individual depth ("why did this person quit?") and on anything the model hasn't seen.

How do you use synthetic users well?

Use them where the research says they hold up, and build in the fixes the papers found.

  1. Bring options, not open questions. Write two prices, three headlines or two segments before you run anything. Ranking options is the job the evidence supports. If you can't write two versions yet, you're still in discovery, so go talk to buyers.
  2. Describe the buyer in detail. Job title, company size, what they use today, what they've said in calls. Thin personas lost most of their signal in both the PyMC and Stanford studies.
  3. Read the words, not just the score. The best result came from free-text reactions scored afterwards. The objections and the phrases that lost people are the useful part.
  4. Add a skeptic on purpose. Synthetic buyers lean agreeable, so include a persona whose job is to push back and one who prefers a competitor. Treat unanimous praise as a warning sign.
  5. Shuffle the order and the framing. Swap which option comes first and run it again. If the winner flips, you don't have a winner.
  6. Run it twice. If two runs disagree, the difference between your options is smaller than the noise.
  7. Take the top objection to a real buyer. Use the simulation to decide which question to ask in your next five conversations, then let real sign-ups or replies settle it.

For how to split work between simulations and interviews, see simulated buyers vs customer interviews. If you're choosing a tool, synthetic user research tools compared covers who each one suits and what it costs.

Symptom, check, fix

Symptom Check Fix
Every persona likes your idea Did you include a skeptic or competitor-minded buyer? Add them, and ask for objections before ratings.
Scores bunch in the middle Are you asking for a number directly? Ask for a reaction in words and score that.
Results flip between runs Do two runs of the same input agree? Treat the options as tied and test both with real buyers.
The first option always wins Does swapping the order change the result? Randomise order, or present one option per run.
Generic answers for a niche buyer Is your persona one line of demographics? Add role, tools, budget owner and real quotes from calls.
Great simulation, no sign-ups Were you measuring intent or behaviour? Run a small real test: a landing page, 20 cold emails, or a quote on a call.

FAQ

Do synthetic users work?

For some jobs, yes. In a 57-survey study with 9,300 real respondents, AI consumers ranked personal care concepts at about 90% of the consistency real panels have with themselves. But Nielsen Norman Group found synthetic interview answers too shallow and too agreeable to replace real users. Use them to rank and stress-test options, then confirm with real buyers.

How do synthetic users work?

A large language model is given a persona, like a job, age, income or a set of past interview notes, and asked to answer as that person. The tool repeats this many times with different personas and adds up the answers. Better tools ground personas in real data and ask for free-text reactions instead of forcing a number.

What does Nielsen Norman Group say about synthetic users?

NN/g re-ran three of its own real-user studies with synthetic users and found the answers shallow, over-positive and lacking priorities. Its advice is to use synthetic users for desk research and hypotheses, treat the output as hypotheses to test, and never present it as real-user research.

Can synthetic users predict whether people will actually buy?

Not directly. The strongest studies compare AI answers with what real people said in surveys, which is stated intent, not purchases. The PyMC Labs paper says the method can't fully capture the real-world contingencies of buying. Treat a high score as a reason to run a real test, not as a sales forecast.

When should I not use synthetic users?

When the category is new and the model has little to draw on, when you're studying a niche group, when you need to know why people behave the way they do, or when the decision is expensive and hard to reverse. Those need real conversations or a real market test.

Next step

Write down the two versions of the decision you're least sure about this week, two positioning angles or two prices, and run both past simulated buyers with a skeptic in the room. Then take the top objection into your next real sales call.

Compare your positioning against your alternatives with RightPositioning →

Originally published on Right Suite.