Most ideas lose when tested. Name the riskiest belief, decide how strong the evidence must be, then pick the cheapest of 55 product experiments that gets you there.

Product experiments are small, planned tests of one belief your plan depends on, with a pass signal written before you start. To choose one, name the belief, name its risk (desirability, usability, feasibility or viability), decide how strong the evidence must be, then pick the cheapest method that reaches that strength. This guide sorts 55 methods that way, with time and effort for each.

Key takeaways

  • Most ideas lose when tested. Only one third of the ideas tested at Microsoft improved the metric they were designed to improve, and about 10% of Google's controlled experiments led to business changes (Kohavi and Longbotham, 2023).
  • Test the riskiest belief, not the whole idea. Marty Cagan's four risks are value, usability, feasibility and business viability (SVPG, 2017).
  • Evidence comes in strengths. What people say is weakest; money or a signature is strongest. Strategyzer's library for Testing Business Ideas rates 44 experiments from 1 to 5 on cost, setup time, run time and evidence strength.
  • Cheap does not mean weak. In that library, a concierge test scores 1 of 5 for cost and 5 of 5 for evidence.
  • Write the pass signal before you start. Moving it afterwards is how most teams fool themselves.

Why do product ideas need experiments at all?

Most product ideas do not fail because the team built badly. They fail on a guess nobody checked.

The record from teams that measure is humbling. Ron Kohavi and Roger Longbotham collect it in one place: a third of ideas tested at Microsoft moved their target metric, Jim Manzi wrote that about 10% of Google's experiments led to business changes, and Netflix is reported to consider 90% of what it tries to be wrong (Kohavi and Longbotham, 2023). These companies employ excellent product people. Their first guesses still lose most of the time.

Skipping the check costs more later. CB Insights studied 431 venture-backed companies that shut down since 2023 and found reasons for 385 of them: 70% ran out of capital, 43% had poor product-market fit, 29% cited bad timing and 19% had unsustainable unit economics (CB Insights, 2026). Companies could give more than one reason. Poor fit and broken unit economics are exactly the doubts a cheap experiment can probe months before the money runs out.

A product experiment is how you check a guess before you spend real money on it. You do not need a research team, and you do not need all 55 methods in this guide. You need the one test that could change your mind about the belief your plan depends on most.

What is a product experiment?

A product experiment is a small, planned test of one belief, with a result you define before you start. Good product experiments have four parts:

  1. The belief, written as a sentence that could be false. "Freelancers lose work because follow-ups slip," not "freelancers need this."
  2. The method: the smallest action that produces evidence about that belief.
  3. The measure: one number, plus one thing that must not get worse.
  4. The pass signal: the result that would make you say yes, set in advance.

Teresa Torres puts the core move plainly: stop testing whole ideas and test the assumptions they depend on (Product Talk, 2023). An idea like "a reminder app for vets" holds a dozen beliefs. Most are safe. One or two would kill it, and those are the ones worth an experiment. If you are starting from a feature request, the product problem statement template turns it into a belief you can test.

What are the four product risks an experiment can test?

Every belief carries a kind of risk, and the kind decides which methods fit. Marty Cagan of Silicon Valley Product Group names four (SVPG, 2017):

Risk Cagan's question Strategyzer's word Usual owner Cheap first test Strong test
Value Will customers buy it, or choose to use it? Desirability Product manager Customer interview, fake door Concierge, letter of intent
Usability Can users figure out how to use it? Tagged as desirability Product designer Paper prototype Usability test on a real build
Feasibility Can engineers build it with the time, skills and technology we have? Feasibility Lead engineer Expert interview, technical spike Wizard of Oz, mash-up
Business viability Does it work for the rest of the business? Viability Product manager Pricing survey, market research Funnel test, letter of intent

Bland and Osterwalder's Testing Business Ideas (Wiley, November 2019, 368 pages) groups risk into three questions: do customers want it, can we build it, and will it earn sustainable revenue (Wiley; Strategyzer, 2025). Its experiment library tags usability-style tests, such as card sorting and clickable prototypes, as desirability. Cagan keeps usability separate, and so do we, because "they want it" and "they can use it" fail in different ways and need different tests.

Torres adds a fifth category, ethical: should we build this at all? Cagan's advice on order is blunt: tackle the big risks early, especially value and business viability. For a deeper look at what counts as proof for each risk, read what counts as evidence for the four product risks.

How strong is the evidence from each kind of experiment?

The higher the step, the more you can trust the result.

Not all evidence is equal. A kind word and a signed letter are both "positive feedback," but only one should move your roadmap. Alex Osterwalder makes the same point: what customers say and what they do are two different things, so interviews and surveys give valuable but weak evidence, while a simulated sale gives strong evidence (Strategyzer, 2017).

Strength What it means Example Treat it as
What people say Someone told you something An interview, a survey A reason to run a stronger test
Early signal Real behaviour, but cheap or indirect An ad click, a search trend A clue about where to look
What people do Someone took a real action Finishing a task, leaving an email Good evidence for that specific action
Money or a signature Someone gave up something of value A payment, a signed letter The strongest evidence you can get before launch

Specky uses the same ladder. A belief can only be marked Supported by what people do, or by money or a signature. Anything weaker stays an Early signal, however good it sounded.

Does a stronger test always cost more?

No. Strategyzer's experiment library poster rates every method from 1 to 5, and cost and evidence strength move independently (Strategyzer experiment library). A sample:

Method Cost (1-5) Evidence strength (1-5) What the pairing tells you
Customer interview 2 1 Cheap and fast, but only words
Discovery survey 2 1 Counts a problem; proves no demand
Paper prototype 1 1 The cheapest rung for a flow
Search trend analysis 1 3 Strangers' behaviour for almost nothing
404 test 1 3 Also rated 1 for setup and run time
Concierge 1 5 Low cost, top evidence, but slow to run
Wizard of Oz 2 5 Real service, faked automation
Extreme programming spike 2 5 Strong evidence on feasibility only
Presale 3 5 Money on the table
Single-feature MVP 4 5 Strong, but you are now building

The lesson: the expensive part of a strong test is usually your time, not your budget. A concierge test costs little money but weeks of manual work.

How do you choose the right product experiment?

Answer five questions, in order:

  1. What is the belief? Write it as one sentence that could turn out to be false.
  2. What kind of risk is it? Do they want it (desirability)? Can they use it (usability)? Can we do it (feasibility)? Will it make money (viability)?
  3. What would it cost to be wrong? A guess that decides the next six months deserves stronger evidence than one that decides a button colour. Barry O'Reilly makes the same point about evidence thresholds: a rocket launch needs a higher bar than a sign-up flow (Thoughtworks, 2014).
  4. What is the cheapest test that gives enough evidence? Start low on the ladder when you know little. Climb as the stakes rise.
  5. What would count as a pass? Write it down before you begin. O'Reilly's reason is the right one: signals stated in advance reduce biased readings of the result.

A rule of thumb we use: if being wrong costs a week, what people say is enough. If it costs a quarter, get what people do. If it costs a hire, a year or your runway, get money or a signature.

When you have several beliefs and cannot tell which is riskiest, map them first. Our guide to assumption mapping and the free assumption map tool sort beliefs by impact and evidence, so the test goes where it matters.

Product experiment decision table: which test for which risk and evidence level?

Find the row for your risk. Move right until you reach the evidence strength that being wrong deserves. Pick the cheapest method in that cell.

Risk What people say Early signal What people do Money or a signature
Desirability: is the problem real? Customer interview, discovery survey, forum research Search trends, support tickets, analytics check Diary study Move on to the offer row
Desirability: do they want this offer? Product box, brochure, validation survey Online ad, 404 test, email campaign Fake door, landing page, feature stub Concierge, crowdfunding
Usability: can they use it? Rarely useful: people misjudge their own ease of use Web traffic analysis Paper prototype, clickable prototype, usability test, card sorting Not the right question; usability is shown by doing
Feasibility: can we deliver it? Partner and supplier interviews, expert interviews Technical spike Pretend to own, mash-up, Wizard of Oz Concierge (paid delivery by hand)
Viability: will it make money? Pricing survey, sales team feedback, investor feedback Online market research, presale interest Funnel test Letter of intent, referral program, pop-up store
Typical time 1 day to 2 weeks 1 day to 1 week 2 days to 3 weeks 2 to 4 weeks
Typical effort Low Low Low to high Medium to high

Times and effort come from the method tables below and assume a small team. Read across a row to see how to climb. Read down a column to compare risks at the same strength.

Which product experiments answer which question? The full library of 55

Every method below links to its own guide, with a copyable template, a pass signal set in advance and a worked example.

Desirability: is the problem real, and who has it?

Start here when you are still unsure the pain exists. Most of these cost a few days and no code.

Test Evidence strength Time Effort Reach for it when
Customer interview What people say 1 week low You have a hunch and need the real story
A day in the life What people say 1 week medium People cannot describe work they do on autopilot
Discovery survey What people say 1 week low You want to know how common a problem is in one group
Survey What people say 1 week low You know the group and just need to settle one choice
Discussion forums What people say 2 days low You want people's exact words without taking their time
Storyboard What people say 2 days low The situation is hard to explain in words
Speed boat What people say 1 day low You have a product and want polite people to criticise it
Search trend analysis Early signal 1 day low You want a one-day read on whether people look for this
Web traffic analysis Early signal 1 day low You have steady visits and need to pick the step to test
Customer support analysis Early signal 2 days low You have a live product and a full inbox
Analytics Early signal 1 day low The behaviour is already logged
Diary study What people do 2 weeks medium The problem comes and goes, and people misremember it

Once you have run a handful of interviews, analyse them with a synthesis template before you count anything.

Desirability: will people want this offer?

Use these once the problem looks real. Now you are finding out if your particular answer gets a reaction.

Test Evidence strength Time Effort Reach for it when
Product box What people say 1 day low Your team cannot agree on the main benefit
Brochure What people say 3 days low You need to check your offer is clear
Validation survey What people say 1 week low People have seen the offer and you want to hear what would stop them
Online ad Early signal 1 week low You want a reaction from strangers
Link tracking Early signal 1 week low You are comparing messages or channels
404 test Early signal 3 days low You want the cheapest possible check on curiosity
Email campaign Early signal 3 days low You have a list that chose to hear from you
Social media campaign Early signal 1 week low Your customers gather somewhere you are welcome
Data sheet Early signal 1 week low Your buyers evaluate on numbers
Explainer video Early signal 1 week medium It is easier to show than to describe
Fake door What people do 1 week low You can state the offer in one sentence
Feature stub What people do 1 week medium You want to catch current users in the moment
Simple landing page What people do 1 week low You can send strangers to one clear page
Boomerang (competitor testing) What people do 1 week low Good alternatives already exist
Buy a feature What people do 2 days low Customers say everything matters
Split test What people do 2 weeks medium You have the traffic to prove a change caused a result

Usability: can people use it, and will they come back?

These put something close to the real thing in front of people. Ease of use and repeat use are separate questions, so read each row carefully.

Test Evidence strength Time Effort Reach for it when
Paper prototype What people do 2 days low You are still working out the steps
Clickable prototype What people do 1 week medium The flow makes sense and you want to test it for real
Usability test What people do 3 days low Something is built and people are getting stuck
Card sorting What people do 3 days low You are deciding how to name or organise things
3D print What people do 2 weeks high It is a thing people hold, wear, or mount
Life-sized prototype What people do 3 weeks high Size and space change the experience
Single-feature MVP What people do 3 weeks high Lighter tests passed and you need to see repeat use
Concierge Money or a signature 3 weeks medium You can deliver the outcome by hand and ask for money

Feasibility: can we build or deliver it?

Some ideas are loved and still impossible. Use these when the doubt is on your side of the table.

Test Evidence strength Time Effort Reach for it when
Partner and supplier interviews What people say 1 week low Your plan depends on someone outside your company
Expert stakeholder interviews What people say 1 week low Someone could block or fund this
Extreme programming spike Early signal 3 days medium One technical unknown could sink the plan
Pretend to own What people do 1 week medium You want to learn what one real delivery takes
Mash-up What people do 2 weeks medium Existing tools could do the job if connected
Wizard of Oz What people do 2 weeks medium The hard part is automating it, and the experience needs testing first

Viability: will it make money?

Interest is cheap. These tests get closer to the moment someone gives up something of value.

Test Evidence strength Time Effort Reach for it when
Sales force feedback What people say 3 days low Deals stall and nobody agrees why
Investor interviews and pitches What people say 2 weeks medium You want your advantage stress-tested
Pricing survey What people say 1 week low You have no idea what to charge
Mock purchase intent What people say 1 week low You want to compare two prices
Online market research Early signal 3 days low You need to know if the sums could ever work
Presale interest Early signal 1 week low You want to see if interest survives a price
Funnel test What people do 2 weeks medium You need to know what a customer costs to reach
Referral program Money or a signature 2 weeks medium You want to know if users will vouch for you
Letter of intent Money or a signature 4 weeks medium You sell to businesses and can reach who signs
Crowdfunding Money or a signature 4 weeks high It is a consumer product and you can deliver
Pop-up store Money or a signature 2 weeks high People need to see or touch it before buying

Not sure where to start?

If the idea is still a blur, or nothing here fits, begin with one of these.

Test Evidence strength Time Effort Reach for it when
Lean Canvas What people say 1 day low The idea is unstructured, or your team disagrees
Other Early signal 1 week low None of the 54 methods above fits

What does climbing the evidence ladder look like in practice?

This sequence is made up to show the idea. Imagine you think independent gym owners lose members because they do not notice who has stopped coming.

  1. Five customer interviews confirm owners have tried to fix this. That is what people say, and it costs a week.
  2. A fake door shows owners will leave an email for an offer. Say 40 of 400 visitors from a gym-owners' newsletter do, which is 10%. Now you have behaviour.
  3. A concierge test sends weekly lists by hand to five gyms, and three pay for a second month. Now you have commitment.
  4. Only now do you build, starting with a single-feature MVP. Brief your coding agent with the AI coding agent PRD template, and record why you went ahead in a product decision log.

Using the times in the tables, steps 1 to 3 take about five weeks: one for interviews, one for the fake door, three for the concierge. Each step cost more than the last, and each one was earned by the one before it. If any had failed, you would have stopped early and saved everything after it.

Notice what the sequence skipped. Nobody ran a survey to ask whether gym owners "would use" a churn tool, and nobody built a dashboard to see if it got traffic. Each test answered the next riskiest question and nothing more. For where this sits in a full discovery cycle, see the product discovery process guide.

How many product experiments and participants do you need?

There is no fixed number of experiments. Keep testing until the belief your plan depends on most has evidence strong enough for what being wrong would cost. Strategyzer's summary of the book suggests a practical cadence: run three to five small experiments on your highest-priority assumptions in the first month, and start with the smallest test that could confirm or kill each one (Strategyzer, 2025).

Pair weak and strong. One "say" test tells you what to look for; one "do" test tells you whether it holds. Two "say" tests that agree are still two opinions.

Participant numbers depend on the method:

  • Usability tests: five people per round. Jakob Nielsen's model found five users uncover about 85% of usability problems and 15 find nearly all, so he advises spending the budget on several small rounds (Nielsen Norman Group, 2000).
  • Surveys: aim for about 50 valid answers from one named segment before you trust a percentage. Our discovery survey guide explains why.
  • Split tests: these need real traffic. Kohavi and Longbotham note that a 50/50 split gives the most statistical power once a short ramp-up has ruled out obvious errors. If you lack the visitors, use a fake door or landing page instead. Our guide to A/B testing covers the maths.
  • Letters of intent and presales: a handful can be enough when each one is a real buyer with a budget.

What are the most common mistakes with product experiments?

  • Testing the idea instead of the riskiest belief. "Will this work?" cannot fail cleanly. "Will owners pay €50 a month for a weekly list?" can.
  • Picking the method you already know. Teams that like surveys survey everything. Let the risk choose the method.
  • Treating what people say as what they do. Twenty enthusiastic interviews are still the bottom rung.
  • Writing the pass signal afterwards. A result you judge after seeing it is a story, not a test.
  • Split-testing without traffic. A few hundred visits rarely prove that a change caused a result.
  • Skipping the unglamorous risks. Founders love desirability tests. Cagan's advice is to tackle value and business viability early, and feasibility can sink a loved idea just as fast.
  • Climbing too fast. A crowdfunding campaign before a single interview spends weeks to learn what a conversation could have told you.

When should you not run a product experiment?

Skip the experiment when the decision is cheap and easy to reverse; shipping and watching is the test. Our piece on reversible and irreversible product decisions helps you tell the two apart. Skip it, too, when you will build regardless of the result. A test you would ignore is theatre. And when none of the 55 methods fits your setting, such as a year-long public-sector sale, design a custom experiment rather than forcing one.

Is it ethical to test something that does not exist yet?

Several methods, including the fake door, the 404 test, presales and the Wizard of Oz, show people something that is not fully real. Keep them honest:

  • Tell people on the next screen that the feature is not ready, and thank them.
  • Never take money you cannot refund or deliver on. Say when delivery would happen.
  • If a person secretly does the work in a Wizard of Oz test, tell participants afterwards when it would matter to them.
  • Ask before recording anyone, and only keep the email addresses people chose to give you, for the purpose you stated.

How to run this in Specky

Fill in a Lean Canvas on the Assumptions page, and Specky drafts the assumptions behind it. Each one appears on a board by impact against evidence, with the riskiest numbered. Click a note, pick a test, and the plan fills in with steps, a pass signal, and a time to run.

Who does the work depends on the method. Some tests are AI interviews in Vouch, at vouch.specky.space. For some, Specky builds a page and collects responses. For some, it reads your connected analytics. For the rest, you run the test and Specky keeps the plan and the result. Each guide above says which.

Recording a result moves the note and updates any bet resting on it. One contradicted assumption marks the whole bet as contradicted. A bet only counts as supported when every assumption under it is.

Frequently asked questions

What is a product experiment?

A product experiment is a small, planned test of one belief about a product, with a pass signal defined before you start. It lets a team learn whether something is true before spending real money on it. A good one names the belief, the method, the number it will measure and the result that would count as a yes.

How do I choose the right product experiment?

Write the belief you are least sure about, name its risk (desirability, usability, feasibility or viability), and decide how strong the evidence must be given what being wrong would cost. Then pick the cheapest method that reaches that strength. Early on, conversations are usually enough. As stakes rise, ask people to act or pay.

What are the four types of product risk?

Marty Cagan's four risks are value (will they buy or use it?), usability (can they figure it out?), feasibility (can we build it with our time, skills and technology?) and business viability (does it work for the rest of the business?). Strategyzer's Testing Business Ideas uses three, desirability, feasibility and viability, and its library tags usability tests as desirability.

What is the strongest evidence in product validation?

Money or a signature is the strongest evidence, such as a payment, a pledge or a signed letter of intent. It beats what people do, which beats what people say. Osterwalder's rule of thumb is that evidence gets stronger the closer it comes to a real purchase, which is why a simulated sale outranks any interview or survey.

Should I start with customer interviews or a landing page?

Start with interviews when you are still unsure the problem is real, because you need people's own words before you can write an offer. Start with a landing page when the problem is clear and you can state your offer in one sentence. Many teams run both inside two weeks: interviews first, then a page built from what they heard.

How many experiments should I run before building?

There is no fixed number. Keep testing until the belief your plan depends on most has evidence strong enough for what being wrong would cost. As a cadence, Strategyzer's book summary suggests three to five small experiments on your highest-priority assumptions in the first month. Stop early when one fails clearly.

Sources

  1. Silicon Valley Product Group — The Four Big Risks (Marty Cagan, 2017)
  2. Strategyzer — Testing Business Ideas Experiment Library (poster)
  3. Wiley — Testing Business Ideas: A Field Guide for Rapid Experimentation (Bland and Osterwalder, 2019)
  4. Strategyzer — Testing Business Ideas: Book Summary and Key Takeaways (2025)
  5. Strategyzer — How Strong Is Your Innovation Evidence? (Alex Osterwalder, 2017)
  6. Kohavi and Longbotham — Online Controlled Experiments and A/B Tests, Encyclopedia of Machine Learning and Data Science (2023)
  7. CB Insights — Top reasons startups fail (report, March 2026)
  8. Product Talk — Assumption Testing: Everything You Need to Know to Get Started (Teresa Torres, 2023)
  9. Thoughtworks — How to Implement Hypothesis-Driven Development (Barry O'Reilly, 2014)
  10. Nielsen Norman Group — Why You Only Need to Test with 5 Users (2000)

This story was originally published on 9 November 2026 at https://www.specky.space/blog/how-to-choose-a-product-experiment.

Fathy Shalaby is a scientist by training (master's). He worked at LexisNexis and Canva, and writes How to Be a Product Engineer.