Four questions expose most weak backtests in minutes: trial count, costs, sample span, and what chose the parameters. Plus the baseline test every claim must pass.
You do not need a quant degree to audit a backtest. You need four questions, asked in order, and the willingness to walk away when the answers are vague. Most weak backtests fail at least two of these in the first ten minutes.
Question 1: How many trials produced this result?
If a vendor tried 500 parameter combinations and shows you the best, the headline number is partly luck. The more alternatives explored, the less a naive performance statistic means — unless it is adjusted for multiple testing. Honest answers sound like “we tried 12 configurations, here is the correction.” Evasive answers sound like “our AI found the optimal settings.” Count the trials, or discount the claim to zero.
Question 2: Were costs modelled realistically?
Commissions, spread, slippage, borrow costs, market impact. A backtest that books fills at the decision price is testing a market that does not exist. Ask for net-of-cost results specifically. If the edge disappears under realistic friction, there was no edge — there was an accounting choice. Execution friction is measurable, so an honest backtest can include it.
Question 3: Does the sample span enough market conditions?
A strategy tested only on a bull market has not been tested; it has been photographed in good light. Demand results sliced by year and by regime — uptrend, downtrend, high and low volatility. Concentration of returns in one narrow slice is a warning sign, not a track record. And remember: a strategy that looks great on six trades is a small sample wearing a big number. Six trades is a caution flag, not a victory lap.
Question 4: Did the same data choose and validate the parameters?
This is the fastest way to expose a weak backtest. If the full dataset informed the design, the features, or the tuning, then the “validation” on that same data is contaminated — even if it carries a validation label. Parameters must be chosen on training data and judged on data that played no part in choosing them. Purged, embargoed folds are the strict version of this; at minimum, demand genuine out-of-sample periods.
The baseline test (30 seconds)
Ask what the dumbest rule scored on the same data. If a directional model claims 70% accuracy but always predicting “up” scores 68%, the model added almost nothing. On a bare CSV with no price history, two honest baselines exist: always-bullish and no-change. Any claim that cannot beat those has nothing to say, however sophisticated the model sounds.
What a clean audit looks like
A credible result survives this sequence: accuracy recomputed from raw predictions (claimed columns ignored), baselines beaten, walk-forward folds stable rather than decaying, leakage gate passed, trial count disclosed and corrected for. That is the standard BuildIT Index applies mechanically — it returns a verdict and an exit code, so a failing backtest fails a pipeline instead of persuading a meeting.
Print these four questions and keep them beside every pitch deck, vendor demo, and course sales page. They will save you more money than any indicator ever will.
No trial count, no net-of-cost results, no out-of-sample period, no baseline. Any one missing answer is enough to stop the conversation.
This article is educational, not financial advice. Past backtest results do not predict future returns.
Originally published at startbuildit.online
