Backtesting is running an investment rule over past market data to see how it would have behaved. It is a hypothetical result, never a record of real trading, and its value depends on whether the simulation used only information that was available at each decision date, charged realistic costs, and avoided tuning the rule to the same history it is judged on.
Key terms
- Look-ahead bias: using data in a simulated decision that was not available on that date, such as a company's annual results before they were published.
- Survivorship bias: testing only on assets that still exist today, which leaves out the ones that failed or were taken over.
- Overfitting: adjusting a rule until it fits past noise, so it looks strong in the test and weak afterwards.
- Walk-forward testing: re-estimating the rule at each step with only past data and testing it on the period that follows, then rolling forward.
1. Start with a falsifiable research hypothesis
Backtests should evaluate a clearly stated hypothesis, not search for profitable parameter combinations. Define the economic rationale, expected behavior, and failure conditions before running simulations.
2. Use walk-forward validation as default
A single train/test split is fragile. Walk-forward evaluation repeatedly re-estimates models on rolling windows and tests on unseen future slices, providing better evidence across shifting market regimes.
3. Include implementation frictions
Backtests should include realistic assumptions for every rebalance decision:
- Transaction costs, bid-ask spreads, and market impact proxies.
- Execution delay, slippage, and partial fill assumptions.
- Liquidity and volume participation limits for position sizing.
4. Eliminate data leakage and survivorship bias
Each rebalance must use only information available at that timestamp. Universe construction should include delisted names where possible to avoid survivorship inflation. Bias control is mandatory for credibility.
Survivorship bias is concrete enough to measure. Elton, Gruber and Blake measured it directly by tracking all the funds that existed at the end of 1976, including those that later merged or disappeared, because funds tend to disappear after poor performance (Survivor Bias and Mutual Fund Performance, Review of Financial Studies, 1996).
5. Evaluate distributions, not only averages
Mean returns can hide severe tail behavior. Evaluate drawdown depth and duration, left-tail outcomes, volatility clustering, and dependency on specific market windows.
6. Apply robustness and perturbation tests
Perturb key parameters, shift rebalance dates, and test alternative cost assumptions. A robust strategy should degrade gracefully, not collapse under minor configuration changes.
7. Separate model selection from final evaluation
Use a dedicated validation stage for model selection and preserve a final untouched test period for performance estimation. Re-using evaluation windows for tuning materially overstates expected live results.
8. Maintain an auditable experiment log
Every experiment should record assumptions, input versions, parameter sets, and outcomes. This supports review, reproducibility, and post-deployment learning when live behavior diverges from simulation.
9. Pre-deployment release checklist
- Hypothesis and economic rationale documented.
- Walk-forward and out-of-sample results meet policy thresholds.
- Cost and slippage assumptions stress-tested.
- Bias checks completed and signed off.
- Monitoring thresholds defined for live performance drift.
10. Governance note
Backtesting is a risk-management tool, not a marketing artifact. Decision teams should treat it as one part of a broader investment governance process that includes policy, oversight, and periodic review.
Why a backtest is hypothetical
U.S. regulators treat it that way. In its 2020 marketing rule for investment advisers, the Securities and Exchange Commission states that “the final rule will treat backtested performance as a type of hypothetical performance” (Release IA-5653), because it is calculated after the fact rather than produced by decisions made in real time with real money.
Acubic's backtests are walk-forward, can charge transaction costs on every trade, and use financial data only from the date it was published; the methodology page lists what they still do not model. Past performance, simulated or real, does not guarantee future results, and this guide is not personal financial advice.
Last reviewed