Backtesting guide
Backtest Statistical Significance
Backtest statistical significance asks whether an observed performance statistic is sufficiently unusual under a declared benchmark and sampling model. The answer depends on sample size, return dependence, non-normality, and how many strategy variants were tried.
Read the practical guideKey Takeaways
- State the null hypothesis, benchmark, statistic, and sampling unit before testing.
- Trade count alone is not an effective sample size when observations are dependent.
- Skew and heavy tails can make simple normal approximations misleading.
- A selected winner needs an adjustment for the full research search.
Define the claim before the test
Specify what would count as no edge, the benchmark to exceed, and the statistic used for the decision. Also define whether observations are trades, daily returns, or another interval. Changing the benchmark or statistic after seeing the result invalidates the intended error rate.
- Write a null hypothesis tied to the strategy decision.
- Choose a benchmark that reflects the relevant alternative use of capital.
- Set the observation period and evaluation statistic in advance.
Model sampling uncertainty
The uncertainty estimate should reflect serial dependence, overlapping positions, skew, and fat tails when they are present. A long backtest with few independent opportunities may contain less information than its bar count suggests. Probabilistic Sharpe Ratio is one method designed to assess a Sharpe estimate against a benchmark while accounting for sample length, skewness, and kurtosis.
- Inspect the return definition and frequency before choosing a test.
- Avoid treating overlapping observations as independent by default.
- Report an interval or probability with its assumptions rather than a pass label.
Account for strategy selection
Testing many indicators, parameters, markets, or start dates increases the chance of finding an apparently strong result by luck. Record all trials and decide which tests belong to the same research family. Deflated Sharpe Ratio and other multiple-testing methods address parts of this selection problem, but their validity still depends on complete trial records and suitable assumptions.
- Count discarded and failed candidates along with saved strategies.
- Keep confirmation data unavailable during strategy selection.
- Treat statistical significance as one input to a decision, not proof of future profit.
Related Pineify workflow
Start with a clean description of the return evidence
Pineify Backtest Deep Report provides standard risk metrics, trade distributions, rolling windows, and bootstrap scenarios from an uploaded TradingView backtest. Those outputs can support a separate significance analysis.
Inspect the return distribution, drawdowns, rolling behavior, and bootstrap scenarios before choosing a statistical test.
Analyze the backtestBoundary: The report does not run a hypothesis test, calculate a p-value, or correct a selected result for the number of strategy trials.
Primary sources
- Bailey and Lopez de Prado, The Sharpe Ratio Efficient FrontierVerified 2026-08-16
- Bailey and Lopez de Prado, The Deflated Sharpe RatioVerified 2026-08-16
- SEC Investor.gov, Performance ClaimsVerified 2026-08-16
This page is educational and does not provide investment advice. Backtests are hypothetical, depend on their data and assumptions, and do not guarantee future results. Trading can result in substantial loss.