What is Probability of Backtest Overfitting?

PBO estimates how often the in-sample winner from a set of strategy trials ranks below the median of those alternatives on complementary out-of-sample data.

What data does a PBO calculation require?

It requires an aligned performance matrix for the full set of strategy trials across an ordered sample that can be divided into symmetric blocks.

Is PBO the same as walk-forward testing?

No. Walk-forward testing follows a sequence of training and testing windows. PBO uses CSCV combinations to evaluate how reliably in-sample selection ranks out-of-sample.

Does a low PBO prove a strategy is safe from overfitting?

No. It addresses the tested selection process under its candidate, block, and metric definitions. Other forms of leakage, bias, cost error, and regime dependence can remain.

Backtesting guide

Probability of Backtest Overfitting

Probability of Backtest Overfitting, or PBO, estimates how often the strategy selected as best in one part of the data performs below the median of its alternatives in the complementary data. The method is built around Combinatorially Symmetric Cross-Validation.

Read the practical guide

Key Takeaways

PBO evaluates a research process with many strategy trials, not one equity curve.
The input is a performance matrix across trials and ordered data blocks.
Each split selects an in-sample winner and ranks it out-of-sample.
The estimate depends on the candidate set, metric, blocks, and stationarity assumptions.

Build the performance matrix

Collect returns or performance observations for every strategy configuration that participated in the same selection decision. Divide the ordered sample into an even number of blocks. The candidate set must include weak and discarded trials because removing them changes the measured selection process.

Use the same observation dates and performance definition for every trial.
Keep the full set of parameters, models, and variants considered together.
Choose enough blocks to create informative combinations without making each block unusably short.

Apply the CSCV selection test

For each symmetric split, use half the blocks as in-sample data and the complement as out-of-sample data. Select the candidate with the best in-sample metric. Then locate that same candidate in the out-of-sample ranking. Repeating this across the combinations produces a distribution of relative out-of-sample ranks.

Keep the selection metric fixed across every split.
Select only from the in-sample half before consulting its complement.
Record the winner and its out-of-sample rank for every combination.

Interpret PBO in context

PBO summarizes how frequently in-sample selection points to an out-of-sample underperformer within the tested candidate set. A high estimate is evidence that the selection process is unreliable. A low estimate does not prove future performance because the candidate universe, market regime, costs, leakage, or return dependence may still be misspecified.

Report the candidate count, block construction, metric, and full estimate.
Check whether small changes to blocks or candidates alter the conclusion.
Reserve later data for a separate confirmation after research decisions are frozen.

Related Pineify workflow

Keep the strategy trials needed for a separate PBO analysis

Pineify Strategy Optimizer can generate and compare parameter trials with out-of-sample and walk-forward views. Retaining those trial results is useful preparation for an external PBO workflow.

Strategy Optimizer

Run related strategy configurations and review their in-sample and out-of-sample behavior.

Open Strategy Optimizer

Boundary: Strategy Optimizer does not run Combinatorially Symmetric Cross-Validation or calculate Probability of Backtest Overfitting.

Primary sources

Bailey, Borwein, Lopez de Prado, and Zhu, The Probability of Backtest OverfittingVerified 2026-08-16
Bailey and Lopez de Prado, The Deflated Sharpe RatioVerified 2026-08-16

Related Resources

Combinatorially Symmetric Cross-Validation Deflated Sharpe Ratio Parameter stability backtest Strategy Optimizer

This page is educational and does not provide investment advice. Backtests are hypothetical, depend on their data and assumptions, and do not guarantee future results. Trading can result in substantial loss.