Backtesting guide
Combinatorially Symmetric Cross Validation
Combinatorially Symmetric Cross Validation, or CSCV, divides an aligned matrix of strategy trials into an even number of ordered blocks and evaluates symmetric in-sample and out-of-sample combinations. Its original use is to estimate Probability of Backtest Overfitting.
Read the practical guideKey Takeaways
- CSCV requires many aligned strategy trials across the same observations.
- Every split uses half the blocks in-sample and the complementary half out-of-sample.
- The in-sample winner is evaluated by its relative out-of-sample rank.
- CSCV is distinct from purged cross-validation and combinatorial purged CV.
Prepare a common trial matrix
Place each candidate in a column and its returns on the same ordered observation rows. Divide the rows into an even number of equal blocks; document any trimming needed. Aggregate profit or Sharpe values for each full backtest are not enough to reconstruct this matrix. Include the research competition rather than retaining only winning trials.
- Align dates and the performance measure across every candidate.
- Include all variants considered in the same selection decision.
- Use blocks long enough to retain meaningful performance observations.
Run the symmetric combinations
Choose half of the blocks in-sample and use the complement out-of-sample, preserving row order within each set. Select the in-sample winner and measure its out-of-sample relative rank. Evaluate all S choose S/2 selections: four blocks produce six selections, including each reversed train/test assignment. Keep the metric and tie-breaking rule fixed. These are symmetric historical splits, not strictly forward-only deployment tests.
- Select a candidate using only the in-sample blocks.
- Record its relative rank on the complementary blocks.
- Retain each split result so the final distribution can be audited.
Distinguish CSCV from related methods
CSCV was introduced to study overfitting through symmetric candidate rankings. Purged cross-validation removes training labels that overlap validation intervals. Combinatorial purged cross-validation constructs multiple purged train-test paths. Similar names do not make these procedures interchangeable, and a CSCV implementation should state whether any additional purging or embargo is applied.
- Use CSCV when the target is the reliability of selecting among strategy trials.
- Use purging when label intervals can leak across train and validation sets.
- Document blocks, combinations, rankings, and any leakage controls separately.
Related Pineify workflow
Generate consistent trials before an external CSCV study
Pineify Strategy Optimizer can compare parameter configurations and export their result rows. For CSCV, separately retain an aligned return series for every candidate; the aggregate optimizer export is not the required return matrix.
Run comparable parameter configurations and inspect their historical metrics.
Open Strategy OptimizerBoundary: Strategy Optimizer does not construct symmetric block combinations, rank in-sample winners out-of-sample, or calculate PBO. An external implementation must supply the aligned observations and document leakage controls.
Primary sources
This page is educational and does not provide investment advice. Backtests are hypothetical, depend on their data and assumptions, and do not guarantee future results. Trading can result in substantial loss.