What is a randomization test for a trading strategy?

It compares an observed strategy statistic with statistics recomputed after null-valid transformations of signals, labels, returns, or blocks.

What should be randomized in a strategy test?

The choice depends on the null hypothesis. Signals, labels, signs, timing, or blocks may be transformed, but the rule should preserve every data feature that remains valid under the null.

Is a randomization test the same as a Monte Carlo backtest?

No. Monte Carlo is a broad label for simulated scenarios. A randomization test has a specific null hypothesis, valid transformations, and an empirical comparison rule.

Is permutation testing the same as bootstrapping?

No. Permutation or randomization applies transformations justified under a null. Bootstrap methods draw observations or blocks with replacement to estimate sampling variation.

Backtesting guide

Randomization Test Trading Strategy

A randomization test trading strategy workflow recomputes a selected performance statistic after applying transformations that would be valid under a declared no-edge hypothesis. The observed statistic is then compared with the empirical distribution of randomized results.

Read the practical guide

Key Takeaways

The randomization rule must follow from a specific null hypothesis.
Transformations should preserve time-series features that remain valid under that null.
The complete strategy calculation must be repeated for each randomized sample.
A permutation test and a bootstrap answer different questions.

Define the null and transformation

State what the data would look like if the strategy had no edge. Depending on the claim, a valid transformation might shuffle labels, shift signals, flip signs, or rearrange blocks. An unrestricted row shuffle often destroys autocorrelation, volatility clustering, or market regimes that should remain under the null, so the transformation must be justified rather than chosen for convenience.

Name the statistic and one no-edge hypothesis before randomizing.
List the data properties that the null requires the sample to preserve.
Choose block or constrained transformations when order dependence matters.

Recompute the full strategy

For every randomized sample, rerun signal generation, order rules, sizing, costs, and the selected metric. Use enough repetitions to describe the relevant empirical tail, and set the random seed for reproducibility. If parameters are reselected in the original research, that selection step must also occur inside each randomization.

Keep execution and cost assumptions identical across repetitions.
Store the seed, transformation, and statistic from every run.
Repeat the complete selection process when testing a selected winner.

Interpret the empirical tail

Compare the observed statistic with the randomized distribution and report how frequently the randomized results are at least as extreme. The conclusion is conditional on the null and transformation. Bootstrap resampling estimates sampling variation by drawing observations or blocks with replacement; a randomization test instead applies transformations justified under a null. The two procedures are not interchangeable.

Report the number of repetitions and treatment of ties.
Inspect whether the randomized samples retain the intended dependence.
Add a multiple-testing adjustment when many strategies or statistics were searched.

Related Pineify workflow

Use bootstrap scenarios as a separate diagnostic

Pineify Backtest Deep Report runs 1,000 bootstrap simulations from an uploaded TradingView trade history. This can describe sampling variability alongside a separately designed randomization test.

Backtest Deep Report

Review the uploaded trade distribution and bootstrap scenario range.

Run a deep report

Boundary: The report bootstrap is not a universal permutation or randomization test. It does not choose a null hypothesis, preserve every time-series dependence pattern, or rerun strategy selection inside each sample.

Primary sources

Efron, Bootstrap Methods: Another Look at the JackknifeVerified 2026-08-16
White, A Reality Check for Data SnoopingVerified 2026-08-16
TradingView Pine Script strategies documentationVerified 2026-08-16

Related Resources

Backtest statistical significance Multiple-testing correction Probability of backtest overfitting Backtest Deep Report

This page is educational and does not provide investment advice. Backtests are hypothetical, depend on their data and assumptions, and do not guarantee future results. Trading can result in substantial loss.