Should I choose Holm or Benjamini-Hochberg?

Choose the error target before seeing the results. Holm controls the probability of any false rejection within the family. BH controls the expected false-discovery fraction under its dependence conditions. Choosing whichever retains more attractive signals after the analysis changes the research procedure.

Does a BH-adjusted p-value of 0.03 mean a 3% chance the signal is false?

No. It is a threshold for the BH rejection procedure on that family. It is neither a posterior probability that this signal is false nor a probability of losing money in live trading.

Can I correct only the signals with raw p-values below 0.05?

No. Selecting that subset after inspecting results discards tests that belong to the family. The example uses all five hypotheses even though D and E have large raw p-values.

Is multiple hypothesis testing the same as Deflated Sharpe Ratio?

No. Bonferroni, Holm and BH operate on a declared family of hypothesis p-values. DSR evaluates a selected Sharpe estimate against a benchmark adjusted for a modeled search history. A DSR value is not an interchangeable input to this table.

TradingView Backtesting

Multiple Testing for Trading Signals: Families and Error Rates

Multiple hypothesis testing accounts for the false discoveries that can arise when several trading signals are tested on the same data. Define the hypothesis family before inspecting results, obtain valid raw p-values, then choose an error rate: FWER limits the probability of any false rejection; FDR limits the expected fraction of false rejections among all rejections. These controls answer different questions and require explicit assumptions.

See the decision guide

Define one family around one research decision

Suppose the decision is which of five fixed signals to carry into confirmation testing on one instrument. Register signals A through E before evaluating them on the same dated sample. For each signal j, define H0,j: its expected net excess return is at most zero, against the one-sided alternative that it is positive. Specify the benchmark, costs, return frequency, test statistic and sampling method before computing any p-values.

A family is the set of hypotheses covered by one error-control claim. If signal thresholds, holding periods, instruments or outcome metrics were searched to reach that decision, five saved finalists do not describe the full family.
Record code revision, parameters, symbol, timeframe, sample dates, benchmark, costs, hypothesis direction, raw p-value and decision for every evaluated candidate. Preserve failed runs with reasons; a missing result is not a p-value of zero.
Do not split a family into convenient subgroups after seeing the results. Separate families need a defensible, prespecified interpretation, and separate 5% controls do not create a combined 5% guarantee.
If the search history is missing or the hypotheses evolved after inspecting the same outcomes, report the limitation. Merely guessing a larger trial count does not establish the validity of adaptively selected raw p-values. Freeze the resulting hypotheses before using untouched confirmation data.

FWER and FDR control different errors

Let V be the number of rejected null hypotheses that are actually true, and R the total number rejected. Define FWER = P(V ≥ 1) and FDR = E[V / max(R, 1)]. The fraction is zero when nothing is rejected. Both are properties of a repeated testing procedure under its assumptions, not probabilities assigned to a particular trading signal.

FDR = 5% does not mean each discovery has a 5% probability of being false. It also does not guarantee that at most 5% of this particular set is false, or that exactly one of twenty discoveries is false.
If all null hypotheses are true, every nonempty rejection set is entirely false, so FDR equals FWER. With some real effects present, FDR allows a different error tradeoff.
For illustration, five independent true-null tests each run at an exact 5% size have unadjusted FWER = 1 - 0.95^5 = 22.6219%. That formula requires independence; signals sharing price history generally do not justify it.

Target	Meaning at 5%	Research use
FWER	Probability of at least one false rejection in the family is at most 5%	A small set of confirmatory claims where any false discovery matters
FDR	Expected false-discovery fraction is at most 5%	A discovery screen whose retained candidates still need confirmation

A sorted five-signal calculation

Assume the five registered signals produce the raw p-values below. These are invented inputs for checking the adjustment arithmetic, not p-values estimated from real returns. Let m = 5 and set both the FWER target α and the FDR target q to 0.05. The BH interpretation additionally assumes its required dependence condition holds. Values are sorted ascending; retain signal IDs when returning results to the original order.

Bonferroni: adjusted p(i) = min(1, m × p(i)). Its raw threshold is 0.05 / 5 = 0.01, so reject A only.
Holm: adjusted p(i) = min(1, max over j ≤ i of [(m - j + 1) × p(j)]). The running maximum is necessary: each adjusted value must be at least the preceding one.
The Holm raw thresholds are 0.010000, 0.012500, 0.016667, 0.025000 and 0.050000. A and B pass; C fails. Stop at that first failure and retain C, D and E. Reject A and B only.
BH: adjusted p(i) = min(1, min over j ≥ i of [m × p(j) / j]). Compute the running minimum from the largest rank backward. These are BH-adjusted p-values, not estimates of each signal's probability of being false.
The BH raw thresholds i × q / m are 0.01, 0.02, 0.03, 0.04 and 0.05. The largest passing rank is k = 3 because 0.028 ≤ 0.03. Reject every hypothesis through k: A, B and C. BH does not stop at the first failed rank.
Compare full-precision adjusted values with the chosen level using ≤. The displayed 0.046667 is rounded from 0.14 / 3. Do not round before deciding.

Rank i / signal	Raw p(i)	Bonferroni adjusted p	Holm adjusted p	BH adjusted p
1 / A	0.004	0.020	0.020	0.020
2 / B	0.012	0.060	0.048	0.030
3 / C	0.028	0.140	0.084	0.046667
4 / D	0.200	1.000	0.400	0.250
5 / E	0.600	1.000	0.600	0.600

Reproduce the table in a spreadsheet

Enter ranks 1 through 5 in A2:A6 and the sorted raw p-values in B2:B6. Store the family size 5 in H1. The formulas below produce columns C, D and E. Validate that every p-value is numeric and between 0 and 1 and that H1 matches the declared complete family. A blank or failed test must not silently shrink that family.

For a second implementation, R provides p.adjust(c(0.004, 0.012, 0.028, 0.200, 0.600), method = "holm"). Replace "holm" with "bonferroni" or "BH" to reproduce the other columns.
For m = 1, all three methods return the original p-value. Adjusted values stay in [0, 1]; Bonferroni values above 1 are capped. Ties require the same running-maximum or running-minimum rules, not arbitrary different decisions for equal p-values.

Column	Formula	Fill instruction
Bonferroni	=MIN(1,$H$1*B2)	Fill C2 down through C6
Holm first row	=MIN(1,($H$1-A2+1)*B2)	Enter in D2
Holm subsequent rows	=MIN(1,MAX(D2,($H$1-A3+1)*B3))	Enter in D3; fill down through D6
BH last row	=MIN(1,$H$1*B6/A6)	Enter in E6
BH earlier rows	=MIN(1,E6,$H$1*B5/A5)	Enter in E5; fill upward through E2

Distinguish cross-signal dependence from invalid raw tests

Signals computed on the same prices may share trades, market exposure and losses. Bonferroni and Holm allow arbitrary dependence across valid tests. Standard BH controls FDR under independence or the positive regression dependence condition on the true nulls (PRDS). A positive sample correlation between two strategy returns does not by itself establish PRDS for their test statistics.

If arbitrary cross-test dependence must be allowed, Benjamini-Yekutieli (BY) uses q / Hm in the BH rule, where Hm = sum from i = 1 to m of 1/i. It is more conservative. For five tests, H5 = 2.283333 and q / H5 = 0.02189781; this example then rejects A only.
All these guarantees presuppose valid marginal null p-values: for each true null, P(p ≤ u) ≤ u for every u in [0, 1]. Merely storing a number between zero and one is insufficient.
Serial correlation, overlapping holding periods and volatility changes can invalidate a naive independent-observation t-test. A tiny p-value from that test remains unjustified after multiplying it by five. Specify a time-series inference method appropriate to the null, dependence and sample before applying the multiple-testing adjustment.
A dependence-preserving bootstrap may be appropriate for some null models, but shuffling individual daily returns or trades destroys their ordering. Block choice, stationarity and null construction still need justification; no adjustment fixes future information in the signal or unrealistic execution costs.
Keep the actual family size m in these procedures. Replacing five correlated tests with a guessed effective count changes the guarantee. DSR has its own trial-model assumptions and is a separate analysis of a selected Sharpe ratio.

Carry the research record alongside TradingView exports

TradingView documents separate trade-list and summary CSV exports, and its current strategy documentation also describes XLSX report downloads. Export every candidate with its settings and reconcile the time range, trade count and costs. A winning trade file alone contains neither the complete hypothesis family nor valid p-values for the discarded signals.

Backtest Deep Report accepts a TradingView trade CSV and reports profit, drawdown, Sharpe and other trade statistics. If the export is a workbook, prepare a compatible trades CSV and verify its parsed rows before analysis.
The current report does not implement Bonferroni, Holm, BH or BY corrections. Keep the family ledger, raw-test method and adjustment worksheet as separate research artifacts. A report score or Sharpe ratio is not a raw p-value.
Pine Script AI Coding Agent can help draft a specified signal or return-recording rule. Review and validate the generated script; it cannot recover unlogged searches or certify that a statistical null model fits the returns.
After screening, freeze the candidate definitions and confirmation protocol before inspecting new data. Report net effect size, uncertainty, costs and failures alongside the corrected decisions. Statistical rejection alone does not establish an executable or profitable strategy.

Where Pineify fits

Backtest Deep Report

Inspect supported trade and risk metrics from a TradingView trade CSV. Keep hypothesis testing and multiple-testing corrections in a separate, documented calculation.

Also useful: Pine Script AI Coding Agent. Draft a signal from frozen rules, then review and validate its implementation before collecting results for the declared hypothesis family.

Explore Backtest Deep Report

Related resources

TradingView Backtesting Guides Deflated Sharpe Ratio Backtest Statistical Significance Selection-Aware Backtesting Methods

Frequently asked questions

Educational research methods, not investment advice. The p-values are hypothetical teaching inputs. Error-rate controls depend on valid tests and stated assumptions; they do not guarantee future returns.

Sources and verification

Last verified: 2026-10-18

Published by Pineify. Pineify is an independent product and is not affiliated with TradingView. Sources support the technical explanations; examples do not establish trading performance.