Define what one expensive trial actually measures
For a TradingView rule, let x contain an integer lookback L and an ATR stop multiplier m. An illustrative domain is L = 10 through 60 and m = 1.0 through 3.0 in 0.1 steps: 51 × 21 = 1,071 settings. Keep the entry logic, sizing, instrument, timeframe and fill assumptions fixed. Otherwise the search compares different experiments.
- Choose one objective before searching. For example, J(x) can be the mean of net account returns across three equal-duration development windows, each starting with the same capital. Declare costs, the return denominator and how positions close at window boundaries. This arithmetic mean is not a compounded full-period return.
- Declare eligibility rules separately, such as a minimum trade count and a drawdown ceiling. A run with missing data, an undefined ratio or too few trades is invalid or infeasible under that rule; do not silently score it as zero.
- Fit a surrogate to the observed pairs (x, J(x)). A Gaussian process uses a mean function and covariance kernel to describe how scores may vary across parameter settings. Its posterior supplies a predicted mean and uncertainty for untested candidates.
- Maximize an acquisition function over feasible candidates, evaluate the chosen backtest, and update the surrogate. Integer lengths and rounded multipliers must map to actual distinct settings; cache duplicates rather than spending trials on the same configuration.
Recalculate one expected-improvement choice
This teaching example supplies hypothetical Gaussian posterior predictions; it is not a fitted TradingView experiment. Maximize J, measured in percentage points of mean window return, and assume the best observed score is b = 1.00. For posterior mean μ and standard deviation σ > 0, expected improvement is EI = (μ - b) × Φ(z) + σ × φ(z), where z = (μ - b) / σ. Φ and φ are the standard normal CDF and density. There is no extra exploration offset in this example.
- For B, Φ(-0.5) = 0.308538 and φ(-0.5) = 0.352065. Thus EI = (-0.15 × 0.308538) + (0.30 × 0.352065) = 0.059339 pp. B wins this three-candidate comparison despite having the lowest predicted mean.
- C has the highest predicted mean, but B has more posterior probability of a large improvement. Acquisition value ranks the usefulness of an evaluation under the model; it is not the expected live return of the trading strategy.
- After evaluating B, append its actual score and refit before selecting again. The next choice cannot be inferred from this table without that observation and the updated model. A real search also needs to optimize acquisition over the full permitted domain.
- At σ = 0, use EI = max(μ - b, 0), avoiding division by zero. The displayed formula assumes a noise-free objective. For noisy simulation outputs, use a method that accounts for observation noise rather than treating the highest noisy score as a known incumbent.
| Candidate (L, m) | μ (pp) | σ (pp) | z | EI (pp) |
|---|
| A: (20, 1.5) | 0.90 | 0.10 | -1.00 | 0.008332 |
| B: (40, 2.0) | 0.85 | 0.30 | -0.50 | 0.059339 |
| C: (30, 1.8) | 1.02 | 0.02 | 1.00 | 0.021666 |
Use three data roles with chronological boundaries
The dates below are an example research protocol, not an instruction to reuse an already inspected period. A fixed historical backtest can be computationally deterministic while still being an uncertain estimate of performance in other periods. Running the identical history again does not create another independent market sample.
- Within development, any fitted preprocessing or prediction model must use observations available before its scoring window. Forward splits preserve this ordering. Randomly shuffled folds can train on future observations.
- A date split alone does not remove overlap. Exclude training labels whose outcome intervals extend into the next scoring period. Set any gap from the actual holding or label horizon, not an arbitrary universal number of bars.
- Past bars may initialize an indicator at a later window start, provided the rule only uses information available at each decision. Specify whether positions are flattened or carried across boundaries and attribute P&L consistently. Never use future bars to warm up earlier decisions.
- If the final test disappoints, report that result. Changing the strategy after seeing it consumes the test; a new evaluation requires genuinely unseen data or a separately prespecified outer evaluation process.
| Data role | Example period | Permitted use |
|---|
| Search objective set | 2020 through 2022 | Compute J on declared development windows; fit the surrogate and choose trials. |
| Validation set | 2023 | Apply a prespecified shortlist and selection rule after the search. Any feedback used to change the search makes this development data too. |
| Sealed final test set | 2024 | Evaluate the frozen strategy or frozen selection procedure once; no candidate, bound or metric selection from these results. |
A fixed validation set can become an adaptive objective
Consider this deliberately constructed failure log, separate from the EI table. Each validation entry lists net P&L in USD for the same four quarters of 2023. Every quarter starts with $10,000, uses identical sizing and costs, and closes positions at its end. The analyst keeps changing the search after reading the validation results. These invented numbers illustrate the mechanism; they are not measured backtests or evidence of a typical performance gap.
- The means are (20 - 10 + 15 - 5) / 4 = $5, (20 + 0 + 25 - 5) / 4 = $10, and (30 + 10 + 25 + 15) / 4 = $20. With the fixed $10,000 denominator these are 0.05%, 0.10% and 0.20% mean quarterly returns.
- Suppose R3 is then frozen and the sealed 2024 quarters reveal -$20, $0, $10 and -$10. The mean is -$5, or -0.05%. The validation-to-test gap is $25 per quarter, equivalent to 0.25 percentage points under these assumptions.
- The defect is visible before opening 2024: validation results influenced the next parameter domain. The analyst and optimizer together optimized against 2023. Calling that period out-of-sample does not restore its independence from selection.
- The constructed gap illustrates one possible failure. In real data, a gap alone cannot distinguish selection bias from market change or ordinary sampling variation. Cawley and Talbot explain why optimizing a finite-sample selection criterion can overfit that criterion; adaptive holdout research addresses the same feedback problem.
| Round and decision | Search J (pp) | Fixed validation Q1, Q2, Q3, Q4 ($) | Mean validation P&L ($) |
|---|
| 1: shortlist rule R1; inspect weak quarters | 1.00 | 20, -10, 15, -5 | 5 |
| 2: alter stop bounds after that inspection; select R2 | 0.95 | 20, 0, 25, -5 | 10 |
| 3: alter lookback bounds after another inspection; select R3 | 0.90 | 30, 10, 25, 15 | 20 |
Budget evaluations and freeze the stopping rule
An illustrative budget of 24 parameter evaluations might allocate six initial settings and 18 sequential acquisition choices. With three development windows per setting, that is 24 × 3 = 72 backtest runs before validation. A predefined three-candidate validation shortlist adds three runs, and one final test adds one: 76 runs in this simplified accounting. These counts are a planning example, not a recommended sample size.
- Save trial ID, full parameter vector, data version, symbol, timeframe, dates, costs, script hash, objective, feasibility status, acquisition settings and random seed. Save failed attempts and duplicate proposals too; distinguish attempts from completed evaluations.
- Count restarts, alternative objectives, changed bounds, symbols and manually tested variants in the research history. A fresh optimizer seed does not reset the amount of selection performed on the same data.
- Stop at the declared budget or a rule fixed before search. Small EI describes the surrogate under its assumptions; it does not certify a global optimum or a tradable edge. Increasing the budget because validation looks almost acceptable adds another adaptive decision.
- If validation guides revisions, acknowledge that role and retain a separate final test. For repeated method development, an outer chronological evaluation can assess the entire search procedure: each outer fold reruns all selection using only earlier data. Do not select the procedure by its outer results and still call those results final.
Recognize when the surrogate is a poor approximation
A smooth model can struggle with stepwise trade counts, abrupt stop-trigger changes, categorical rule switches and narrow isolated peaks. More acquisition optimization cannot repair missing costs, lookahead, unrealistic fills or an objective dominated by a few trades. Compare with a simple baseline search under the same evaluation budget and data protocol before attributing any benefit to Bayesian optimization.
- Check neighboring feasible settings and performance by development window. A narrow optimum may depend on one regime; use a parameter-stability criterion declared before inspecting the final test.
- Keep risk and feasibility checks explicit. Maximizing net profit can favor more exposure, and an undefined profit factor with no losing trades is not automatically a superior candidate.
- Posterior uncertainty concerns the surrogate objective. It does not incorporate every source of market uncertainty, data error, execution risk or regime change.
Use Pineify to inspect selected trials, with an external search log
Export the relevant TradingView Strategy Report data for each selected parameter set and preserve the experiment manifest separately. Pineify Backtest Deep Report reads a List of Trades CSV with trade IDs, entry and exit times, prices, quantity, net P&L and excursion fields. Its supported summaries include net profit, profit factor and drawdown derived from closed-trade equity. Reconcile trade counts and net P&L before comparing trials.
- If your TradingView download is a workbook, save its trades sheet as CSV and check headers, dates and numeric columns before import. Closed-trade equity is not a complete intrabar or mark-to-market risk series.
- Calculate the surrogate, expected improvement, trial budget and data-split audit outside the report. This page does not claim Pineify has built-in Bayesian search, adaptive-holdout protection or the independent statistics shown here.
- Pine Script AI Coding Agent can help express the chosen parameter inputs and date rules for review. Validate generated code and reconcile its trades. A working script does not validate the search design or recover an already consumed test set.
