Backtest validation and leakage.

Why a good backtest proves little

Try enough variations of anything on the same history and one of them will look good. Bailey, Borwein, López de Prado and Zhu show that high simulated performance is easy to reach after testing a relatively small number of configurations [1]. In a follow-up paper they argue that a single hold-out is an unreliable defence and propose a way to estimate the probability that a backtest is overfit [2]. The same problem has older names: White calls it data snooping and gives a test for whether the best model from a search beats a benchmark at all [4]. Harvey, Liu and Zhu argue that, after hundreds of published factors, a new one should clear a t-statistic of about 3 rather than 2 [5].

The practical point: the equity curve you are looking at was selected. Its quality depends on what else was tried, which a single report rarely tells you.

Three ways leakage gets in

1. Features that look ahead. A centred moving average, a session high read before the session ends, a volume filter that uses the day's total volume at 10:00. These compute fine on a full history and cannot exist at decision time.
2. Fills that look ahead. Deciding on a bar's close and filling at that same bar's open, or assuming the target was hit before the stop when the bar touched both.
3. Data you have already seen. Checking the test period, tweaking, and checking again. No code is wrong. The test period simply stops being out of sample.

The first two are bugs, so they can be caught by code. The third is a habit, so it has to be recorded and counted.

Features: available_at and truncation invariance

Evrant strategies are C# classes with one entry point, OnCompletedBar. The runner calls it once per completed bar, in timestamp order. The SDK exposes no forming bar, data reader, clock or service, so there is no API through which a strategy can ask for the future.

Indicators come from the engine, not from strategy code. Each value is returned with two timestamps, the close of the bar it describes and the time it became available, plus a readiness state (warming up, ready or unavailable with a reason such as an incomplete session). The engine rejects a read whose available_at is after the decision time.

A new feature definition also has to pass a truncation invariance test before it can be used. The engine computes the feature on the full history and again on prefixes that stop at the first bar, a quarter, a half, three quarters and the end. If any value up to the cut changes, the definition is rejected. A centred window fails immediately, because its values near the cut depend on bars after it. The check, lightly trimmed from the engine:

// Rejects a definition if any sampled prefix changes prior values.
var full = Compute(bars, definition);
var cuts = new SortedSet<int> { 1, bars.Count };
for (int quarter = 1; quarter <= 3; quarter++)
    cuts.Add(Math.Max(1, bars.Count * quarter / 4));
foreach (int count in cuts)
{
    var prefix = Compute(bars.Take(count).ToArray(), definition);
    for (int i = 0; i < count; i++)
        if (prefix[i] != full[i])
            throw new InvalidDataException("Feature definition failed truncation invariance.");
}

You can see the same test working on the validation page: switch the example feature from trailing to centred and move the cut point.

Fills: the next bar, not this one

An order produced on a bar's close is evaluated no earlier than the next bar. A market order fills at the next open plus the slippage in your execution profile. A stop that the market gaps through fills at the open, not at the stop price. A limit needs price to trade through it and never fills better than the limit.

When one bar's range contains both the stop and the target and neither was hit at the open, the engine has no way to know which came first, so the stop wins. The full rules, and what one-minute bars can and cannot show, are in the one-minute bars guide.

Splits, embargoes and sealed data

Each strategy has a split plan: development, validation and, when there is enough history, a sealed range. By default it divides the common data coverage 55 / 25 / 20 with a one-week embargo between ranges. You can set boundaries by date or proportion, change the embargo, or turn the sealed range off when history is short. An embargo is an unassigned gap, so the last trades of one range cannot overlap the first of the next.

Ranges are half-open UTC intervals, and the check runs on every job. A development backtest that asks for validation dates is refused with a message naming the plan version and the permitted range, for example: The development range … is outside split plan v2: development runs may use only … Validation … and sealed … history is reserved. Walk-forward uses development and validation only, never sealed.

Validation runs are counted per strategy lineage. A version created after its lineage has seen validation results is marked validation_informed, with the count at that time, and the candidate dossier shows it. Peeking is allowed, but it is never invisible.

Sealed evaluation is a human action on a frozen configuration. The person types a fixed confirmation, I confirm this sealed-range evaluation will expose previously unseen history, and gives a reason. A second use of the same configuration needs a second confirmation, and every sealed exposure is listed in the dossier. The research agent cannot request sealed jobs, and an MCP token cannot be delegated to confirm one.

Machine-learning datasets follow the same idea at row level, after López de Prado [6]. A training row is purged if its label only became known inside the test window, rows inside the embargo after a previous test window are dropped, and a feature that became available after the decision time makes the dataset invalid. The split reports how many rows it excluded and why. Generic tools offer a simpler version of this: scikit-learn's TimeSeriesSplit has a gap parameter that drops samples between train and test [7].

Counting the attempts

Evrant records the number of configurations evaluated per branch and per strategy. The engine uses that count to compute the probabilistic and deflated Sharpe ratios of Bailey and López de Prado [3] on completed-trade returns. It treats every submitted development run as an independent trial, which is conservative, and does not invent an effective-trial correction. It needs at least 30 closed trades, and the result is labelled as supporting development research, not validation evidence.

The candidate decision shows the trial count next to the results, so a 90-trial optimisation is not presented the same way as a single pre-registered test.

The fragile flag

A robustness study changes one thing at a time and reruns the strategy through the same engine. The smallest stresses are one extra tick of slippage, a one-bar entry delay, one tick of adverse fill and a one-step parameter neighbour. If any of these removes profitability or halves expectancy for a profitable baseline, the configuration is flagged fragile automatically. The flag is a warning in the dossier, not a verdict. The thresholds are still being tuned.

What validation cannot tell you

• History is finite. Surviving a validation range says the rules held under those conditions, not that the conditions will return.
• A sealed range is only unseen once. After that it is ordinary validation data, which is why Evrant counts exposures instead of resetting them.
• Statistics such as the deflated Sharpe ratio adjust for the trials you recorded. Ideas you tried elsewhere, or ruled out in your head, are not in the count.
• Bar-based simulation has its own limits, covered in the one-minute bars guide.

A final research disposition is not a promise of profit. Evidence reports end in one of four outcomes: supported under tested conditions, needs work, insufficient evidence, or abandon.

What is built

"In the engine" below means the behaviour exists in the current engine code that runs Evrant’s backtests and studies.

A checklist for any platform

These questions expose common failure modes on any platform:

1. Can strategy code read the current bar before it has closed, or any later bar?
2. If you compute an indicator on data cut at time t, do its values up to t change?
3. Does an order decided at a close fill at that close, or at the next tradable price?
4. When a bar touches both stop and target, which does the simulator assume came first?
5. How many configurations were tried before this one, and is that number written down?
6. How many times has the test period been looked at? Is there any data nobody has seen?
7. Does a one-tick or one-bar change break the result?

To see how these show up in one strategy's record, read the example evidence dossier or the full research journey.

References

1. Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2014). Pseudo-mathematics and financial charlatanism: the effects of backtest overfitting on out-of-sample performance. Notices of the American Mathematical Society 61(5), 458–471.
2. Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2017). The probability of backtest overfitting. Journal of Computational Finance 20(4). doi:10.21314/JCF.2016.322
3. Bailey, D. H. and López de Prado, M. (2014). The deflated Sharpe ratio: correcting for selection bias, backtest overfitting and non-normality. Journal of Portfolio Management 40(5), 94–107. doi:10.3905/jpm.2014.40.5.094
4. White, H. (2000). A reality check for data snooping. Econometrica 68(5), 1097–1126. doi:10.1111/1468-0262.00152
5. Harvey, C. R., Liu, Y. and Zhu, H. (2016). …and the cross-section of expected returns. Review of Financial Studies 29(1), 5–68. doi:10.1093/rfs/hhv059
6. López de Prado, M. (2018). Advances in Financial Machine Learning, chapter 7: cross-validation in finance (purging and embargo). Wiley.
7. scikit-learn. TimeSeriesSplit, including the gap parameter that drops samples between train and test.

Product details come from Evrant’s engine source and product specification as of the review date. Nothing on this page is a performance claim.