Walk-forward testing against in-sample optimization

If you tune a strategy's parameters on a stretch of history and then report how well it did on that same stretch, you have measured how well you can fit the past. The number is always flattering and it says almost nothing about what comes next.

The cure is to hold out the future#

Fit the parameters on one window. Judge them on the next window, which the fit never saw. Then roll both forward and repeat. Only the judged windows count toward the result, and each of them was unseen when its parameters were chosen.

from typing import Iterator, NamedTuple

class Fold(NamedTuple):
    train: range
    test: range

def walk_forward(n: int, train: int, test: int, *, step: int | None = None,
                 embargo: int = 0, anchored: bool = False) -> Iterator[Fold]:
    """Fit on `train`, score only on the later `test` window, then roll forward.
    `embargo` skips bars between the two, so a trade opened near the end of the
    train window cannot have its outcome (test-window prices) leak into fitting."""
    step = step or test
    start = 0
    while (test_end := start + train + embargo + test) <= n:
        train_end = start + train
        yield Fold(train=range(0 if anchored else start, train_end),
                   test=range(train_end + embargo, test_end))
        start += step

The embargo is a small detail with a large effect. A trade that opens near the end of the training window can close inside the test window, so its outcome leaks the future into the fit. Skipping a gap of bars removes that.

Decide at the close, fill at the next open#

Walk-forward only works if the simulation itself does not peek. The rule we use in every backtest: the strategy sees closed bars only, and an order decided at a bar's close fills at the next bar's open, never at the price it just looked at.

def backtest(bars, decide, cash: float, fee_rate=0.0006, slippage=0.0002):
    """bars: oldest-first dicts with 'open' and 'close'.
    decide(closed_bars) returns a target position in units.
    A decision taken at the close of bar i fills at the OPEN of bar i+1."""
    pos, target, curve = 0.0, None, []
    for i, bar in enumerate(bars):
        if target is not None and target != pos:
            delta = target - pos
            px = bar["open"] * ((1 + slippage) if delta > 0 else (1 - slippage))
            cash -= delta * px + abs(delta) * px * fee_rate
            pos = target
        target = decide(bars[: i + 1])        # sees closed bars only, never bar i+1
        curve.append(cash + pos * bar["close"])
    return curve                              # the final decision has no next open: dropped

Count what you tried#

If you test fifty variants and report the best, you have not found a good strategy. You have found the luckiest of fifty. The score has to account for how many variants were tried: this is the idea behind a deflated Sharpe ratio. A strategy moves from research to demo to live on evidence that survives that correction, and on nothing else.

One more habit: the backtest and the live system run the same strategy code, and parity tests check that both make the same decisions on the same data. If they do not, one of them is wrong, and you want to find out before the exchange does.