The best of a hundred coin flips looks like talent

Take a fair coin. Each step you win one unit on heads and lose one on tails, for 250 steps. That is a strategy with no edge at all, and any result it shows is luck.

Now make a hundred of them, and keep the best.

What luck looks like at scale#

One such strategy ends the 250 steps at zero or thereabouts, give or take about 16 units. Roughly half of them finish above zero. Nothing odd about that. But the best of many is a different animal. This table comes from simulating the coins (the code is below) and shows the median result of the best strategy, over thousands of simulated runs:

A hundred fair coins, picked afterwards, give you a line that gained 38 units over 250 steps with nothing in it but chance. On a chart it looks like a strategy. Add the usual story about why it works and it looks like a good one.

import random

def best_of(n: int, steps: int = 250, seed: int = 1) -> int:
    rng = random.Random(seed)
    finals = []
    for _ in range(n):
        finals.append(sum(1 if rng.random() < 0.5 else -1 for _ in range(steps)))
    return max(finals)

Where the tries hide#

Nobody sets out to test a hundred coins. The tries accumulate:

• Every parameter you sweep is a try. Ten lookback lengths times ten thresholds is a hundred.
• Every rule you add after seeing a bad stretch is a try.
• Every market, timeframe or pair you test and then drop is a try.
• Every idea you abandoned last month but remember is a try.

The number that ends up in the write-up is the winner. The number of tries, which is the thing that tells you how impressed to be, is nowhere.

What helps#

1. Count the tries, the abandoned ones included, and write the count next to the result.
2. Compare with luck, not with zero. Run the same sizing, stops and fees on random entries, many seeds. The result to beat is the best of that many random strategies, not a flat line. Why every strategy has to beat a coin toss does this in detail.
3. Keep data you never touched while searching, and look at it once, at the end.
4. Roll forward. Fit on one window, judge on the next. A strategy that only works on the window it was fitted to has told you nothing. See walk-forward testing.
5. Prefer few knobs. Every knob is another place for chance to hide.
6. Be suspicious of smooth, steep curves. Real strategies have flat patches and bad months.

The lab has the machine: slide the number of strategies up and watch the cyan line climb while every one of the lines has the same nothing inside.