Multiple testing / data-snooping

Test one strategy and a win rate a shade above a coin flip looks like an edge. Test a thousand strategies and, by chance alone, several will clear that same bar with room to spare. None of them found anything real. Someone just had to win the lottery. Multiple testing (also called data-snooping) is what happens when that number of attempts gets left out of the significance math. The usual pass/fail line for a backtest, a t-statistic near 2, was built for testing one idea at a time. Run a thousand ideas through that same line and the false positives outnumber the real signals before anyone has even started looking.

The fix isn't a different test. It's a higher bar. The more strategies compete for the title of the one that works, the further out on the tail a result has to sit before it means anything beyond luck finding a needle in a haystack of its own making. A backtest that would have impressed a lone researcher decades ago is unremarkable noise once it's the best of a thousand attempts run back to back.

This is the reason our rejection numbers run as high as they do. We tested 1,000+ strategies and 1,700+ indicators, 2,700+ scripts in total, and a good-looking equity curve is exactly what that scale predicts even if every single one of them were worthless. A single flattering backtest proves nothing on its own; see no real edge for how often the underlying signal turns out to be noise wearing a strategy's clothes. Overfitting is a strategy fitting its own noise; multiple testing is us fitting to the noise of a thousand strategies at once. Same trap, different scale.

We treat every survivor as guilty until it clears a bar sized for the whole search, not one lone backtest. Part of that is running a placebo test against scrambled versions of the same data: if a strategy still looks good on noise, the original result was never about the market. Part of it is refusing to stop at the first promising run and call it finished.

The honest number at the end of that filter is small. 78.4% of the strategies we tested were rejected outright, 14.8% showed a conditional edge under specific conditions, and 0.7% (seven strategies) cleared every hurdle and earned a deployable verdict. That ratio isn't pessimism. It's what happens when you correct for how many chances noise had to fool you.