Skip to content
AnthonyTerminal
AnthonyTerminal
Learning Hub

Edge Validation & Statistical Traps

How this system decides an edge is real — and the traps that fool most traders into believing noise

Learning mode: use this track to build a working mental model. For deeper claims, caveats, evidence status, source provenance, and TradingOS implementation links, open the Knowledge Base / Concept Wiki. A concept being catalogued does not by itself make it a validated trading edge.

A backtest is not evidence that a strategy works. It is evidence that a strategy worked on the data you tuned it on. Every parameter you adjust after seeing results — a lookback, a threshold, a filter — quietly fits the strategy to the past.

The overfitting haircut
Academic estimates (Kelly & Xiu, Yale/AQR survey) put the gap between in-sample and realistically achievable performance at roughly an order of magnitude: strategies that look like Sharpe ~2.5 with perfect hindsight deliver ~0.5 once parameters must be estimated from limited data. Expect a large haircut as the NORM, not a sign something went wrong.
Probability of Backtest Overfitting (PBO)
Bailey & López de Prado showed that trying enough strategy variants GUARANTEES an impressive backtest by chance. The more configurations you tried, the higher the probability your "best" one is noise. Always ask: how many variants were tested before this one won?
WARNING: A published claim of "97% accuracy" means almost nothing: a 2022 review of 138 academic ML trading papers found 64% never stated any validation method, only 5 of 138 used proper walk-forward testing, and most ignored transaction costs entirely.
How to read this: The only honest test is data the strategy has NEVER touched, walked forward in time — train on the past, test on the future, never the reverse. This system's backtest engine and shadow-validation harness are built around that rule.