Data snooping, also called multiple testing or p-hacking, happens when many strategies or signals are tried on the same data and only the best-looking one is kept. By chance alone, some will look impressive, even if none has real predictive power.
A worked example
Suppose a result is called significant at the 5% level, meaning that with no real effect there is a 5% chance of seeing it by luck. Test 20 useless strategies:
| Strategies tested | Chance at least one looks significant by luck |
|---|---|
| 1 | 5% |
| 20 | 1 - 0.9520 = 64% |
| 100 | 1 - 0.95100 = over 99% |
So testing many variations and reporting the best one almost guarantees a result that looks real.
How it happens in practice
- Trying many parameters, such as look-back windows and thresholds, and picking the best.
- Trying many signals, universes and time periods before settling on one.
- Stopping when the backtest looks good and not reporting what was tried.
- Researchers across a whole industry testing the same history, so published factors carry the same problem.
What helps
- Keep a count of everything tested.
- Raise the bar. Statistical corrections such as Bonferroni or the false discovery rate, and, as Harvey, Liu and Zhu (2016) argued, a t-statistic above 3 for new factors instead of the usual 2.
- Test out of sample and walk forward. See out-of-sample testing and walk-forward testing.
- Require an economic reason for the signal.
- Prefer robust, wide plateaus of parameters over one sharp peak.
Data snooping is the main source of overfitting and a key risk in any backtest.
Frequently asked questions
What is data snooping?
It is the practice of testing many ideas on the same data and keeping the best result, which makes chance findings look like real ones.
What is p-hacking?
Another name for the same problem: adjusting analyses until a result becomes statistically significant.
How does multiple testing create false discoveries?
Each test has a chance of looking significant by luck, so the more tests run, the more likely at least one false positive appears.
How can data snooping be reduced?
By counting all tests, applying stricter significance thresholds, testing out of sample and requiring an economic rationale.