Skip to main content
Blog home
Newsletter
News
Video
Podcast
Reading list
Wright Brief
Sign In

What is Data Snooping (Multiple Testing)?

A plain-language explanation of Data Snooping (Multiple Testing), with a worked example, common pitfalls and the questions people ask most.

Listen to this article
Audio · ~2 min
Add as a preferred source on Google

Data snooping, also called multiple testing or p-hacking, happens when many strategies or signals are tried on the same data and only the best-looking one is kept. By chance alone, some will look impressive, even if none has real predictive power.

A worked example

Suppose a result is called significant at the 5% level, meaning that with no real effect there is a 5% chance of seeing it by luck. Test 20 useless strategies:

Strategies testedChance at least one looks significant by luck
15%
201 - 0.9520 = 64%
1001 - 0.95100 = over 99%

So testing many variations and reporting the best one almost guarantees a result that looks real.

How it happens in practice

  • Trying many parameters, such as look-back windows and thresholds, and picking the best.
  • Trying many signals, universes and time periods before settling on one.
  • Stopping when the backtest looks good and not reporting what was tried.
  • Researchers across a whole industry testing the same history, so published factors carry the same problem.

What helps

  • Keep a count of everything tested.
  • Raise the bar. Statistical corrections such as Bonferroni or the false discovery rate, and, as Harvey, Liu and Zhu (2016) argued, a t-statistic above 3 for new factors instead of the usual 2.
  • Test out of sample and walk forward. See out-of-sample testing and walk-forward testing.
  • Require an economic reason for the signal.
  • Prefer robust, wide plateaus of parameters over one sharp peak.

Data snooping is the main source of overfitting and a key risk in any backtest.

Frequently asked questions

What is data snooping?

It is the practice of testing many ideas on the same data and keeping the best result, which makes chance findings look like real ones.

What is p-hacking?

Another name for the same problem: adjusting analyses until a result becomes statistically significant.

How does multiple testing create false discoveries?

Each test has a chance of looking significant by luck, so the more tests run, the more likely at least one false positive appears.

How can data snooping be reduced?

By counting all tests, applying stricter significance thresholds, testing out of sample and requiring an economic rationale.

Further reading

  1. White, H. (2000). A reality check for data snooping. Econometrica 68(5).
  2. Harvey, C., Liu, Y. and Zhu, H. (2016). ...and the cross-section of expected returns. Review of Financial Studies 29(1).
Browse the full Quant Glossary →

Disclaimer: Investment in securities market are subject to market risks. Read all the related documents carefully before investing. Registration granted by SEBI, membership of a SEBI recognized supervisory body (if any) and certification from NISM in no way guarantee performance of the intermediary or provide any assurance of returns to investors.

The content in these posts/articles is for informational and educational purposes only and should not be construed as professional financial advice and nor to be construed as an offer to buy/sell or the solicitation of an offer to buy/sell any security or financial products. Users must make their own investment decisions based on their specific investment objective and financial position and using such independent advisors as they believe necessary.

Wryght Research & Capital Pvt (Brand name: Wright Research) is a SEBI Registered Portfolio Manager Reg No: INP000007979 (Validity: Apr 03, 2023 – Perpetual) and a SEBI Registered Research Analyst No: INH000017295 (Validity: Jul 03, 2024 – Perpetual), with its registered office at 103, Shagun Vatika Prag Narayan Road, Lucknow, UP, 226001 India and CIN: U67100UP2019PTC123244. Past performance may or may not be sustained in future. Performance provided there in is not verified by SEBI. Investment in securities is subject to market and other risks, and there is no assurance or guarantee that the objectives of any of the strategies of the Portfolio Management Services will be achieved. Registration granted by SEBI, enlistment as RA with Exchange and certification from National Institute of Securities Markets (NISM) in no way guarantee performance of the intermediary or provide any assurance of returns to investors. Please read the Disclosure document carefully before investing. Securities quoted are for illustration only and are not recommendatory. Charts shown are for illustration only. For more information and disclosures, visit our disclosures page here.

Found this useful? Share it.
Wright PMS · Portfolio Management Service

Put this research to work

The same 300+ factor research engine behind this article — applied to a professionally managed portfolio, end to end.

300+
Factors tracked
2L+
Investors
₹1,200+ Cr
Invested
SEBI
Registered PMS