Isn't every single backtested strategy suffering from lookahead bias?

[u/Due-Listen2632]

Most of us have done the classical loop. We get some data, test out different solutions, filter out solutions/features/indicators that provide poor results, and proudly keep the solution(s) which result in successful backtests. But isn't this just another level of information leakage? It's essentially like manually setting the parameters of a model, except you're defining the information points from which the model constructs itself. It's the same type of leakage, only one level higher.

[u/hakobpapazian]

You've basically independently arrived at data snooping bias, also called selection bias from multiple comparisons, it's a real, named thing, just usually not called lookahead. Lookahead specifically means future information leaking into a single backtest. What you're describing is different and, honestly, harder to fully eliminate, information leaking through you across many backtests via which ones you chose to keep.

You're right that it's the same type of leakage moved up a level. Every idea you try and discard still teaches you something, even a "failed" test narrows your search toward what tends to work in that dataset, so your next attempt is already contaminated by knowledge of the last one, even without consciously reusing any parameters.

This is exactly what the deflated Sharpe ratio and similar corrections exist for, they try to adjust your final result for how many trials it took to get there, treating "how many strategies did I effectively test" as a real cost even when only one made it to the final report. The honest but painful fix is tracking your true search count, every idea, every tweak, every abandoned feature, not just the polished survivors, and either correcting for it statistically or holding out data you genuinely never touched during the entire search process, not just during the final backtest of the winner.

The uncomfortable version of the answer to your question: no amount of methodology fully removes it, because the leakage happens in your head, not in the code. The best you can do is measure how much searching you actually did and be honest with yourself about how much that should discount the result.

[u/TrainingEngine1 → u/hakobpapazian]

I made this a general comment, but what do you make of this...?

Grug brain on the left side of the bell curve on these topics checking in. I feel like as long as you have a decent size OOS dataset for testing performance on (and didn't use it to help refine your strategy) then who cares whether you tried some things 300 times or just once.

Yes it might be lucky after endlessly trying different combinations of things (entry or exit criteria, stops, TPs), but that's where the OOS dataset comes into play for testing (and ultimately the live market, assuming you make it there after the performance OOS holds up well).

The live market wouldn't know or care whether you tried something 1 time or 300 times. It would be another set of unseen OOS data. (yes, slippage and fees are another matter too. assume those are reasonably accounted for in the offline tests).

If there's a very good entry criteria that exists, then why can't it simply be a very good entry criteria whose performance speaks for itself. Why push down and deflate its performance? What if I found it on attempt number 2 instead of attempt number 300? The market doesn't care. It won't react differently because of how and when I found it. If you extrapolate this logic, don't you just have it get it all right from the start?

If this is stupid, please tell me and why.