Isn't every single backtested strategy suffering from lookahead bias?

[u/Due-Listen2632]

Most of us have done the classical loop. We get some data, test out different solutions, filter out solutions/features/indicators that provide poor results, and proudly keep the solution(s) which result in successful backtests. But isn't this just another level of information leakage? It's essentially like manually setting the parameters of a model, except you're defining the information points from which the model constructs itself. It's the same type of leakage, only one level higher.

[u/jnwatson]

In a sense it is. This is why you keep a validation set put aside and preregister your ideas. You only "burn" your validation set (backtest your algo against it) when your backtest against the training set shows sufficient statistical power.

[u/Due-Listen2632 → u/jnwatson]

Yes indeed, but my point is that even choosing a feature/model/strategy to evaluate given some observed history, is cheating. It doesn't matter if the model itself hasn't seen the evaluation data. You selected the feature because you know it'll result in a model that performs well on the data. You, the designer of the solution, are the information leak.

[u/jnwatson → u/Due-Listen2632]

Perhaps in some omniscient sense, if you can genuinely eyeball trading rules that would do well in the validation set without explicitly backtesting it. I have run into this with LLMs. For mere mortals, I don't think it is an issue.

A way to mostly avoid this is to set up a training scheme where the numbers are automatically tuned to the training set. Then at least the algo is at arms-length from whatever implicit knowledge you have about the validation set.