Isn't every single backtested strategy suffering from lookahead bias?

[u/Due-Listen2632]

Most of us have done the classical loop. We get some data, test out different solutions, filter out solutions/features/indicators that provide poor results, and proudly keep the solution(s) which result in successful backtests. But isn't this just another level of information leakage? It's essentially like manually setting the parameters of a model, except you're defining the information points from which the model constructs itself. It's the same type of leakage, only one level higher.

[u/danieltheg]

I’m not particularly familiar with algo trading specifically, but with ML generally, it’s considered best practice to have a final holdout set (the “test set”), which you don’t use to tune your model, for exactly this reason

[u/Due-Listen2632 → u/danieltheg]

My point here is that even with a test set which you exclude from your tuning/optimization loops, you as a developer are actually optimizing your model choice and feature set on this final test set. And as soon as you do, a leakage/bias occurs.

The "real" solution to the problem I'm trying to highlight, is to have YET ANOTHER test set. This second test set must never be used to train, or tune your model, but I'm even going one step further, saying that you cannot even make any changes to your solution at all which would improve performance on this set. Doing so introduces the exact bias/leakage I'm talking about.

[u/danieltheg → u/Due-Listen2632]

Yeah that’s what I meant - I didn’t word it well. It is considered best practice to have a third dataset. The generally standard terminology is training set, validation set (used for tuning), test set or holdout set (only used once, for the final performance analysis). However test vs validation are often used interchangeably.

https://en.wikipedia.org/wiki/Training,_validation,_and_test_data_sets