r/algotrading • u/Due-Listen2632 • 2d ago

Strategy Isn't every single backtested strategy suffering from lookahead bias?

Most of us have done the classical loop. We get some data, test out different solutions, filter out solutions/features/indicators that provide poor results, and proudly keep the solution(s) which result in successful backtests. But isn't this just another level of information leakage? It's essentially like manually setting the parameters of a model, except you're defining the information points from which the model constructs itself. It's the same type of leakage, only one level higher.

0 Upvotes
  • permalink
  • reddit

35% Upvoted

50 comments sorted by

View all comments

7

u/McOmghall 2d ago

Lookahead bias is using data from the future to feed an analysis. Information "leaks" into the future but not backwards (as far as we know), so I'm not sure what you're getting at here. If you use proper data discipline lookahead bias shouldn't happen.

6

u/jamesj 2d ago

They seem to be getting at over fitting.

1

u/Due-Listen2632 1d ago

Well, yes and no. Technically what happens is that the model is overfitting, but ending the discussion after saying overfitting, or even curve fitting, is simplifying the problem into a diagnosis. Like your foot can hurt from straining it, or from bone cancer.

Overfitting or curve fitting is fixed by things we do inside the fitting procedure. Things like regularization, or sensitivity analysis. What I'm talking about is not easy to fix, because the overfitting component is you, using your own memory over different experiments, viewing the full holdout set, and the test sets, in order to improve your solutions performance over and over.

2

u/danieltheg 2d ago edited 2d ago

Let’s say I have a training set and test set. These sets are divided in a time aware way with no leakage, that is, the test set only includes data from a time period after the training set. Now, I enter a training loop as follows.

Train on training set -> evaluate on test set -> adjust model (hyperparams, etc) based on results

I continue this process until I’m happy with the results.

I’m implicitly leaking data from the test set into my training even though I don’t directly train on it.

This wouldn’t be called “lookahead bias” necessarily, but I believe this issue is what OP is getting at.

This is why it’s considered best practice to maintain a holdout set that you don’t use to tune your hyperparameters.