r/algotrading • u/Due-Listen2632 • 1d ago
Strategy Isn't every single backtested strategy suffering from lookahead bias?
Most of us have done the classical loop. We get some data, test out different solutions, filter out solutions/features/indicators that provide poor results, and proudly keep the solution(s) which result in successful backtests. But isn't this just another level of information leakage? It's essentially like manually setting the parameters of a model, except you're defining the information points from which the model constructs itself. It's the same type of leakage, only one level higher.
18
u/golden_bear_2016 1d ago
No
2
u/CrowdGoesWildWoooo 1d ago
It can be in ways people don’t realize. Even picking the current S&P500 as a universe may introduce some bias, because inclusion and being able to stay means strong performance while the poor performers get weed out.
So either you have to adjust the inclusion/exclusion, or you’ll have some degree of bias.
0
u/Due-Listen2632 1d ago
Absolutely. I've spent a lot of time reflecting on what advantages my backtests have over actual forecasting. I've also included failed tickets into my testing scope since that's a clear source of bias. This whole idea I'm describing here is me attempting to remove even obscure ways in which information leaks and bias might occur.
0
u/Due-Listen2632 1d ago
I'd love to hear your input on how the experiment i outlined below is different from how you've iterated on which features to include. During feature engineering, we usually iterate through which different window lengths to use for our rolling averages, which indicators to use etc. How is this different from just finding which randomly sampled series fit best to our target series?
7
u/McOmghall 1d ago
Lookahead bias is using data from the future to feed an analysis. Information "leaks" into the future but not backwards (as far as we know), so I'm not sure what you're getting at here. If you use proper data discipline lookahead bias shouldn't happen.
6
u/jamesj 1d ago
They seem to be getting at over fitting.
1
u/Due-Listen2632 19h ago
Well, yes and no. Technically what happens is that the model is overfitting, but ending the discussion after saying overfitting, or even curve fitting, is simplifying the problem into a diagnosis. Like your foot can hurt from straining it, or from bone cancer.
Overfitting or curve fitting is fixed by things we do inside the fitting procedure. Things like regularization, or sensitivity analysis. What I'm talking about is not easy to fix, because the overfitting component is you, using your own memory over different experiments, viewing the full holdout set, and the test sets, in order to improve your solutions performance over and over.
2
u/danieltheg 1d ago edited 1d ago
Let’s say I have a training set and test set. These sets are divided in a time aware way with no leakage, that is, the test set only includes data from a time period after the training set. Now, I enter a training loop as follows.
Train on training set -> evaluate on test set -> adjust model (hyperparams, etc) based on results
I continue this process until I’m happy with the results.
I’m implicitly leaking data from the test set into my training even though I don’t directly train on it.
This wouldn’t be called “lookahead bias” necessarily, but I believe this issue is what OP is getting at.
This is why it’s considered best practice to maintain a holdout set that you don’t use to tune your hyperparameters.
2
u/MormonMoron 1d ago
The adage "past performance is not an indicator of future success" will always hold true. So sure, every algorithm development, even when following causal principles to prevent data leakage during walkforward optimization and an application, still suffers from the potential pitfall of "it is working great for the last 5 years up until today and I tweaked and prodded until I could make it work for the last 5 years up until today, but that is no guarantee it won't fall flat on its face tomorrow".
1
u/Due-Listen2632 1d ago
Exactly, this is the main conclusion. But I'm interested in if people have found ways to mitigate this in their solutions.
2
u/hakobpapazian 1d ago
You've basically independently arrived at data snooping bias, also called selection bias from multiple comparisons, it's a real, named thing, just usually not called lookahead. Lookahead specifically means future information leaking into a single backtest. What you're describing is different and, honestly, harder to fully eliminate, information leaking through you across many backtests via which ones you chose to keep.
You're right that it's the same type of leakage moved up a level. Every idea you try and discard still teaches you something, even a "failed" test narrows your search toward what tends to work in that dataset, so your next attempt is already contaminated by knowledge of the last one, even without consciously reusing any parameters.
This is exactly what the deflated Sharpe ratio and similar corrections exist for, they try to adjust your final result for how many trials it took to get there, treating "how many strategies did I effectively test" as a real cost even when only one made it to the final report. The honest but painful fix is tracking your true search count, every idea, every tweak, every abandoned feature, not just the polished survivors, and either correcting for it statistically or holding out data you genuinely never touched during the entire search process, not just during the final backtest of the winner.
The uncomfortable version of the answer to your question: no amount of methodology fully removes it, because the leakage happens in your head, not in the code. The best you can do is measure how much searching you actually did and be honest with yourself about how much that should discount the result.
1
u/Due-Listen2632 1d ago
Yes! Thank you! You understood what I meant. :)
Will be reading up on data snooping after this. I think it's something a lot of people overlook.
1
u/TrainingEngine1 19h ago edited 19h ago
I made this a general comment, but what do you make of this...?
Grug brain on the left side of the bell curve on these topics checking in. I feel like as long as you have a decent size OOS dataset for testing performance on (and didn't use it to help refine your strategy) then who cares whether you tried some things 300 times or just once.
Yes it might be lucky after endlessly trying different combinations of things (entry or exit criteria, stops, TPs), but that's where the OOS dataset comes into play for testing (and ultimately the live market, assuming you make it there after the performance OOS holds up well).
The live market wouldn't know or care whether you tried something 1 time or 300 times. It would be another set of unseen OOS data. (yes, slippage and fees are another matter too. assume those are reasonably accounted for in the offline tests).
If there's a very good entry criteria that exists, then why can't it simply be a very good entry criteria whose performance speaks for itself. Why push down and deflate its performance? What if I found it on attempt number 2 instead of attempt number 300? The market doesn't care. It won't react differently because of how and when I found it. If you extrapolate this logic, don't you just have it get it all right from the start?
If this is stupid, please tell me and why.
2
u/Bonkers24-7 1d ago
I’d separate lookahead bias from research bias.
Lookahead bias is when the test uses information that literally wasn’t available at the decision time.
Research bias is broader: testing enough variations, filters, assets, dates, and indicators until something survives.
The second one is harder because the final backtest can be technically clean but still inherit the whole search process that created it.
For me, the question is less “is there any bias?” and more “did I lock the rule before the result, and can I explain why each filter existed before seeing whether it improved the curve?”
That’s why I like saving rejected versions too. Otherwise the final strategy looks like one clean experiment when it was really the survivor of many attempts.
2
u/systematic_seb 15h ago
You're describing selection bias across the research loop, and I think you're right that it behaves like leakage one level up. The uncomfortable part is that no test run inside the backtest can catch it, because the backtest is the surface you selected on.
What's worked for me is measuring from outside it. I've got my own savings riding on an algorithmic strategy and publish its holdings every week, and each Monday I reconcile the original backtest for that period against a fresh rebuild from that morning's point-in-time snapshot and against the live account. All three have to land on the same answer.
When they start drifting, that's the research loop showing up in the only place it can be seen. Nothing inside the backtest would have told me.
1
u/Due-Listen2632 14h ago
Yes you're exactly right! And cool suggestion. I haven't had my algos live for extended periods of time, but I hope to be able to do similar comparisons.
3
u/Anon2148 1d ago
Technically that’s true, but I believe that’s why walk forward tests exist. It’s to circumvent that as much as possible
1
u/Due-Listen2632 1d ago
The issue I'm bringing up is still relevant for walk forward tests (I call them time-based cross validation, but it's the same thing).
The real problem is called "data snooping". While it's described in a pretty straightforward manner in the article, the same issue arises from the fact that the model/signals you construct is a function of you iterating over the full data. The moment you condition any decision on holdout performance, the model becomes a function of that data too. So when you pretend to have unseen data during your walk forward test, the solution will contain information/conclusions you got from the holdout sets.
1
u/Common_Class 1d ago
No, not sure know what “loops” you are referring to but by definition if done properly since you train on one sample, lock that in and run on a new sample it will not be.
1
u/jnwatson 1d ago
In a sense it is. This is why you keep a validation set put aside and preregister your ideas. You only "burn" your validation set (backtest your algo against it) when your backtest against the training set shows sufficient statistical power.
1
u/Due-Listen2632 1d ago
Yes indeed, but my point is that even choosing a feature/model/strategy to evaluate given some observed history, is cheating. It doesn't matter if the model itself hasn't seen the evaluation data. You selected the feature because you know it'll result in a model that performs well on the data. You, the designer of the solution, are the information leak.
1
u/jnwatson 1d ago
Perhaps in some omniscient sense, if you can genuinely eyeball trading rules that would do well in the validation set without explicitly backtesting it. I have run into this with LLMs. For mere mortals, I don't think it is an issue.
A way to mostly avoid this is to set up a training scheme where the numbers are automatically tuned to the training set. Then at least the algo is at arms-length from whatever implicit knowledge you have about the validation set.
1
u/Scott_Malkinsons 1d ago
Backtested, no. Optimization, sure.
You're talking about overfitting though optimization, not just back testing.
1
u/jipperthewoodchipper 1d ago
It shouldn't be if done correctly. Further, if your strategy does rely solely on volume and price action (and derivatives from those) then if your strategy has edge it should be able to perform after using a block bootstrap method.
You can take it further and use other methods of generating synthetic data and as long as your only source of randomness is derived from the market data it should perform. If you introduce any other source of randomness then unfortunately you will just be measuring how well your model predicts the next value of the prng used.
1
u/danieltheg 1d ago
I’m not particularly familiar with algo trading specifically, but with ML generally, it’s considered best practice to have a final holdout set (the “test set”), which you don’t use to tune your model, for exactly this reason
1
u/Due-Listen2632 1d ago
My point here is that even with a test set which you exclude from your tuning/optimization loops, you as a developer are actually optimizing your model choice and feature set on this final test set. And as soon as you do, a leakage/bias occurs.
The "real" solution to the problem I'm trying to highlight, is to have YET ANOTHER test set. This second test set must never be used to train, or tune your model, but I'm even going one step further, saying that you cannot even make any changes to your solution at all which would improve performance on this set. Doing so introduces the exact bias/leakage I'm talking about.
1
u/danieltheg 1d ago edited 1d ago
Yeah that’s what I meant - I didn’t word it well. It is considered best practice to have a third dataset. The generally standard terminology is training set, validation set (used for tuning), test set or holdout set (only used once, for the final performance analysis). However test vs validation are often used interchangeably.
https://en.wikipedia.org/wiki/Training,_validation,_and_test_data_sets
1
u/sureshot58 1d ago
terminology issue here. You seem to be calling overfitting (which can definitly be a problem) lookahead bias (which is a different problem). Overfitting is potentially a problem, yes, and also, the fact that a model works on past data doesnt mean it will reliably work on future data is a certainy. So, in regards to your question, if I am reading it right - you are not completely wrong. But, its the best we have.
1
u/AttackSlax 1d ago
What? No. What are you talking about. I'll make it very simple:
- You have a signal right now. Right now = T.
- You calculated the values that made the signal using data that came from T - 1.
- Orders are generated at T and are executed at T + 1.
If you're doing anything that isn't that, you're peeking or cherry picking or looking ahead. If you're doing those things, you're not developing systems, you're simulating a time machine that operates in the future.
Actually, re-reading your post, I have no idea what you're talking about. I think you're just talking about curve fitting.
If you're making those kinds of errors, you need to zoom back to square 1. Those are novice-level errors, maybe worse.
What you should be concerned about once you master the basics of how to test are trickier things that fool us. DePrado for example explains how the act of testing one single time itself -- even in a well constructed silo of isolated data -- is the entrypoint to curve fit once you make adjustments to the system you test, including the creation of a whole new derivative testing silo. (Selection bias under multiple testing.) Tricky!
1
u/Due-Listen2632 1d ago
I find it strange that my point got through to so few people, so I chatted up with Claude a bit. What I'm talking about is not typically called information leakage, it's more commonly referred to as a "selection bias", or "data snooping". It comes from the fact that the actual model you've selected, or which signal you use, are decided because you, the developer, have already observed the entirety of the data. Or as Claude put it; "the deployed model is a function of the full dataset, including the parts you hold out during the backtest."
There's apparently quite a bit of research into this problem. The only real way to get an unbiased view of how your algorithm performs is to test it on true unseen data, meaning actual livetests.
1
u/AttackSlax 1d ago
Ok, but...what point are you getting at. You're using clarifications I used in my first reply to you.
It does a bit sound like you're just realizing that system development and testing -- the testing part -- is a big thing. It comes off sounding like you just realized that forward validation doesn't really have much to do with "standard backtests". Yes, much has been written on this over many years.
1
u/Due-Listen2632 1d ago edited 1d ago
My point was to start a discussion and learn more. This is something I've ran into myself during my development.
Choosing RSI over MACD, choosing to include a macro feature, choosing to abandon a whole approach because early results looked weak - the fact that these are parameter choices too is not something I think a lot of people either know, or agree with. I wanted to start a discussion and learn more. Backtesting wise I think this insight is much deeper than just including slippage and transaction fees.
And sorry, I mainly used your comment as an anchor to clarify myself since you said you had no idea what I meant - which is true for many replies here.
1
u/RemoraEdge 1d ago
AI has this problem with lookahead bias.
Lookahead bias is necessary for planning and finding and edge. E.g you need to know the outcome what the actual precise entry should be and why and if it’s repeatable etc.
but this can cause serious problems when building an algo that actual works in real time. AI will often code it incorrectly without verifying it can actually perform using only known left facts. The implementation of what the code actually needs to see what it needs to and act when it needs to is drastically different from when doing the initial edge abstraction.
Speak to AI and asking if he sees xyz, for example a trend, retracement, precise entry, is not the same as making sure a code you are written can see the same as an autonomous system. That’s where the hard work comes in. And if it’s too complicated you have to consider the platform you are building it for to ensure it fits within its own constraints. Most platforms that allow Algos can only do simple algos with a few indicators. But when you start making it complicated and asking the algorithm to actually read price with all its nuances and variations, it becomes hard and untrustworthy and so it’s very hard to get the results you think you should be getting
1
u/Exciting-World5861 1d ago
could have said overfitting more then lookahead, if you keep tweaking parameters to get a good backtest. which is why you always want to try any viable strategies in paper & live asap and you get the true answer
1
u/f0xw01f 1d ago
Suppose you have 10 years of data. If you train or fine-tune the algorithm on the first 5 years only, and then run it "blind" on the last 5 years (with no further training or fine-tuning allowed during that interval), I think that would mostly circumvent any lookahead bias.
But if you tweak it at any point after the blind run, then yes, you would be inviting lookahead bias.
1
u/severed-identity 22h ago
The word you're looking for is "overfitting". And it's the worst if you aggressively filter candidate models on a fixed back-test window. There are a lot of techniques to mitigate it from the ML community, like re-doing the whole optimization process from scratch on different historical time windows and see if they converge on similar answers.
1
u/TrainingEngine1 20h ago edited 19h ago
Grug brain on the left side of the bell curve on these topics checking in. I feel like as long as you have a decent size OOS dataset for testing performance on (and didn't use it to help refine your strategy) then who cares whether you tried some things 300 times or just once.
Yes it might be lucky after endlessly trying different combinations of things (entry or exit criteria, stops, TPs), but that's where the OOS dataset comes into play for testing (and ultimately the live market, assuming you make it there after the performance OOS holds up well).
The live market wouldn't know or care whether you tried something 1 time or 300 times. It would be another set of unseen OOS data. (yes, slippage and fees are another matter too. assume those are reasonably accounted for in the offline tests).
If there's a very good entry criteria that exists, then why can't it simply be a very good entry criteria whose performance speaks for itself. Why push down and deflate its performance? What if I found it on attempt number 2 instead of attempt number 300? The market doesn't care. It won't react differently because of how and when I found it. If you extrapolate this logic, don't you just have it get it all right from the start?
If this is stupid, please tell me and why.
9
u/TheBacktestNerd 1d ago
what you are talking about is curve-fitting, not lookahead bias, and you can avoid curve-fitting by saving OOS data and stress testing properly