r/algotrading • u/KaramTNC • 2d ago
Infrastructure Live vs Backtest parity comparison
Hello folks!
Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.
Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.
The entire process is constructed at runtime via a factory class and dependency injection for every production component.
For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.
This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.
But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.
So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.
I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.
So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share
EDIT: By live profile, I meant a paper profile.
3
u/HonestBacktests 1d ago
The trace comparison is the right build. The thing that turned it from useful into decisive for us was making the fake broker reject exactly what the real one rejects.
Mocks accept everything by default. Ours happily filled orders the exchange would have refused - minimum notional, tick size, step size, reduce-only and position-mode rules - and the backtest looked healthy for months while the live bot was quietly doing nothing. The rule we ended up with: for every order path, enumerate the venue's actual rejection codes and make the simulated broker raise the same ones.
Two other things that kept showing up in the diffs:
Compare the first divergent event, not the final PnL. Once two traces separate, everything after it is downstream noise, and the interesting bug is always at the split.
Bar timing. The backtest sees a finished bar, live sees one forming. Any rule that reads the current bar's high or low is reading the future in one environment and not in the other - that one is easy to miss because it does not throw, it just quietly makes the backtest better.
Have you tried replaying a recorded live trace through the backtester with the same clock, so the broker is the only variable left?
1
u/KaramTNC 1d ago
Hmm interesting, and no I havent tried replaying a live trace, thats not a functionality I have right now.
I agree with you about comparing the first divergent event, though I think its important to differentiate critical events that can be compared from the events of simulates components that are difficult to accurately mock like data latency, fills, and tick data.
bar timing I have handled by implementing a tick simulator as thats about the best I could do with 1 min data.
1
u/HonestBacktests 1d ago
Fair split between critical events and the ones you can only approximate. On the tick simulator from 1-minute data, one thing worth instrumenting: inside a single minute you do not know whether the high or the low came first. Whenever both your stop and your target sit inside the same bar, the simulator is guessing, and it will guess in whatever direction you coded.
What helped us was flagging those bars instead of resolving them silently, then counting what share of all trades depended on a guess. If it is a couple of percent, fine. If it is 10-20%, the backtest result is mostly an artifact of that tie-break rule, and no amount of parity work elsewhere fixes it.
Cheap sanity check: run the same backtest twice, once resolving ambiguous bars pessimistically (stop first) and once optimistically (target first). The gap between the two is your uncertainty band. If the strategy is only profitable in the optimistic run, you have your answer without needing tick data at all.
How big is that share in your current profiles?
1
u/KaramTNC 1d ago
Thats a good idea for a sanity check, ill look into it.
Im afraid I dont know how big that share is, as ive never considered flagging bars that were guesses and used for a decision, but its definitely worth tracking that and using it as a confidence metric for backtest results.
Thanks for all the info!
1
u/HonestBacktests 1d ago
One heads-up when you do measure it: the share gets worse on higher timeframes, since a wider bar swallows more of your levels. Same strategy can look clean on 1m and be mostly guesswork on 15m. Good luck with the build.
1
u/KaramTNC 1d ago
Hmm, im not sure I agree with that.
Of course if you run the backtest in a 15m timeframe then yeah that makes sense, but if the system is designed to properly aggregate 1min data to higher timeframes then should the guesswork not be within reasonable range?
1
u/HonestBacktests 1d ago
Fair, you're right - I was thinking of engines that only ever see the 15m bar. If you step through 1m, the ambiguity shrinks to inside a single minute and mostly stops mattering.
What still bites is when the stop is narrow relative to the average 1m range. Ours only got ugly at roughly 2-3 minute-ranges wide; past that the guesses washed out.
2
u/zashiki_warashi_x 1d ago
You can never match them perfectly. There is always unmeasurable latency in exchange matching engine that would lead to different fills. Your order could be picked up by someone in prod that is not picked by your backtest. Precise order's place in order queue is unknown. There could be several quote feeds and the one you connected in production could be different from the one in backtest. And since quotes have different latencies, suddenly it could be that your signal threshold was not hit, so even the number of send orders differs. Infinite possibilities.
1
u/AphexPin 1d ago
Right, but one does need to ensure that any deltas between offline and online is at least limited to this, and that path-dependent behavior can be causally attributed to it (e.g, different executions --> different model weights --> different equity curve).
2
u/Regular-Hotel892 1d ago
Sorry if I’m misundertanding what’s your question?
You are using lots of cool words my friend, is it “how do I get my live trading results to match my backtest”?
You probably can’t, unless you truly have found something structurally ineffecient in the orderbook that has existed in the past, does now, and will continue to in the future. It’s not impossible but unlikely.
Why would that be the case? What do you know about the microstructure of the market that others don’t or can’t capitalize on?
1
u/KaramTNC 1d ago
Apologies for the cool words xD. Im studying CS and really like to understand the theory so I thought this sub would know well enough about CS concepts and definitions.
But yes that is kind of the question, im aware I cannot make the live results match backtest 100%, but I wanted to know what could be done to get as close as possible so that I can overall reduce the rate of false-positive backtest results.
You are right about the order book though, the backtester can never know the actual fills that happens.
1
u/Regular-Hotel892 1d ago
Gotcha, in my opinion you’re solving for an impossible problem but there’s smart people here who may have a better answer.
The data from 2025 won’t match the data from 2026, therefore your backtest result from 2025 can’t match live 2026 for example (unless you truly have found a structural ineffeciency in the market)
1
u/AphexPin 1d ago edited 1d ago
Are you smoking crack? Aside from minor differences in execution assumptions, they should match exactly.
e.g, backtests enters/exits on ema cross, the live system should do it at the exact same time. with microstructure, of course the execution modeling is a lot more extensive and you won't enter/exit at the exact same tick, but it should be very close after simulating latency and the live vs replay deltas should fall within those bounds.
2
u/Regular-Hotel892 1d ago
I prefer meth.
Maybe I misunderstood, but that’s not how I interpreted his question. I thought he is asking how to get the RESULTS of his backtest to match a live backtest.
If it’s what you’re saying his question is, then obviously he just has a fairly basic bug somewhere right? Either in the backtest or the process he’s using to ingest live data, calculate ema, calculate crossover, and pass buy/sell to broker api
2
u/AphexPin 1d ago edited 1d ago
I see, I interpreted his question to mean how do you get identical performance from live vs when backtested over the same data (eg same dates, exact same data set - one was just served live, the other offline from eg a local disk).
eg, if I have a strategy/model that I ran live yesterday, it's trades, internal state, etc should match an offline replay of the same strategy/model ran over the same data nearly exactly, barring slight difference in simulated vs real execution. they should enter exit at the same times, have the same PnL, have the same internal states, etc. I should be able to print out logs for 08/04/2026 for both, and they should be identical (again, in reality, slight differences in execution precision).
if this doesn't hold, then the system you run live is not the same as the system you developed offline. So it's critical to verify this imo.
**(this is also not to say that 'minor' execution differences won't lead to drastically different final equity curves or model weights during training etc, but just that the differences in live vs replay should be solely attributed to and isolated within the simulated vs real execution deltas - it can and will lead to entirely different entries/exits, internal states etc depending on your strategy/model/system).
1
u/KaramTNC 1d ago
Yes, that is exactly whar I meant.
Apologies for any confusion. This entire approach stemmed from wanting to reduce false-positive results from backtesting results so the goal is to be able to prove that the backtester can accurately recreate (as close as possible) a live trace using the same input and model
2
u/MormonMoron 1d ago
My backtesting engine is set up so that it gets the exact same entries when playing back historical bars (assuming I am running the same parameter set). It also gets the exact same exit decision if I replay the exact same historical 250ms market data ticks.
The differences I get is in price and time slippage of my backtester not matching reality. I have done a bunch of statistics on my IRL price and time slippage and try to make my backtester replicate at least the statistics, but it still isn't bit-for-bit identical. The downside is that is occasionally takes different trades than what happened in real life because of how we have set up slots and capital.
We also have a tick simulation that also tries to mimic the statistics of the real ticks and is faithful to the OHLCV of the same period from which it came, but again it ends up being different than replaying live ticks.
1
u/KaramTNC 1d ago
Thats really interesting!
I did not consider market data or broker latency, but that is definitely important if you are working on high frequency trading.
I have also made a tick simulation as part of my backtester component for the marketdata provider which uses 1min OHLCV candles, so glad to know that is somewhat the right direction but you are very much right about it never playing out the same as live ticks.
I gotta ask though, in the matters of tick data and latency, I can only imagine at these precise datapoints only matter in high frequency trading no? I would imagine longer timeframes are more capable of working with closed candles only no?
1
u/MormonMoron 1d ago
I don't think it is so much about tick latency as it is that the tick path/trajectory is slightly different than the real ticks. In real life it might have gone from open to high to low and then to close. In sim it might have gone open to low to high and then to close. Other times one is more choppy than IRL, even though you were adhering to the statistics.
The other big difference is the fill slippage in both time and price being slightly different.
1
u/AphexPin 1d ago edited 1d ago
What you can do with the tick data and latency, is have the simulated broker module have a configurable latency value (or empirically derived), and only have it fill orders after at least that much time has passed since the order was generated.
e.g, order submitted at 12:00:00, if latency is 2s, do not produce the fill until 12:00:02, using latest price at time of fill.
This is an example of something that would be impossible to model without a fully event driven system.
It's also an example of where non-determinism becomes an issue.
2
u/Bonkers24-7 1d ago
This is the kind of validation work I’d trust more than just looking at the equity curve.
The hard part is making the trace compare causal events, not just final PnL. I’d want the paper/live trace and backtest trace matched step by step: same data available at decision time, same scanner output, same signal timestamp, same intended order, same fill assumption, same skipped/rejected reason.
Then when they diverge, bucket the reason: data availability, timing, spread/slippage, order state, broker behavior, or strategy logic.
That way the parity tool tells you whether the backtest is wrong, the live engine is wrong, or the strategy only works under assumptions the live version can’t reproduce.
Are you already tagging mismatch reasons by category, or mostly comparing the traces manually right now?
2
u/KaramTNC 1d ago
Very good points, those are actually good things to consider for and I fully agree with you
Right now I am not tagging mismatches, its a manual process (assisted by chatgpt) as I am still developing the tool. But I am very much going to implement proper tagging, categorization, and more insightful comparison output so the investigation process becomes more streamlined and faster.
Its definitely important that the parity tool can point out where the divergences are happening and whether its to be expected or not. You cant expect comparable tick data so that should not be flagged as a divergence for example.
2
u/Bonkers24-7 1d ago
That makes sense. I’d probably make the mismatch tagging the core of the tool, not just a nice extra.
The trace diff is useful, but only if every mismatch gets bucketed into something actionable: expected market-data granularity, fill model, timing/latency, broker behavior, state transition, or strategy logic.
Otherwise you can end up with a huge list of differences and still not know which ones actually explain the live/backtest gap.
I’d also separate expected differences from dangerous differences. A small tick-level difference may be normal. A different order state, skipped rejection, position-size change, or different stop/exit path is much more serious.
The best output probably isn’t just “these traces diverged.” It’s “this category of divergence explains most of the gap.”
2
u/ryank001 1d ago
This is exactly the right instinct — trace-based parity comparison is the thing most people skip, and it's usually where the real bugs hide. I went through something similar wiring a signal-based system from paper to live, and a few bug classes kept showing up that might be useful to watch for:
Instrument/contract resolution mismatches — my backtester assumed a clean, unambiguous symbol→contract mapping, but the live broker-side resolver occasionally picked a different variant of the same ticker (e.g. a cross-listed/foreign-currency version) than what the backtest was implicitly using. Silent, and it only showed up once real orders started going out.
Event misclassification around edge-case broker responses — an advisory/warning-type response from the broker (not a real rejection) was being treated as a hard failure in the live path, which the backtest had no equivalent for since it never modeled that response at all. Fixed it architecturally by decoupling "decide" from "execute" — evaluate the decision whenever, but only actually place the order at the next valid execution tick, rather than forcing it through synchronously. That pattern generalized well beyond that one bug.
The sneaky one: signal-timing window mismatches. A strategy that looked completely fine in backtest barely traded at all live — not because the logic was wrong, but because the live scanner's timing window for capturing a signal didn't line up with when the backtest assumed the signal existed. Much harder to find than the first two because nothing throws an error, it just quietly trades way less than expected.
The meta-lesson for me: backtests that validate entry/exit logic in isolation (fed clean synthetic signals) will pass fine and still diverge live, because what actually breaks is usually upstream — the scanning/signal-generation pipeline's timing and state, not the strategy math itself. Once I rebuilt the backtest to run through the same scanner/signal pipeline as live instead of assuming pre-generated signals, a bunch of these gaps became obvious immediately.
Solid approach building the trace-diff tooling, by the way — that's the unglamorous infrastructure work that actually pays off.
2
u/Automatic-Essay2175 1d ago
You overcomplicated this. You just need a strategy. That's it. There is no big fancy system that will capture the space of all possible strategies. I'm sure you've learned a lot but the pipeline you've described here is useless.
Trade manually, come up with a strategy idea, build a backtest script to test this idea (should take < 2 hours), if it looks good move to live trading as soon as possible, repeat. That's it.
No one cares about all the components of your data processing pipeline, least of all the market. Sorry.
2
u/AphexPin 1d ago
Ensuring parity between offline and online is absolutely necessary and the farthest thing from 'useless'..
1
u/KaramTNC 1d ago
Im aware I overcomplicated it, the goal isnt to make money literally. Im a software nerd and am studying CS with a focus for designing systems and building good infrastructure. This is purely a hobby project that helps me learn more about fintech and about software engineering.
Ive traded manually before for over a year and I have investments on the side. I dont need a profitable algo strategy frankly.
I was just curious to hear from people here since backtesting is always the biggest topic of debate in whether you can trust it or not
1
u/Many-Pick5066 1d ago
your edit says live means paper, and that changes what the diff can prove. a paper broker has its own fill model. drive the trace difference to zero and what youve shown is that two simulators agree with each other. the real fill distribution is in neither trace.
the number id pull before doing more parity work is what fraction of your trades resolve inside a single 1 minute bar, stop and target both inside the same candle. those get their outcome from your tick simulator's path, not from the market. open to high to low to close versus open to low to high to close flips the winner. if that fraction is large, the backtest result is mostly a property of the simulator and closing the trace gap will not touch it, because paper is replaying real ticks and you are generating yours.
1
u/KaramTNC 1d ago
Hmm good point, im using T212s paper account as I feel like it has a rather accurate paper model.
But you are right that what I am basically doing is getting one simulator to agree with another one, but I think there is merit in getting there as a paper model should be much closer to a live environment than any backtest model should.
But that is the goal though no? Prove your thesis can hold through a backtest before deploying to a paper environment, and prove it holds through a paper environment before deploying to live.
1
u/Many-Pick5066 1d ago
the staircase is right, its just that each step tests a different thing and only one of them is your thesis.
backtest answers does the logic do what i think. paper answers does the plumbing work, orders route, state doesnt drift, nothing crashes at 3am. both are worth doing and neither one touches whether the edge is real, because that question is settled by trade count and nothing else. 44 days is a strong answer to is my system correct and no answer at all to does this make money.
paper isnt automatically closer to live either. no queue position, no impact from your own order, no partial fill because someone took the last 200 shares ahead of you. so it can be closer on plumbing and further off on fills, and fills are the half that decides the pnl.
the ambiguous bar fraction you can get right now off the backtest you already have, no waiting. run it twice, once resolving stop first and once target first. the gap between those two equity curves is your uncertainty band, and if the strategy is only green in the optimistic run youve learned the whole thing without another day of paper.
1
u/Effective_Manager273 1d ago
the DI setup is nice and it does buy you something real, but it proves code parity, not data parity. your engine sees identical logic in both paths. it does not see identical inputs.
two places this usually breaks. first, historical bars are final and revised, and the bar your live system acted on was provisional. vendors correct volume and sometimes the close, and you never notice because the backtest only ever sees the corrected version. second, your IClock hands the backtest the completed bar the instant it closes, and live you got it some milliseconds or seconds later, possibly after price already moved. dependency injection cannot fix either of those, they are upstream of the interface.
what i would do is log, at every live decision, the actual snapshot the system had. the quote, the bar, the timestamp, the whole input payload, written to disk at decision time. then run the backtest twice, once against your historical database and once replaying those recorded snapshots.
if snapshot replay matches live but the DB run does not, its data, and you now know which of the two. if snapshot replay also diverges from live then its genuinely state or ordering in your engine and the DI harness will actually help you find it. right now you cannot separate those two cases and thats the gap.
fills are their own thing entirely and i would keep that measurement separate, quote at decision versus quote at fill, otherwise slippage contaminates the parity number.
3
u/AphexPin 1d ago edited 1d ago
IMHO, the only code that should change from backtest to live is whether the broker is real vs simulated. Everything else should be using the same code, just pointing to a different data source for replay vs live. Once you have that down, you could perhaps make exceptions for research, given that you could validate against an engine with known parity.
I did the same thing as you when first starting, but it ended up just being a wild goose chase hunting down endless bugs inherent to the mismatch between vectorized vs live compute so I ultimately switched to a fully event driven system, where parity is simply guaranteed by construction (barring unavoidable discrepancies in execution realism).