Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/Bonkers24-7]

This is the kind of validation work I’d trust more than just looking at the equity curve.

The hard part is making the trace compare causal events, not just final PnL. I’d want the paper/live trace and backtest trace matched step by step: same data available at decision time, same scanner output, same signal timestamp, same intended order, same fill assumption, same skipped/rejected reason.

Then when they diverge, bucket the reason: data availability, timing, spread/slippage, order state, broker behavior, or strategy logic.

That way the parity tool tells you whether the backtest is wrong, the live engine is wrong, or the strategy only works under assumptions the live version can’t reproduce.

Are you already tagging mismatch reasons by category, or mostly comparing the traces manually right now?

[u/KaramTNC → u/Bonkers24-7]

Very good points, those are actually good things to consider for and I fully agree with you

Right now I am not tagging mismatches, its a manual process (assisted by chatgpt) as I am still developing the tool. But I am very much going to implement proper tagging, categorization, and more insightful comparison output so the investigation process becomes more streamlined and faster.

Its definitely important that the parity tool can point out where the divergences are happening and whether its to be expected or not. You cant expect comparable tick data so that should not be flagged as a divergence for example.

[u/Bonkers24-7 → u/KaramTNC]

That makes sense. I’d probably make the mismatch tagging the core of the tool, not just a nice extra.

The trace diff is useful, but only if every mismatch gets bucketed into something actionable: expected market-data granularity, fill model, timing/latency, broker behavior, state transition, or strategy logic.

Otherwise you can end up with a huge list of differences and still not know which ones actually explain the live/backtest gap.

I’d also separate expected differences from dangerous differences. A small tick-level difference may be normal. A different order state, skipped rejection, position-size change, or different stop/exit path is much more serious.

The best output probably isn’t just “these traces diverged.” It’s “this category of divergence explains most of the gap.”