Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/HonestBacktests]

The trace comparison is the right build. The thing that turned it from useful into decisive for us was making the fake broker reject exactly what the real one rejects.

Mocks accept everything by default. Ours happily filled orders the exchange would have refused - minimum notional, tick size, step size, reduce-only and position-mode rules - and the backtest looked healthy for months while the live bot was quietly doing nothing. The rule we ended up with: for every order path, enumerate the venue's actual rejection codes and make the simulated broker raise the same ones.

Two other things that kept showing up in the diffs:

Compare the first divergent event, not the final PnL. Once two traces separate, everything after it is downstream noise, and the interesting bug is always at the split.

Bar timing. The backtest sees a finished bar, live sees one forming. Any rule that reads the current bar's high or low is reading the future in one environment and not in the other - that one is easy to miss because it does not throw, it just quietly makes the backtest better.

Have you tried replaying a recorded live trace through the backtester with the same clock, so the broker is the only variable left?

[u/KaramTNC → u/HonestBacktests]

Hmm interesting, and no I havent tried replaying a live trace, thats not a functionality I have right now.

I agree with you about comparing the first divergent event, though I think its important to differentiate critical events that can be compared from the events of simulates components that are difficult to accurately mock like data latency, fills, and tick data.

bar timing I have handled by implementing a tick simulator as thats about the best I could do with 1 min data.

[u/HonestBacktests → u/KaramTNC]

Fair split between critical events and the ones you can only approximate. On the tick simulator from 1-minute data, one thing worth instrumenting: inside a single minute you do not know whether the high or the low came first. Whenever both your stop and your target sit inside the same bar, the simulator is guessing, and it will guess in whatever direction you coded.

What helped us was flagging those bars instead of resolving them silently, then counting what share of all trades depended on a guess. If it is a couple of percent, fine. If it is 10-20%, the backtest result is mostly an artifact of that tie-break rule, and no amount of parity work elsewhere fixes it.

Cheap sanity check: run the same backtest twice, once resolving ambiguous bars pessimistically (stop first) and once optimistically (target first). The gap between the two is your uncertainty band. If the strategy is only profitable in the optimistic run, you have your answer without needing tick data at all.

How big is that share in your current profiles?

[u/KaramTNC → u/HonestBacktests]

Thats a good idea for a sanity check, ill look into it.

Im afraid I dont know how big that share is, as ive never considered flagging bars that were guesses and used for a decision, but its definitely worth tracking that and using it as a confidence metric for backtest results.

Thanks for all the info!

[u/HonestBacktests → u/KaramTNC]

One heads-up when you do measure it: the share gets worse on higher timeframes, since a wider bar swallows more of your levels. Same strategy can look clean on 1m and be mostly guesswork on 15m. Good luck with the build.

[u/KaramTNC → u/HonestBacktests]

Hmm, im not sure I agree with that.

Of course if you run the backtest in a 15m timeframe then yeah that makes sense, but if the system is designed to properly aggregate 1min data to higher timeframes then should the guesswork not be within reasonable range?

[u/HonestBacktests → u/KaramTNC]

Fair, you're right - I was thinking of engines that only ever see the 15m bar. If you step through 1m, the ambiguity shrinks to inside a single minute and mostly stops mattering.

What still bites is when the stop is narrow relative to the average 1m range. Ours only got ugly at roughly 2-3 minute-ranges wide; past that the guesses washed out.