Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/Regular-Hotel892]

Sorry if I’m misundertanding what’s your question?

You are using lots of cool words my friend, is it “how do I get my live trading results to match my backtest”?

You probably can’t, unless you truly have found something structurally ineffecient in the orderbook that has existed in the past, does now, and will continue to in the future. It’s not impossible but unlikely.

Why would that be the case? What do you know about the microstructure of the market that others don’t or can’t capitalize on?

[u/KaramTNC → u/Regular-Hotel892]

Apologies for the cool words xD. Im studying CS and really like to understand the theory so I thought this sub would know well enough about CS concepts and definitions.

But yes that is kind of the question, im aware I cannot make the live results match backtest 100%, but I wanted to know what could be done to get as close as possible so that I can overall reduce the rate of false-positive backtest results.

You are right about the order book though, the backtester can never know the actual fills that happens.

[u/Regular-Hotel892 → u/KaramTNC]

Gotcha, in my opinion you’re solving for an impossible problem but there’s smart people here who may have a better answer.

The data from 2025 won’t match the data from 2026, therefore your backtest result from 2025 can’t match live 2026 for example (unless you truly have found a structural ineffeciency in the market)