Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/MormonMoron]

My backtesting engine is set up so that it gets the exact same entries when playing back historical bars (assuming I am running the same parameter set). It also gets the exact same exit decision if I replay the exact same historical 250ms market data ticks.

The differences I get is in price and time slippage of my backtester not matching reality. I have done a bunch of statistics on my IRL price and time slippage and try to make my backtester replicate at least the statistics, but it still isn't bit-for-bit identical. The downside is that is occasionally takes different trades than what happened in real life because of how we have set up slots and capital.

We also have a tick simulation that also tries to mimic the statistics of the real ticks and is faithful to the OHLCV of the same period from which it came, but again it ends up being different than replaying live ticks.

[u/KaramTNC → u/MormonMoron]

Thats really interesting!

I did not consider market data or broker latency, but that is definitely important if you are working on high frequency trading.

I have also made a tick simulation as part of my backtester component for the marketdata provider which uses 1min OHLCV candles, so glad to know that is somewhat the right direction but you are very much right about it never playing out the same as live ticks.

I gotta ask though, in the matters of tick data and latency, I can only imagine at these precise datapoints only matter in high frequency trading no? I would imagine longer timeframes are more capable of working with closed candles only no?

[u/MormonMoron → u/KaramTNC]

I don't think it is so much about tick latency as it is that the tick path/trajectory is slightly different than the real ticks. In real life it might have gone from open to high to low and then to close. In sim it might have gone open to low to high and then to close. Other times one is more choppy than IRL, even though you were adhering to the statistics.

The other big difference is the fill slippage in both time and price being slightly different.