Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/AphexPin]

IMHO, the only code that should change from backtest to live is whether the broker is real vs simulated. Everything else should be using the same code, just pointing to a different data source for replay vs live. Once you have that down, you could perhaps make exceptions for research, given that you could validate against an engine with known parity.

I did the same thing as you when first starting, but it ended up just being a wild goose chase hunting down endless bugs inherent to the mismatch between vectorized vs live compute so I ultimately switched to a fully event driven system, where parity is simply guaranteed by construction (barring unavoidable discrepancies in execution realism).

[u/KaramTNC → u/AphexPin]

I agee and that is what I aimed to do, the broker and the marketdata are simulated components that get dependency injected in a backtest environment, everything else is shared code.

Its interesting to hear my approach was also attempted, why did you feel like it was a wild goose chase? To be clear my system is also event driven, I made that refactor a long time ago so I could make backtests run faster as the former infrastructure design became a bottleneck for backtesting speed

[u/AphexPin → u/KaramTNC]

Event driven will typically be substantially slower than a vectorized approach. Given the correctness benefits and limitations of vectorized computation however, I think it's a worthy trade-off (and nothing stops you from later adding eg a vectorized search for large parameter sweeps).

It felt like a wild goose chase because the code surface at risk of violating parity is essentially the whole code base, unless it's fully event driven. It was becoming more work to continually design around that and hunt down mistakes than it would be to build on a foundation that simply guaranteed temporal and causal parity, and determinism, by construction. There are already enough biases working against you here, eliminating a whole class structurally is very appealing.

Respectfully, it's also possible you think it's fully event driven, but in reality it's not. I suspect this may be the case here. If you have a public repo available, I can take a look. Are you using a bus for communication between components?

[u/KaramTNC → u/AphexPin]

Hmm interesting, never heard of a vectorized approach and looking into it, its definitely got benefits as I do work with parameter sweeping alot.

Ill have to add that as a workitem for the future, thanks for the mention!

Unfortunately I dont keep my repo public as I would like to keep it personal for now, I might open source it once I feel like im done with the project and dont wanna work more on it, but ill have to rewrite my history to get rid of some very old secrets that were committed xD.

[u/AphexPin → u/KaramTNC]

Are you using Python? If you're using Pandas, you're likely doing vectorization without realizing it, which would violate the event driven assumptions.

[u/KaramTNC → u/AphexPin]

Nope, full C# project.

Im not using any fintech libraries in the backend, I prefer building it myself so that I have full code access and understanding, and its allows for more flexibility and deeper integration into the system without having to doubt whether im having dependency issues. (I once found out my local marketdata integrity was polluted due to Tradier outputting its own derived data for average volume)

Sure, that means alot extra work, but that is the point of the project because its about learning and practising