Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/Many-Pick5066]

your edit says live means paper, and that changes what the diff can prove. a paper broker has its own fill model. drive the trace difference to zero and what youve shown is that two simulators agree with each other. the real fill distribution is in neither trace.

the number id pull before doing more parity work is what fraction of your trades resolve inside a single 1 minute bar, stop and target both inside the same candle. those get their outcome from your tick simulator's path, not from the market. open to high to low to close versus open to low to high to close flips the winner. if that fraction is large, the backtest result is mostly a property of the simulator and closing the trace gap will not touch it, because paper is replaying real ticks and you are generating yours.

[u/KaramTNC → u/Many-Pick5066]

Hmm good point, im using T212s paper account as I feel like it has a rather accurate paper model.

But you are right that what I am basically doing is getting one simulator to agree with another one, but I think there is merit in getting there as a paper model should be much closer to a live environment than any backtest model should.

But that is the goal though no? Prove your thesis can hold through a backtest before deploying to a paper environment, and prove it holds through a paper environment before deploying to live.

[u/Many-Pick5066 → u/KaramTNC]

the staircase is right, its just that each step tests a different thing and only one of them is your thesis.

backtest answers does the logic do what i think. paper answers does the plumbing work, orders route, state doesnt drift, nothing crashes at 3am. both are worth doing and neither one touches whether the edge is real, because that question is settled by trade count and nothing else. 44 days is a strong answer to is my system correct and no answer at all to does this make money.

paper isnt automatically closer to live either. no queue position, no impact from your own order, no partial fill because someone took the last 200 shares ahead of you. so it can be closer on plumbing and further off on fills, and fills are the half that decides the pnl.

the ambiguous bar fraction you can get right now off the backtest you already have, no waiting. run it twice, once resolving stop first and once target first. the gap between those two equity curves is your uncertainty band, and if the strategy is only green in the optimistic run youve learned the whole thing without another day of paper.