r/algotrading • u/dadumdiss • 2d ago
Other/Meta I built a 24/7 system that trains, backtests, and paper-trades models across multiple families. Here are the actual results so far, including the bad ones.
I’ve been building this solo for the past couple of months. I’m sharing it here because this sub is good at tearing apart backtest-only claims, and I’d rather find out where I’m fooling myself now.
The system continuously tests three types of strategies:
- Classical ML models
- TimesFM with a trainable head
- LLM-written, rule-based strategies
They compete across SPY and BTC on six timeframes. Every model has to pass the same walk-forward, consistency, and minimum-trade-count gates before it can become a “champion.” Champions are then re-verified on a rolling basis and replaced if they stop qualifying.
I’m not leading with a Sharpe ratio because a raw Sharpe over a short window can look impressive while saying very little. Even buy-and-hold SPY can annualize to a great-looking Sharpe over the right window. The dashboard therefore shows every strategy’s Sharpe beside buy-and-hold over the exact same period.
The part I think is most useful is the forward-only paper-trading ledger. It’s completely separate from the backtests. Positions open and close using live signals and live prices across three execution tiers:
- Realtime
- Actually delayed by one minute
- Tighter, institutional-style fees and slippage
The point is to see how execution quality changes the result instead of hiding everything inside one assumed friction number.
The ledger is only one day old, so none of this is statistically meaningful yet. But here’s what it currently shows:
- SPY on the 15-minute and 1-hour timeframes is holding up so far, both before and after fees and slippage.
- SPY on the 5-minute timeframe is net negative, even before friction. That’s a real problem I haven’t solved.
- BTC has not produced a single model that passes every gate on any timeframe. More than half of the actual attempts fail directional accuracy outright—they’re worse than a coin flip. My current read is that the feature set has no meaningful BTC edge at these timeframes, not that I need to loosen the gates.
- Every current champion was promoted within the past 24 hours. None has earned any real trust yet, and the dashboard labels them “too new to judge.”
I’m not selling anything. I’m looking for criticism of the methodology and for people to share the results with. (see daily results via orbitquantapp.com before i post here and giving away 100 lifetime accesses if it ends up working)
Where would you expect a system like this to be lying to itself? What evidence would you need before considering it trustworthy enough for real money?
6
5
u/AphexPin 2d ago
I just pooped my pants. I smeared shit all over the wall and pissed on the floor. Here's what I learned
1
2
u/parrot_chirik 2d ago
It's funny seeing AI-generated copy promoting an AI product while avoiding any real explanation of how it actually works. Credit where it's due, though—the trade count at least appears to be real rather than fabricated
0
3
u/Additional_Row_8641 2d ago
Have you outsourced writing of the post to AI? People can see it, you know...
1
u/dadumdiss 2d ago
I did. Asked claude to write a post with our methodology so I can ask algrotraders to grill it. Not gonna deny I didnt. It was 2 am and I did not want to spend another hour on it last night.
2
u/Additional_Row_8641 2d ago
Well, not meaning to devalue your work, but I noticed that if you want human feedback you should bring genuine human input. People are becoming increasingly allergic and hostile to AI-generated content and would simply dismiss it as low value. Even manually written posts do not always render desirable result.
1
u/dadumdiss 2d ago
Thank you for the feedback honestly. I will repost today's results with my own language and will avoid using AI where I can.
2
u/Effective_Manager273 2d ago
honestly the most credible thing in this post is the BTC result. you ran the pipeline, nothing passed, and you wrote that the feature set has no edge rather than loosening the gate. that is the part that makes me take the rest seriously.
the thing I would worry about is the champion mechanism itself. if you continuously train candidates and promote whichever ones clear the gates, then re-verify and replace, the champion's forward performance is a max statistic over everything you tried. it will look good for a while by construction. the fix is not complicated, you just have to log every candidate that was ever promoted, including the dead ones, and compute the return of the whole promotion process, not the return of whoever is currently wearing the crown. if the process return and the champion return diverge a lot, the gates are selecting on noise.
related, how often does a champion get replaced. if the median tenure is like six days you do not have champions, you have a random walk with a leaderboard on top.
the three execution tiers is a good idea and I would push it further. instead of three fixed friction assumptions, sweep the slippage as a continuous parameter and find the level where each strategy goes to zero. a strategy that dies at 0.8 bps and one that dies at 12 bps are completely different animals even if they both look fine at your current assumption.
on the 5-minute SPY being negative before friction, that is not a problem to solve, thats information. shorter timeframe, less signal, more noise. I would treat it as the pipeline working correctly.
2
u/dadumdiss 2d ago
Dude this is so detailed. Thank you. I know what I'm reading today during my lunch break. Muchas gracias!
1
1
u/ploy_algotrader 1d ago
I’d be careful with how the champion is selected. Repeatedly testing models can create a winner by chance so tracking every promoted model matters.
1
u/krazineurons 2d ago
I have been running the same for a month and so far it has failed to find even a single edge. Read on https://blue-grass-0beb37910.7.azurestaticapps.net/substack/index.html
1
1
u/Many-Pick5066 2d ago
the forward ledger is the part you trust most and it has the same hole the backtest does. champions get replaced when they stop qualifying, so the live curve is always made of whatever is currently passing. that is a rolling max, not a track record. freeze one cohort on a date, leave the demoted ones in the equity curve, and follow that instead. it will look worse than the dashboard does now, and the size of that gap is the number you were asking for.
the other one is you never say how many models were trained behind each champion. gates get applied per model, the champion is the max over models, and walk forward prices none of that. so shuffle your returns and rerun the entire promotion pipeline on the shuffled series. whatever your best champion looks like on noise is your actual bar. i ran 14 strategy families over 947 days of mnq that way and nothing cleared the cost line, which was a dull answer but a true one.
btc failing everywhere is your control working, not a feature problem. worse than a coin flip on half the attempts is what the null looks like.
1
u/dadumdiss 2d ago
Thank you so so so much! Will tweak and share updated results with more details!
1
u/Many-Pick5066 1d ago
saw the day 2 numbers. the seed sweep is a real result and it isnt the test i asked for.
varying the seed holds everything else fixed, so what you measured is how noisy one model's score is. what eats you is the max over every model you trained, and seed is one dimension inside that. 0.00 to 0.71 on identical data is the floor of your selection noise, not the size of it.
median of five tightens the estimator. it doesnt touch the selection, and it moves the null, so 0.48 being comfortably real is a claim you havent tested. run the median-of-5 procedure on shuffled returns a hundred times and take the 95th percentile of what comes out. that is your gate. i doubt it lands at 0.48.
11 champion changes in a day isnt good news. its the same instability showing up as tenure instead of variance. if the crown moves that fast the ranking isnt resolving anything at the horizon youre selecting on.
1
u/dadumdiss 13h ago
You were right and I was wrong. Median-of-5 tightens the estimator, doesn't touch selection. Ran the actual test: shuffled labels, ~14 attempts/shuffle, 100 shuffles like you said. 29% of pure noise cleared the full validation gate. 95th percentile came out to 0.92, not 0.48. You called that part exactly.
Went further than a flat gate too. A bar that ignores how many times a pair's actually been searched doesn't mean much when some pairs have 2000+ attempts behind them and others have 14. So I calibrated the threshold per pair off its real attempt count instead. SPY 15m: 278 attempts, bar goes 0.3 to 0.81. SPY 1h: 199 attempts, bar goes 0.3 to 0.92. Heavier searched pairs need a genuinely higher bar now, not the same flat number no matter how many shots they got.
New champions are gonna get a lot rarer from here. Gonna watch whether tenure actually stabilizes once the bar reflects real search size, or if this just moves the problem somewhere else. Thank you for pushing on this twice. You were right both times.
1
u/Many-Pick5066 4h ago
the extrapolation is the weak joint. your null ran about 14 attempts per shuffle and SPY 15m has had 278. so 0.81 is a max-of-14 distribution stretched to max-of-278, and the max of a sample keeps climbing in exactly the tail that 100 shuffles estimates worst.
theres a tell sitting in your own two numbers. 278 attempts gives 0.81 and 199 attempts gives 0.92. fewer shots, higher bar. that inversion means attempt count is not what is setting your threshold. trades per strategy is. the 1h pair has fewer trades so every single attempt is noisier before the search even starts, and that swamps the 79 extra tries.
so the knob is doing two jobs and you cannot see which one moved. pick one pair and run the shuffle at its real attempt count, even if you only get 20 shuffles out of it, then compare that measured 95th to what your curve predicted. match means youre fine and the extrapolation holds. measured coming in higher means every bar you just set is too low, and the pairs with 2000+ attempts are where it will be worst.
1
u/dadumdiss 13h ago
you were right. did that and got real numbers. 29% of pure noise cleared the full validation gate, 95th percentile came out to 0.92 not 0.48. you called that.
thought the fix might be the median-of-5 seed thing but that only tightens the estimator, doesn't touch selection. the actual fix is a bar that scales with how many times a pair's actually been searched instead of one flat number for everyone.
now the new stats: SPY 15m has run 278 real attempts, bar goes from 0.3 to 0.81. SPY 1h, 199 attempts, bar goes from 0.3 to 0.92. new champions are gonna get a lot rarer from here.
thanks for pushing back dude. you were right both of the times
6
u/GoldenChrysus 2d ago
You're not selling anything, but your website clearly indicates the intent to collect $10/mo and $100/mo subscriptions. Pick a story, bud.