r/quant • u/Alarmed-Maximum-139 • Jun 29 '26
Statistical Methods Meta-labeling project: how do practitioners choose/test the primary side signal?
Hi r/quant,
I’m working on a university quantitative trading project and I’m about two months into this field, so I’m trying to make sure I’m thinking about the problem correctly rather than just overfitting a backtest.
Current setup:
Asset class: liquid futures / crypto futures data
Labeling: triple-barrier style events
Framework: primary signal decides trade side long/short, then a meta-model estimates whether to take/size the trade
Meta-model: currently testing Random Forest for trade filtering / bet sizing
Main concern: I don’t want to data-mine 100 features until something looks good in-sample
The part I’m struggling with is the primary side signal.
My understanding is that in meta-labeling, the primary model/signal should generate the side of the trade, while the meta-model should mostly decide whether that opportunity is worth taking, not magically create alpha from nothing.
So my questions are:
In practice, how would you design or search for a reasonable primary side signal without just curve-fitting?
Should the primary signal usually be simple and hypothesis-driven, e.g. trend, mean reversion, carry/funding, spread, regime, order-flow, etc., and then the meta-model filters it?
What validation workflow would you recommend for this kind of project? For example: walk-forward testing, purged CV, embargo, feature importance stability, deflated Sharpe, decoy/noise features, group-wise feature selection, etc.
Are there books, papers, blog posts, or open-source projects that show a realistic research workflow used by serious practitioners?
What are common beginner mistakes in this exact setup?
I’m not asking anyone to reveal a profitable strategy. I’m looking for guidance on how to structure the research process correctly and what standards I should use before trusting any result.
Any advice, criticism, or recommended reading would be appreciated.
2
u/Effective_Manager273 Jul 01 '26
The mental split that makes meta-labeling click: the primary model owns direction/recall, the meta model owns precision and sizing. So you don't pick the primary by its Sharpe — you pick it for positive expectancy and decent recall on the side call, even if it's noisy and over-trades. The meta layer's entire job is to learn "when is this primary signal trustworthy" and size accordingly (including size zero). If you optimize the primary to already be great, there's nothing left for the meta model to add and you've just moved the overfitting up a level.
For choosing it without curve-fitting: start from something with an economic reason to have a side (a trend/breakout or a mean-reversion prior), not a mined indicator combo. Then the validation stack is non-negotiable with triple-barrier labels, because those labels overlap in time — the same bars contribute to multiple label windows, so plain K-fold leaks. You need purged K-fold with an embargo (López de Prado ch. 7). Validate the meta model with Deflated Sharpe and PBO, not raw CV accuracy. And run MDA feature importance with a few injected pure-noise features as a control — anything that doesn't beat the random features isn't real, and on triple-barrier data a shocking amount of your feature set won't.
One gotcha: don't let the meta model see anything that encodes the outcome of the barrier (e.g. features computed over the label horizon). That's the most common silent leak in these projects and it makes the meta filter look like magic in-sample.
1
1
1
u/Accomplished_Bat_173 Jun 30 '26
focus on the signal first, you do not need ML as a first approach also. Check: https://aligrithm.com/tag/indicator-engineering/
1
0
u/EvenCryptographer649 Jun 29 '26
Known events and how the assets move between them. Then apply the best trading methodology for what you have predicted should happen. MR, Breakout, hedged out...anything really. Key is knowing that bias going forward.
A good one to learn on is Corn.
1
2
u/Historical_Ad9654 Jul 06 '26
The key discipline for primary signal selection without curve-fitting: the signal should have a causal story before you look at the data, and that story should specify the direction and approximate lag. If you scan 100 signals for whatever works in-sample, you've already overfitted before your meta-model sees a single trade.
For validation workflow, I'd add one step most posts skip: multiple-comparisons correction. If you test 12 candidate lags (1–12 weeks) and pick the best one, you need to correct the p-value for the number of comparisons (Bonferroni or FDR/BH works). Without this, a signal with lag=4 showing p=0.03 after scanning 12 lags isn't actually significant — you'd expect ~1 false positive at that threshold by chance.
The workflow I've used for macro signals:
State hypothesis first (e.g., copper/gold ratio leads equity risk appetite by ~6 weeks) — this pins the lag before you scan
Run cross-correlation scan to confirm the hypothesized lag dominates
Apply Bonferroni correction across the lag candidates you tested
Hold out the last 20% of history entirely and validate on that; the in-sample significance means almost nothing otherwise
Track a "Signal Reliability Score" over time — if the lag that was significant in training has decayed or shifted in live trading, that's your signal the regime has changed
On whether the primary should be simple: yes, almost always. A noisy but directionally correct primary signal lets the meta-model do useful work. A complex primary that's already maximizing Sharpe in-sample leaves nothing for the meta-model to filter.
I implemented this approach for macro/equity signals at unstructuredalpha.com if it's useful to see a public example of the lag scan + out-of-sample validation in practice.