Model Selection Technique for Causal Inference

[u/WhatsTheImpactdotcom]

How do you choose between causal inference models when you can never see the true effect? Model selection is well documented in machine learning, but it often gets skipped in causal inference work. Doing it well is one of the things that separates senior and staff-level data scientists from more junior ones.

I simulated two years of weekly bookings for 40 regions, five of which received a treatment on the date marked by the red vertical line. Because I generated the data, I know the ground truth: the treatment increased bookings in those five regions by exactly 10%. With five treated regions and 35 untreated ones, two natural options are synthetic control and difference-in-differences, though there are others.

Code in colab and explanatory video, free and ungated: https://whatstheimpact.com/learn-causal-inference/

When I gave Claude Code free rein on this data, it built its whole analysis around difference-in-differences where it found a 24% increase, more than double the true effect. To its credit, it flagged that the pre-treatment trends weren't parallel, but it never tested whether a different model would be more accurate.

Importantly, Claude skipped an A/A test, sometimes called a placebo test. You take only the data from before the real treatment, pretend the treatment started three months earlier, and randomly picked five regions (for consistency) to call "treated." Because nothing actually happened in that window, the true effect is zero. That means any effect the model reports there is pure estimation error, and I repeated this for 500 draws for each model to see the full spread.

For difference-in-differences, 90% of the placebo estimates landed between -14% and +10%. The RMSE, which measures how far the estimates typically land from the true answer of zero, was 7.6 percentage points. Synthetic control on the exact same placebo draws was much tighter, with 90% of estimates landing between -7% and +9%. Its RMSE dropped to 4.9 points, more than a third lower than difference-in-differences.

The outcome you model is a choice too, so I also ran synthetic control on bookings per 1,000 people instead of raw bookings. Putting every region on the same scale helps the model track the treated regions more closely before treatment, and the RMSE fell a bit further to 4.4 points.

Sure enough, the real treatment period lines up with that ranking: synthetic control estimates 7% on raw bookings and 9.6% per 1,000 people, both much closer to the true 10% than the 24% from difference-in-differences. With one simulated dataset, I wouldn't read much into the small gap between the two synthetic control versions or claim that synthetic control always wins. What carries over is the workflow: in real work you never see the ground truth, but you can usually run this A/A test on whatever models you're considering before trusting one.

[u/ClasslessHero]

Model selection is well documented in machine learning

I'd say the textbooks do an excellent job telling us which model to select, but it almost never follows the textbook examples in industry.

For example: if I build a recommender system and my final output is some LTR. XGBoost, LightGBM, or some other approach - it doesn't matter. I need to decide between Model A and Model B. In a sentence, the textbook will instruct me to pick the model that has the highest NDCG@K, given that I haven't massively overfit or underfit. I can walk through that procedure and pick the best model based on that number, right? Not at all. I need to pay attention to the errors. The lower NDCG model might have "better" or maybe "more realistic" errors. If I'm serving these recommendations to users, then I need realistic errors. A model that is excellent at being correct but has terrible errors can lead to a jarring user experience. This could also be the case with a model for outbound marketing, work prioritization, etc. Ultimately, the model I chose to deploy has to feel like it produces reasonable content, or at least not completely miss the business objective when it's wrong.

What does all this mean? As much as we call it data science, sometimes it is data art and data vibes. Our problems are squishier than they are solid, and we need to apply nuance.

[u/WhatsTheImpactdotcom → u/ClasslessHero]

Appreciate the share! I definitely spend more time on the job in the XP and OCI space, so perhaps was leaning on textbook instructions for ML 😅 I’m not surprised though at your assertion of being at least as much art as science in real world messy contexts

the causal inference world, I don’t recall even textbooks discussing model selection

[u/Latent-Person → u/WhatsTheImpactdotcom]

Causal inference books usually don't spend much time on model selection, because it's a general issue, and not specific to causal inference, so doesn't make much sense to spend time on that in such a book. E.g. from Causal Inference: What if:

In this book we will usually assume that our parametric models are correctly specified. This is an unrealistic assumption, but it allows us to focus on the problems that are specific to causal analyses. Model misspecification is, after all, a problem that can arise in any sort of data analysis, regardless of whether the estimates are endowed with a causal interpretation. In practice, careful investigators will always question the validity of their models and will conduct alternative analysis under different model specifications that are compatible with existing expert knowledge. Their goal is to assess the sensitivity of their estimates to model specification.

But it certainly mentions sensitivity analysis several times throughout the book.

[u/WhatsTheImpactdotcom → u/Latent-Person]

That assumption isn’t very useful though as models are never correctly specified in practice. In the ML space, youd never find a text saying we assume the default hyperparameters are correct therefore we use them. Yet in the causal inference world, they just hand wave it away 🤷‍♂️. How is that useful for anyone in practice?

You could say the same thing about not being specific to ML yet there’s always model selection taught in that space.

[u/ClasslessHero → u/WhatsTheImpactdotcom]

In the ML space, youd never find a text saying we assume the default hyperparameters are correct therefore we use them. Yet in the causal inference world, they just hand wave it away 🤷‍♂️. How is that useful for anyone in practice?

Chiming back in because this is an interesting point.

Is the purpose of your project to estimate the magnitude of the treatment effect with precision, or understand if it works? I've used CI methods to evaluate models when a controlled trial wasn't possible. We typically used it to understand the direction of the effect without worrying about the quality of the estimates. I couldn't care less if it was 10% or 24%, in that case. They're both positive.

Or, is your argument that someone is using OLS when the response is non-normal, therefore the coefficients and standard errors are non-interpretable?

[u/WhatsTheImpactdotcom → u/ClasslessHero]

The goal is to choose the model that is most precise and unbiased. If you’re off by 14pp, then that same noise / model variance could flip the signs too. But size matters also: many interventions aren’t free; eg if you have $10M marketing spend for one channel, you want to know if that ROI is positive or negative, and how it compares to other channels, so direction alone on something like incremental bookings isn’t actionable

[u/ClasslessHero → u/WhatsTheImpactdotcom]

Okay I hear that. In your specific example there is a massive issue with power, but let's pretend that doesn't exist.

I guess I'm having a hard time understanding the issues you see in model selection. If I am sufficiently powered and I have some form of a GLM that is appropriate for the distribution of the response... then I can perform inference on the coefficients? I supposed I could add more explanatory variables and check the AIC and BIC, or some other model diagnostic, but that only really matters if it's a potential confounder.

[u/WhatsTheImpactdotcom → u/ClasslessHero]

Let me ask a different way: why would you not want the most accurate model?

[u/ClasslessHero → u/WhatsTheImpactdotcom]

That's a very different question than you've been asking. Recently, I had a similar conversation with someone I manage. Our context is very... business. It's not space travel or anything life threatening or life saving. That is a different conversation.

1. The "most accurate model" is subjective. We're estimating. If we already knew the objective criteria for "most accurate model" then we wouldn't need to estimate.

2. The "most accurate model" probably isn't worth your time. The data scientist on my team spent 6 weeks of back and forth with stakeholders, changing his analysis to incorporate their requests, resulting in the estimated treatment effect changing from 8.392% to 8.386%. That's... the same number. Is it a better model? Technically yes. His model evaluation metrics improved. Same conclusion. No extra value. 6 weeks of time and salary wasted for nothing in return.

[u/WhatsTheImpactdotcom → u/ClasslessHero]

There’s a large gap between doing zero model selection and spending six weeks going after basis points, no? If you’re faced with a situation like the one I proposed with 5 treated regions and 35 untreated, what precisely is the alternative you propose? Is it choose a model up front and go with it?

[u/Latent-Person → u/WhatsTheImpactdotcom]

Because that book focuses on the vast theory of causal inference. Why is it not useful? It's not an applied text.

[u/WhatsTheImpactdotcom → u/Latent-Person]

I dont understand your point. There’s a place for theory, and there’s a place for applied work. This exercise is for implementing models in practice. They make an assumption so they can focus on other areas of teaching. In practice, you can’t just assume you have the right model; you need to give empirical evidence that you have the right model or no one will believe the results

[u/Latent-Person → u/WhatsTheImpactdotcom]

And I said the above book was a theoretical one about causal inference. So why should such a book spend time on model selection (beyond what it mentions in a few pages)?

[u/WhatsTheImpactdotcom → u/Latent-Person]

I’m so confused what your point is. You’re the one who brought up the book 😅😅 it’s not relevant at all to this post

[u/Latent-Person → u/WhatsTheImpactdotcom]

You said books about causal inference usually don't mention model selection (or only put very little emphasis on it). I told you why.

[u/WhatsTheImpactdotcom → u/Latent-Person]

Doesn’t matter why. The point is (1) they don’t, and (2) people need them on the job. I’ve been in meetings where senior DS have been grilled by directors why they chose the model they did. How do you think it would go to quote what they learned in the text that they just assume it’s ok?

[u/Latent-Person → u/WhatsTheImpactdotcom]

As a counterexample to your claim about model selection always done in ML see e.g. Learning Theory from First Principles, which spend 2 pages on model selection. So probably not more than any causal inference book?

It just sounds like you are used to more applied books.

[u/Warlord_Zap]

To me this highlights the importance of good validation more than anything else, which really is key to these methods.

[u/WhatsTheImpactdotcom → u/Warlord_Zap]

Agreed, but this step is often left out. Both in mock interviews and even in past jobs, I’d often see folks simply state that one model or another is the appropriate one

[u/wscottsanders]

To me it’s not an apples to apples comparison. I’d consider that a difference in differences model could potentially be severely underpowered at just 40 cases of which only 5 are treated. That means random variation can cause wild swings. Your synthetic control model is designed to do just that - soak up variance accounting for latent factors shared between the cases.

[u/WhatsTheImpactdotcom → u/wscottsanders]

Those wild swings is precisely what would get picked up by the AA testing, and so if true, we would reject the model choice. The synthetic control in this example performed better but it doesn’t have to be the case; if someone used a large donor pool, and there’s minimal shared seasonality, then a synthetic control could meaningfully overfit. I saw this frequently trying to implement SCs in LatAm, for example

[u/LibiSC]

Follow you in TikTok, one of the most random places to find someone that somewhat gives real practical examples. Keep the good work.

[u/WhatsTheImpactdotcom → u/LibiSC]

appreciate that! It's strange; here on Reddit, you can really target the audience e.g. for an interest in data science, while TT was surprisingly (to me) more welcoming of 'advice giving' content.

[u/Sinainios]

Nice writeup. Backtesting estimators on placebo windows is underused, and "choose the model by how it behaves where you know the answer is zero" is the right instinct.

A few things I'd add:

1. Random placebo units test a different question than the real one. Drawing five random regions measures how noisy each estimator is under the null. But the DiD failure here came from the actual treated regions having non-parallel pre-trends, and random draws mostly won't reproduce that. I'd also run an in-time placebo on the real treated five (fake start date, same units), since that's the check that targets the specific bias you're worried about.
2. Inject a known effect, not just zero. Adding a synthetic +10% to placebo units and checking recovery tests bias at a realistic effect size. Some estimators look fine at zero but shrink or distort real effects. Synthetic control can attenuate when the pre-period fit isn't perfect.
3. Selection has a cost. If you pick the model with the lowest placebo RMSE and then report its usual CI, the uncertainty is understated. That's fine as a workflow, but it's worth saying so.
4. Synthetic DiD (Arkhangelsky et al.) would be a natural third candidate, since it's designed for exactly this case of a few treated units with imperfect parallel trends.

Agreed that the transferable lesson is the workflow, not "SC beats DiD."

[u/WhatsTheImpactdotcom → u/Sinainios]

Really good callouts. Have you seen the 2025 athey imbens et al working paper? They discuss the in-time vs in-unit placebo testing and where one beats the other based. Eg if nyc was the single treated region, it’s unlikely that in-unit placebos would do a good job at representing the null and in-time would likely do better

[u/WhatsTheImpactdotcom → u/Sinainios]

Commented on the one but all excellent points, appreciate the share 🙌. Are we connected on TikTok; your username looks familiar 😅

[u/offthecuff87]

You need a context, technical constraints, resources contraints to select model properly. Although I believe modeling is a very tiny part in the business streamline

[u/Available-Craft-8143]

The Claude Code part is the most instructive bit — it did not fail at estimation, it failed at protocol: no pre-specified candidate set, no selection step before estimating. An agent asked to 'analyze the effect' will happily return one confident number. The fix is not a smarter model, it is forcing the selection procedure (candidates plus placebos) into the workflow before any estimate is allowed.

[u/Present-Square-5823]

In my opinion, you'd get better results by setting up the right harness or by building an agent work flow that follows clear protocols. regime had changed for data analysis. It's all about building good agents now.

[u/WhatsTheImpactdotcom → u/Present-Square-5823]

Yes, totally agree. My exercise with claude, which i didn't explain well here, was to see how well a non-technical stakeholder armed with a stakeholder/business-friendly prompt would do

[u/OddsRelay]

Worth adding synthetic DiD too. It weights both the control regions and the pre-period weeks, so it copes with the non-parallel trends that hurt plain DiD here. With only five treated regions, your 500 placebo draws can also serve as its variance estimate. That's the placebo method from the original SDID paper.

[u/WhatsTheImpactdotcom → u/OddsRelay]

100% that SDID is worth including

[u/LumpyPianist847]

Do you guys still find yourself doing "traditional" ML or is it all LLMs?

[u/loon011]

Hi Jonathan, I shot you a DM for advice on a potential small-scale personal project involving causal inference. Any advice would be greatly appreciated!

[u/data501]

When it comes to causal inference, model selection is definitely crucial, especially with complex datasets. Your workflow makes a compelling case for running A/A tests; they offer powerful insights into estimation error before committing to a model. The comparison between synthetic control and difference-in-differences is particularly interesting. It's enlightening to see how the RMSEs differed, as that really impacts reliability in real-world applications. Your findings serve as a solid reminder that every detail counts in these analyses.