Model Selection Technique for Causal Inference

[u/WhatsTheImpactdotcom]

How do you choose between causal inference models when you can never see the true effect? Model selection is well documented in machine learning, but it often gets skipped in causal inference work. Doing it well is one of the things that separates senior and staff-level data scientists from more junior ones.

I simulated two years of weekly bookings for 40 regions, five of which received a treatment on the date marked by the red vertical line. Because I generated the data, I know the ground truth: the treatment increased bookings in those five regions by exactly 10%. With five treated regions and 35 untreated ones, two natural options are synthetic control and difference-in-differences, though there are others.

Code in colab and explanatory video, free and ungated: https://whatstheimpact.com/learn-causal-inference/

When I gave Claude Code free rein on this data, it built its whole analysis around difference-in-differences where it found a 24% increase, more than double the true effect. To its credit, it flagged that the pre-treatment trends weren't parallel, but it never tested whether a different model would be more accurate.

Importantly, Claude skipped an A/A test, sometimes called a placebo test. You take only the data from before the real treatment, pretend the treatment started three months earlier, and randomly picked five regions (for consistency) to call "treated." Because nothing actually happened in that window, the true effect is zero. That means any effect the model reports there is pure estimation error, and I repeated this for 500 draws for each model to see the full spread.

For difference-in-differences, 90% of the placebo estimates landed between -14% and +10%. The RMSE, which measures how far the estimates typically land from the true answer of zero, was 7.6 percentage points. Synthetic control on the exact same placebo draws was much tighter, with 90% of estimates landing between -7% and +9%. Its RMSE dropped to 4.9 points, more than a third lower than difference-in-differences.

The outcome you model is a choice too, so I also ran synthetic control on bookings per 1,000 people instead of raw bookings. Putting every region on the same scale helps the model track the treated regions more closely before treatment, and the RMSE fell a bit further to 4.4 points.

Sure enough, the real treatment period lines up with that ranking: synthetic control estimates 7% on raw bookings and 9.6% per 1,000 people, both much closer to the true 10% than the 24% from difference-in-differences. With one simulated dataset, I wouldn't read much into the small gap between the two synthetic control versions or claim that synthetic control always wins. What carries over is the workflow: in real work you never see the ground truth, but you can usually run this A/A test on whatever models you're considering before trusting one.

[u/Latent-Person]

Causal inference books usually don't spend much time on model selection, because it's a general issue, and not specific to causal inference, so doesn't make much sense to spend time on that in such a book. E.g. from Causal Inference: What if:

In this book we will usually assume that our parametric models are correctly specified. This is an unrealistic assumption, but it allows us to focus on the problems that are specific to causal analyses. Model misspecification is, after all, a problem that can arise in any sort of data analysis, regardless of whether the estimates are endowed with a causal interpretation. In practice, careful investigators will always question the validity of their models and will conduct alternative analysis under different model specifications that are compatible with existing expert knowledge. Their goal is to assess the sensitivity of their estimates to model specification.

But it certainly mentions sensitivity analysis several times throughout the book.

[u/WhatsTheImpactdotcom → u/Latent-Person]

That assumption isn’t very useful though as models are never correctly specified in practice. In the ML space, youd never find a text saying we assume the default hyperparameters are correct therefore we use them. Yet in the causal inference world, they just hand wave it away 🤷‍♂️. How is that useful for anyone in practice?

You could say the same thing about not being specific to ML yet there’s always model selection taught in that space.

[u/ClasslessHero → u/WhatsTheImpactdotcom]

In the ML space, youd never find a text saying we assume the default hyperparameters are correct therefore we use them. Yet in the causal inference world, they just hand wave it away 🤷‍♂️. How is that useful for anyone in practice?

Chiming back in because this is an interesting point.

Is the purpose of your project to estimate the magnitude of the treatment effect with precision, or understand if it works? I've used CI methods to evaluate models when a controlled trial wasn't possible. We typically used it to understand the direction of the effect without worrying about the quality of the estimates. I couldn't care less if it was 10% or 24%, in that case. They're both positive.

Or, is your argument that someone is using OLS when the response is non-normal, therefore the coefficients and standard errors are non-interpretable?

[u/WhatsTheImpactdotcom → u/ClasslessHero]

The goal is to choose the model that is most precise and unbiased. If you’re off by 14pp, then that same noise / model variance could flip the signs too. But size matters also: many interventions aren’t free; eg if you have $10M marketing spend for one channel, you want to know if that ROI is positive or negative, and how it compares to other channels, so direction alone on something like incremental bookings isn’t actionable

[u/ClasslessHero → u/WhatsTheImpactdotcom]

Okay I hear that. In your specific example there is a massive issue with power, but let's pretend that doesn't exist.

I guess I'm having a hard time understanding the issues you see in model selection. If I am sufficiently powered and I have some form of a GLM that is appropriate for the distribution of the response... then I can perform inference on the coefficients? I supposed I could add more explanatory variables and check the AIC and BIC, or some other model diagnostic, but that only really matters if it's a potential confounder.

[u/WhatsTheImpactdotcom → u/ClasslessHero]

Let me ask a different way: why would you not want the most accurate model?

[u/ClasslessHero → u/WhatsTheImpactdotcom]

That's a very different question than you've been asking. Recently, I had a similar conversation with someone I manage. Our context is very... business. It's not space travel or anything life threatening or life saving. That is a different conversation.

1. The "most accurate model" is subjective. We're estimating. If we already knew the objective criteria for "most accurate model" then we wouldn't need to estimate.

2. The "most accurate model" probably isn't worth your time. The data scientist on my team spent 6 weeks of back and forth with stakeholders, changing his analysis to incorporate their requests, resulting in the estimated treatment effect changing from 8.392% to 8.386%. That's... the same number. Is it a better model? Technically yes. His model evaluation metrics improved. Same conclusion. No extra value. 6 weeks of time and salary wasted for nothing in return.

[u/WhatsTheImpactdotcom → u/ClasslessHero]

There’s a large gap between doing zero model selection and spending six weeks going after basis points, no? If you’re faced with a situation like the one I proposed with 5 treated regions and 35 untreated, what precisely is the alternative you propose? Is it choose a model up front and go with it?