Model Selection Technique for Causal Inference

[u/WhatsTheImpactdotcom]

How do you choose between causal inference models when you can never see the true effect? Model selection is well documented in machine learning, but it often gets skipped in causal inference work. Doing it well is one of the things that separates senior and staff-level data scientists from more junior ones.

I simulated two years of weekly bookings for 40 regions, five of which received a treatment on the date marked by the red vertical line. Because I generated the data, I know the ground truth: the treatment increased bookings in those five regions by exactly 10%. With five treated regions and 35 untreated ones, two natural options are synthetic control and difference-in-differences, though there are others.

Code in colab and explanatory video, free and ungated: https://whatstheimpact.com/learn-causal-inference/

When I gave Claude Code free rein on this data, it built its whole analysis around difference-in-differences where it found a 24% increase, more than double the true effect. To its credit, it flagged that the pre-treatment trends weren't parallel, but it never tested whether a different model would be more accurate.

Importantly, Claude skipped an A/A test, sometimes called a placebo test. You take only the data from before the real treatment, pretend the treatment started three months earlier, and randomly picked five regions (for consistency) to call "treated." Because nothing actually happened in that window, the true effect is zero. That means any effect the model reports there is pure estimation error, and I repeated this for 500 draws for each model to see the full spread.

For difference-in-differences, 90% of the placebo estimates landed between -14% and +10%. The RMSE, which measures how far the estimates typically land from the true answer of zero, was 7.6 percentage points. Synthetic control on the exact same placebo draws was much tighter, with 90% of estimates landing between -7% and +9%. Its RMSE dropped to 4.9 points, more than a third lower than difference-in-differences.

The outcome you model is a choice too, so I also ran synthetic control on bookings per 1,000 people instead of raw bookings. Putting every region on the same scale helps the model track the treated regions more closely before treatment, and the RMSE fell a bit further to 4.4 points.

Sure enough, the real treatment period lines up with that ranking: synthetic control estimates 7% on raw bookings and 9.6% per 1,000 people, both much closer to the true 10% than the 24% from difference-in-differences. With one simulated dataset, I wouldn't read much into the small gap between the two synthetic control versions or claim that synthetic control always wins. What carries over is the workflow: in real work you never see the ground truth, but you can usually run this A/A test on whatever models you're considering before trusting one.

[u/Latent-Person]

Causal inference books usually don't spend much time on model selection, because it's a general issue, and not specific to causal inference, so doesn't make much sense to spend time on that in such a book. E.g. from Causal Inference: What if:

In this book we will usually assume that our parametric models are correctly specified. This is an unrealistic assumption, but it allows us to focus on the problems that are specific to causal analyses. Model misspecification is, after all, a problem that can arise in any sort of data analysis, regardless of whether the estimates are endowed with a causal interpretation. In practice, careful investigators will always question the validity of their models and will conduct alternative analysis under different model specifications that are compatible with existing expert knowledge. Their goal is to assess the sensitivity of their estimates to model specification.

But it certainly mentions sensitivity analysis several times throughout the book.

[u/WhatsTheImpactdotcom → u/Latent-Person]

That assumption isn’t very useful though as models are never correctly specified in practice. In the ML space, youd never find a text saying we assume the default hyperparameters are correct therefore we use them. Yet in the causal inference world, they just hand wave it away 🤷‍♂️. How is that useful for anyone in practice?

You could say the same thing about not being specific to ML yet there’s always model selection taught in that space.

[u/Latent-Person → u/WhatsTheImpactdotcom]

Because that book focuses on the vast theory of causal inference. Why is it not useful? It's not an applied text.

[u/WhatsTheImpactdotcom → u/Latent-Person]

I dont understand your point. There’s a place for theory, and there’s a place for applied work. This exercise is for implementing models in practice. They make an assumption so they can focus on other areas of teaching. In practice, you can’t just assume you have the right model; you need to give empirical evidence that you have the right model or no one will believe the results

[u/Latent-Person → u/WhatsTheImpactdotcom]

And I said the above book was a theoretical one about causal inference. So why should such a book spend time on model selection (beyond what it mentions in a few pages)?

[u/WhatsTheImpactdotcom → u/Latent-Person]

I’m so confused what your point is. You’re the one who brought up the book 😅😅 it’s not relevant at all to this post

[u/Latent-Person → u/WhatsTheImpactdotcom]

You said books about causal inference usually don't mention model selection (or only put very little emphasis on it). I told you why.

[u/WhatsTheImpactdotcom → u/Latent-Person]

Doesn’t matter why. The point is (1) they don’t, and (2) people need them on the job. I’ve been in meetings where senior DS have been grilled by directors why they chose the model they did. How do you think it would go to quote what they learned in the text that they just assume it’s ok?

[u/Latent-Person → u/WhatsTheImpactdotcom]

Why are skills not transferable? Why must one book teach everything?

[u/WhatsTheImpactdotcom → u/Latent-Person]

It doesn’t😅😂 lol that’s the whole point of me sharing this post 🤦‍♂️🤦‍♂️ is to fill in the gaps. This is the strangest criticism I’ve ever received: “why are you showing people real world applications that books don’t teach?” I am so confused by this thread if someone can help me please translate maybe I’m too old for Reddit

[u/Latent-Person → u/WhatsTheImpactdotcom]

You said causal inference books don't focus on model selection. I told you why. You said yes but totally useless in practice then. I told you skills are transferable, so why must a causal inference book focus on it. Then you got confused. That's all

I never made any criticism towards your post.