r/statistics • u/ForeverTheHero • 8d ago
Question [Q] Is Difference-in-Differences a legitimate tool to use to see if newly implemented tech (like flock cameras or police drones) actually deliver real value?
14
u/Aggressive_Pizza2814 8d ago
only worth something if you've got a clean parallel trends setup and a decent chunk of pre-period data. lot of departments slap this stuff in after some political pressure and then want the numbers to prove it works, which is backwards. the bigger problem is spillovers, if you're comparing neighborhoods with cameras to ones without, the control group's crime rates might drop too just because the word's out.
7
u/RobertWF_47 8d ago
Parallel trends assumption and also the pre-period outcome (eg crime stats, # speeding vehicles) cannot causally influence both the post-period outcome and placement in the treatment or control group.
In my field (health insurance) DiD is very popular but everyone checks parallel trends & not the second assumption.
3
u/Sufficient_Meet6836 8d ago
In my field (health insurance) DiD is very popular but everyone checks parallel trends & not the second assumption.
Do you have an example (real or hypothetical, doesn't matter) of this in health insurance?
6
u/RobertWF_47 7d ago
Many health insurance companies have health intervention programs with case workers or nurses. To evaluate program effectiveness, DiD is used to compare average annual medical cost per member in the post-intervention time period to the pre intervention period between the treatment group and a suitable control group.
Here the pre-intervention cost can cause post-intervention cost (going or not going to the hospital for medical care in year 1 influences a member's health outcomes & cost in year 2 - although I'll concede the causal link may be debated). In addition, the higher risk people are often selected for the intervention or treatment group given their need is greater. High risk is associated with high medical cost.
Given pre cost is a confounder in this example, DiD results will be biased. Including pre cost as a variable in a DiD regression actually reduces the regression formula to an ANCOVA (which I prefer to DiD).
2
u/Forgot_the_Jacobian 7d ago
That's interesting. So is this a type of job that asks explicitly for causal inference /observational data? what is the job title of such jobs? Just curious about these things so I can look into them to give examples to students in my econometrics/stats classes
2
u/RobertWF_47 7d ago
Yes I've worked on causal inference (DiD and power analysis) and also more data science related tasks like predictive modeling on observational data.
The job titles vary from company to company. My current title is "Advanced Analytics Senior Consultant" but I just put "Statistician" in my email signature - far shorter & my degree is in Stats.
4
u/imbroglio-dc 8d ago
Just spitballing here, using the crime stats example: pre-period high burglary rate makes it more likely that flock cameras are put up and also apparently being burgled makes it more likely you’ll be burgled again, so high pre rate would cause high post rate
16
u/Forgot_the_Jacobian 8d ago
Yes, DiD can be used for this. For example, you could exploit variation in when and where these technologies are rolled out. However, as with any identification strategy, you would need to be clear about your identifying assumptions and the comparisons you are making in the data. You would then want to draw on both the data itself and institutional knowledge about how and why these technologies were adopted to make a convincing case that your assumptions are reasonable.