What if you could untangle not just whether a policy works, but also *how* and *why*—all while harnessing the flexibility and predictive power of modern machine learning? That’s exactly the promise when you combine difference-in-differences (DiD), mediation analysis, and double machine learning (DML). This intersection is at the cutting edge of causal inference, offering a way to rigorously dissect mechanisms in complex, high-dimensional settings. But how does it actually work, and what can you expect in practice?
Short answer: Difference-in-differences can be used for mediation analysis with double machine learning by leveraging DML’s ability to flexibly adjust for confounders in high-dimensional data, while DiD provides a quasi-experimental framework for causal identification. By explicitly modeling both the direct and indirect (mediated) effects of a treatment—using machine learning methods to estimate nuisance parameters—researchers can robustly estimate how much of a treatment’s effect operates through a proposed mediator, even when many covariates are involved or when treatment effects are heterogeneous.
Let’s break down how these tools fit together, why this approach matters, and what the key steps and challenges are.
Difference-in-differences is a staple of applied econometrics for estimating causal effects when randomized control isn’t feasible. It compares changes over time between a treatment group and a control group, relying on the “parallel trends” assumption—that, absent the intervention, the two groups would have evolved similarly. DiD is popular because it’s conceptually straightforward and robust to certain forms of selection bias, especially when combined with panel data.
Mediation analysis, on the other hand, digs deeper than the average treatment effect. Instead of just asking “does this work?”, it asks “how does this work?” The goal is to decompose the total effect of a treatment into components: the part transmitted through a specific mediator (the indirect effect) and the part that’s direct, not explained by the mediator.
Double machine learning is a recent advance in causal inference that allows for robust estimation in the presence of many covariates—sometimes even more covariates than observations—by using machine learning methods to estimate nuisance functions (like propensity scores or outcome models), and then “double debiasing” the final estimator to ensure valid statistical inference. According to the NBER Working Paper by Chernozhukov, Demirer, Duflo, and Fernández-Val, DML works by repeatedly splitting the data to avoid overfitting and by aggregating results across many splits, which “lowers estimation risks over a single split procedure.”
Why Combine Them? The Motivation
Why bring machine learning into mediation analysis in a DiD framework? Traditional mediation analysis (as developed by Imai, Tingley, and Yamamoto and discussed in the NBER methods lectures by Chetty and Imai) often assumes simple linear models and a small number of covariates. But in real-world policy settings—think education, health, or labor markets—there are often dozens or hundreds of relevant variables, and the relationships between them are rarely linear or additive.
Machine learning methods like random forests, neural networks, or boosted trees can flexibly model these complex relationships. But as Mullainathan and Spiess explain in the Journal of Economic Perspectives (aeaweb.org), machine learning is typically about prediction, not causal inference. DML bridges this gap by providing a way to use machine learning for estimating causal parameters, not just predictions, while still allowing for valid inference even in high-dimensional settings.
This is especially valuable in mediation analysis, where accurate adjustment for confounders is crucial. If you want to know whether a job training program improves earnings because it increases self-confidence (the mediator), you need to account for all the factors that could confound both self-confidence and earnings. Traditional regression might miss nonlinearities or interactions, but DML lets you use powerful ML algorithms while still getting unbiased mediation effect estimates.
The Mechanics: How It’s Done
The workflow typically looks like this. First, you set up a DiD design: identify treatment and control groups, and observe them before and after the intervention. Next, you specify a mediation model: you want to estimate the effect of the treatment on the outcome both directly and indirectly through a mediator.
Here’s where double machine learning comes in. For each step—predicting the mediator from covariates, predicting the outcome from the mediator, treatment, and covariates—you use a flexible machine learning model (like a random forest or lasso regression). Crucially, you use sample splitting: part of the data is used to train the model, the other part to estimate the effect. This prevents “overfitting” the nuisance parameters, a key source of bias in naïve machine learning approaches.
According to the NBER paper, this process allows for “estimation and inference based on repeated data splitting to avoid overfitting and achieve validity.” By aggregating across many splits (for example, taking medians of p-values and confidence intervals), you get robust, well-calibrated estimates of both the direct and indirect effects.
The result is a decomposition of the DiD-estimated treatment effect: how much of the change in the outcome is mediated by the variable of interest, and how much is left as a direct effect.
Concrete Example: Immunization in India
To see this in action, consider the application from the NBER working paper, which looked at nudges to increase immunization rates in India. The researchers were interested not just in whether the intervention worked, but in *which households* were most affected and through what mechanisms (potential mediators like health knowledge or perceived convenience).
By applying their generic machine learning inference framework, they could estimate heterogeneous treatment effects—how the intervention’s impact varied across different groups—and also investigate which mediators accounted for the observed increases in immunization. The use of repeated data splitting and quantile aggregation (such as “medians of p-values”) ensured that their estimates were robust, even with many potential confounders and nonlinear relationships.
Key Insights and Challenges
Several important details and caveats emerge from this approach, as highlighted across the sources:
First, as Mullainathan and Spiess (aeaweb.org) emphasize, machine learning excels at capturing complex, nonlinear patterns, but its output must be interpreted with care. DML provides a principled way to use ML for causal questions, but the validity of mediation analysis still rests on strong assumptions: for example, that there are no unmeasured confounders of the mediator-outcome relationship, and that the parallel trends assumption holds in DiD.
Second, the NBER working paper underlines the importance of “building provably better machine learning proxies through causal learning.” In other words, you don’t just want the best prediction of outcomes, but the best models for the *causal effect* of the treatment and the mediator. This is where specifying the right objective functions and using cross-fitting become essential.
Third, the flexibility of the approach is a double-edged sword. As the arxiv.org literature on real-time monitoring of complex systems reminds us (albeit in a different context), when you have powerful, flexible tools that can be adapted “on our demands,” it becomes all the more crucial to have rigorous procedures for validation and inference. DML’s repeated splitting and aggregation are designed to address exactly this risk.
Fourth, this approach is particularly well-suited to situations with high-dimensional covariates or when treatment effects are heterogeneous. For example, if you want to know not just whether a job program works on average, but also whether it works differently for men and women, or for different education levels, DML lets you estimate these heterogeneous effects without overfitting.
Fifth, the NBER materials and Chetty & Imai’s lectures point out that mediation analysis with DML can be extended to “surrogate indices”—sets of mediators that collectively explain the treatment effect. This adds even more flexibility for real-world policy evaluation, where there may not be a single mediator but a bundle of mechanisms at play.
Summary: What You Gain and What to Watch Out For
Putting it all together, using difference-in-differences for mediation analysis with double machine learning allows researchers to robustly unpack causal mechanisms in complex, high-dimensional settings. This approach provides several key advantages: flexible adjustment for many covariates, robustness to overfitting, and the ability to estimate both direct and indirect (mediated) effects within a quasi-experimental DiD framework.
However, the approach also demands careful attention to assumptions and validation. As with any mediation analysis, the results are only as credible as the identifying assumptions—like no unmeasured confounding and parallel trends. The use of repeated data splitting and quantile aggregation, as described by Chernozhukov et al. (NBER), helps ensure that inference is valid, even when using powerful machine learning methods.
In sum, as machine learning becomes ever more prominent in econometrics, methods like DML provide a principled way to bring its predictive flexibility into the heart of causal analysis—answering not just “what works?” but “how, for whom, and why?” That’s a leap forward for evidence-based policy and social science.
