Difference in Differences and Beyond

Preface

Difference in differences (DD) is one of the most popular approaches in economics and other disciplines of social science. An article (November 26, 2016) in The Economi , entitled “Economists are prone to Fads, and the latest is machine learning", analyzed the most frequently used techniques in economics. The analysis was based on key words in the abstracts of NBER working papers, and the most popular methods turned out to be DD, followed by regression discontinuity (RD), laboratory experiment, dynamic stochastic general equilibrium, randomized control trial, and machine-learning/big-data. According to the article, DD has been at the top since 2012, and its popularity has been increasing ever since, unlike some other methods such as dynamic stochastic general equilibrium whose popularity has been declining. Not just in social science, DD has been gaining popularity also in natural science disciplines, as can be seen in Jena et al. (2015), Cataife and Pagano (2017), Uber et al. (2018) and McGrath et al. (2019),
among many others.
There are various existing references for DD: Angrist and Krueger (1999), Shadish et al. (2002), Lee (2005, 2016a), Angrist and Pischke (2009), Morgan and Winship (2014), among many others. Most references are, however, either a little too long or too old. This short book introduces DD to readers equipped with basic graduate-level econometric knowledge and some exposure to panel data, and then examines recent advances in DD from a personal perspective. This book is an extended version of Lee and Sawada (2020), which in turn draws on Lee (2005, 2016a).
More specifically, first, details on DD identification and estimation using panel data and repeated cross-sections are provided for various DD cases such as constant/varying treatment effect and constant/varying treatment timing. Following these basics, topics such as ‘DD in reverse’, ‘fuzzy DD’, ‘synthetic control’, and ‘triple and generalized dif- ferences’ are examined. Throughout this book, many empirical examples appear, and long examples carry an explicit heading whereas short ones are buried in various parts without heading. There are parts with * attached, which are optional. It would be a good idea to skip those at the first reading because they are relatively more involved, and read them later when better motivated to learn DD. For readers in need of a quick re-fresher on treatment effect analysis, the appendix explains the basics and reviews various treatment effect estimators in cross-section context.
Difference in differences is often abbreviated as ‘DiD’ in the literature, but we use instead the simpler notation ‘DD’. Also, ‘difference in differences in differences’ is often abbreviated as ‘DiDiD’, but we use again a simpler notation ‘triple differences (TD)’. TD is a generalization of DD because it is a difference of two DD’s, where the difference can be taken between two groups (‘cross-section group-wise TD’) or between two time periods (‘time-wise TD’). There are other ways of generalizing DD, all of which may be called ‘generalized DD’.
One topic in DD that is not addressed in this book is the ‘DD inference issues’ involving ‘clustering/grouping’ that observations are related to one another by sharing the subject/individual index i (i.e., belonging to the same subject/individual), the time index t (i.e., belonging to the same period), or something else such as age or residential area. A treatment varying only at an aggregate level, not at the individual level, raises yet another DD inference problem. We do not address these inferential problems to keep this book’s technical difficulty at a reasonable level. Interested readers may refer to Bertrand et al. (2004), Lee (2016a), Brewer et al. (2018) and references therein, where the main message for DD inference seems to be “use at least panel generalized least squares estimator with a clustered variance estimator to account for serial correlations (and others)."
It is not too far-fetched to say that the main use of DD is in policy analysis, where finding effects of a policy/program/treatment on a response/outcome variable is the goal; e.g., effects of classroom size on test scores, and effects of minimum wage on employment. Although machine learning and big data analysis are gaining popularity these days, they are essentially for prediction and association, whereas policy analysis is for causal effects. The examples below illustrate well the difference between prediction and causal analysis, which gives a good reason to study causal analysis in the era of big data, and DD will always have its place in policy analysis, if nowhere else.
Suppose we use big data on individual physical attributes such as height, weight, waist-to-hip ratio, body symmetry, etc. to predict income, using ordinary least squares estimator (OLS) or one of the sophisticated machine-learning techniques; this kind of study has been done in ‘beauty economics’. The OLS result gives a prediction equation for income using those physical attributes. This is, however, not a causal analysis, because one may alter his/her body to maximize income according to the OLS result, but that would not make him/her rich. A big belly may be an outcome, not a cause, of having much money, and artificially making a belly big would not make the person rich. High levels of HDL cholesterol are associated with low risks of cardiovascular dis- eases. However, randomized studies (e.g., Schwarz et al. 2012) showed that increasing only HDL with a medication does not change cardiovascular disease occurrences. This means that the association is not causal. Probably there is something else changing
along with HDL cholesterol level, which affects cardiovascular disease occurrences.
The best prediction or association equation obtained using machine learning and big data should not be mistaken as something that reveals causal effects. Differently from prediction or association, if we find the causal effect of waist-to-hip ratio on marriage to be negative, then by reducing the waist-to-hip ratio, one can increase the probability of marriage. This is what ‘structural-form’ causal analysis can do, but ‘reduced-form’ prediction analysis cannot. Econometricians and statisticians may lose many jobs to artificial intelligence and machine learning in the future, but causal analysis will remain “human, not humanoid," subjects.
This book and its source, Lee and Sawada (2020), grew out of the lectures that I have been giving in many institutes. I am grateful to the lecture audiences for their feedback at Asian Development Bank, Australian National University, Korea Development Institute, Korea Institute of Public Finance, Seoul National University, and University of Luxem- bourg, which led to greatly improving the book manuscript. Also, Wonjun Choi, Hyerim Kim, Goeun Lee and Sanghee Mun proofread the book manuscript and provided help- ful comments. This research has been supported by a grant (NRF-2014S1A5B1014360) from the National Research Foundation of Korea.
Finally, a “warning" on using the index is warranted. Constructing an index is never a straightforward business. For example, ‘effect on the treated’ can make a single entry of its own, or may become a sub-entry ‘on the treated’ below the primary entry ‘effect’. Going further, ‘conditional effect on the treated’ may make a single entry of its own, or may become a sub-sub-entry below the sub-entry ‘the treated’ and the primary entry ‘effect’; it can also be a sub-entry ‘conditional’ below the primary entry ‘effect on the treated’. It would be nice to be coherent on this matter throughout the entire book, but this is not easy, as things look different at different times at different locations. So, when looking up some topic in the index, the advice is try different options as illustrated just now.