Lessons From Urban and Environmental Economics in Interpreting Randomized Field Experiments
John Cochrane’s great recent Substack caused me to write this piece about Causality and what we learn from academic field experiments.
To keep this Substack interesting, I want to sketch out an undergraduate lecture I give about where Superman and Average Joe choose to live in a city and what conclusions we can make about how pollution causes sickness. This example is written up in my free textbook on page 106.
PHD economists will see that this is an example of Jim Heckman’s essential heterogeneity IV and I will return to that point below.
The Setup
There is a dirty factory located in the center of a city. It creates jobs and pollution. The pollution lands locally and makes the communities close to the factory really nasty but the commute to the city center is short.
There are two types of people; Superman does not suffer at all from pollution, while Average Joe suffers greatly from pollution exposure. Each type of person knows his own type (Heckman’s essential heterogeneity) but the applied micro researcher does not know who is who!
Hedonic rents for housing rise with distance from the City Center. Why? While commute times rise as a function of distance, pollution exposure declines and this disamenity dominated the commute effect because most of the population are Average Joes (perhaps 80% Average Joes and 20% Supermen).
The Puzzle For the Applied Micro Scholar
At a point in time, in this economy, pollution seems to be good for one’s health!! Why? The average person who lives close to the dirty factory in the dirty communities is sick fewer days than the average person who lives further from the factory in the clean air, suburban areas.
The true data generating process is one of comparative advantage. Supermen arbitrage the cheap rent and locate close to the factory and they never get sick! Average Joes do get sick and they happen to live in clean air places and the naive researcher concludes from this correlation that pollution causes health!
The Missing Data problem here is that we do not observe what would be the health outcomes for residents of the center city neighborhood if they moved to the suburbs AND we don’t observe the health outcomes for suburban residents if they moved to the center city.
Two Field Experiments
#1 In the spirit of Raj Chetty’s work on Move to Opportunity, suppose the residents of the polluted community close to the factory are offered a randomized rental voucher subsidy to move to the suburbs. Some Supermen would accept and they would keep being just as healthy in the suburbs. This new evidence would lead the public health Causality researcher to conclude that for a segment of the population that pollution has no average causal effect on health.
#2 The IRB would block research offering subsidies to suburban residents to move to polluted areas. This would be a forbidden experiment! But suppose that a sneaky researcher figured out how to run this experiment and suppose that 200 Average Joes do move to the center city polluted area and get zapped by the local pollution.
My Key Point
Suppose that Matt and Dora are two Average Joes who know how pollution affects our health and don’t want to be sick more often but because Matt hasn’t published in the Top 5 journals for a few years now, they need to lower their living expenses so they agree to accept the 40% subsidy offered by J-PAL North America to move to the high pollution area. They do save commute time here and their rent is really low.
Matt and Dora are not passive victims here. They know that they are in their early 60s and they used Grok to get up to speed about how their health may be affected by pollution exposure. Given the current technology sold by Amazon, Matt and Dora invest in some self-protection clothing and masks and PM2.5 monitors and they move in BECAUSE of the field experiment’s incentive.
GIVEN that field experiment researchers can only study the treatment effects for those who accept a randomized offer, the researcher will still have a select sample to study. The treatment effect of living in a polluted place on Average Joe (Matt and Dora)’s health depends on which subset of the randomly invited treatment group accepts the offer (This is Heckman’s essential heterogeneity point). This in turn depends on the size of the incentive and on Matt and Dora’s personal calculations of the benefits and costs to us of remaining in the suburbs or moving to the polluted city. If we think we can offset a lot of the pollution through our Amazon purchases, then this is an easy decision for us (we have been randomly assigned a 50% subsidy on the rent). If some firm in China makes a great air filter and Matt can buy this on Amazon, then the causal effect of pollution on this Average Joe’s health will decline even closer to zero! (non-separabilities in the production of health!! The field experiment researcher doesn’t observe our efforts at self protection).
So, what question does the randomized field experiment answer here?
Note that our actual air pollution exposure has not been randomized! We have an expectation that even if air pollution in our new center city neighborhood is high that we can invest in self-protection to offset perhaps 80% of this threat. The empirical researcher doesn’t know our expectation of this 80% number. Even if the researcher sees how many extra days Dora and I are sick each year (relative to our clean suburban baseline) in our new polluted neighborhood, the researcher does not have a denominator in the following calculation;
(Sick Days_dirty_post - Sick Days_clean_pre)/(Pollution_dirty - Pollution_clean)
So the numerator represents Matt’s average Sick days per year before and after moving to the dirty area; The denominator represents his own increase in exposure to pollution by being induced to move to the dirty area because of the randomized incentive.
The researcher can observe the outdoor air pollution but this doesn’t represent Matt’s self protection actual exposure. It provides an upper bound on his true exposure.
EVEN worse for the field experiment design; if more people susceptible to pollution choose to live close to pollution sources and are aware of their baseline extra sickness risk caused by pollution; endogenous innovation kicks in and we develop medicines, and more face mask protection to further reduce our exposure and the health consequences of the remaining exposure.
In this sense, the anticipation of a short run treatment effect from pollution on Matt’s health leads millions of Matt’s to shift out the aggregate demand for solutions and entrepreneurs innovate. The field experiment crew can’t capture these dynamics.
Punchline?
Young economists have career concerns and they know that field experiment designs help them to publish Top 5 papers. Such researchers tend to downplay that the production functions they are studying are not additively separable and time invariant.
They want to write down models of the form;
Effect = a + b*cause + b2*controls + U and then randomly vary the cause.
My points here? Since people in the economy know their own “b”, they sort based on this and they play defense against ugly effects by either hiding out in the suburbs (in this example) or buying products to self-protect. The feedback effects from subjects to society actually form the most interesting basis of field experiment research.
Young researchers think that their causal research designs improve public policy. A capture theory of regulation would tend to chuckle at that thought. I think that the main benefit of randomized field experiments is to help for profit firms to learn about the latent desires of a heterogeneous population to handle emerging challenges. I also do appreciate that information nudges can help individuals to reduce their own inefficiencies in their household production. This was the point of my 2013 field experiment with Frank Wolak. This was also the point of my Flood Risk paper with Sebastian, Daryl and Rob.
If this general discussion interests you, you can listen to my talk titled; “The Causal Effects of Causal Effects”. Headlines in the New York Times saying that wildfire smoke kills zillions actually helps us to adapt by encouraging more Average Joes to research the threat and to increase investment in self-protection. In this sense, the Causality revolution in environmental and Urban economics acts as a type of “Paul Revere” triggering behavioral change. This is the micro Lucas Critique at work!!
UPDATE: Consider another source of pollution exposure variation. Consider a natural experiment induced by a recession such that the polluting factory reduces its production by 30% per year. In this case pollution declines in the city and the Average Joes might move closer to the City Center and spend less per-year on self protection products. In this case, there will be less pollution in the city but the Average Joes’ health may barely improve because they have chosen to take the “windfall” in local amenity improvements (the cleaner air) and translating this into more disposable consumption. The causal effects that researchers recover crucially depend on how different people re-optimize given a change in the local rules of the game.