The Enduring Quality of My Coauthored 2022 Review of Financial Studies Paper
In this Substack, I will tell a true story about the recent retraction of my co-authored paper. I continue to believe that our paper is important and interesting and our empirical findings are robust. Our paper started a new literature. Our paper’s path from publication in a Top Finance journal to retraction offers several lessons for young scholars. While I am embarrassed by this sequence of events, I want to tell our story.
Upfront, let me vent. Our critics were funded by Fannie Mae and one of them was actually employed by Fannie Mae when they started writing their paper back in 2021. At the start of our research back in 2017, we even contacted him and his staff at Fannie Mae to discuss our hypothesis to see if they wanted to work with us on this project! They declined.
The authors of the paper that critique our paper found some errors in our paper that I will discuss below. You can decide whether these errors represent “hanging offenses”. I don’t think so. Their paper features many errors that we discuss in this video.
The Editor of the Review of Financial Studies did not allow us to correct our errors in print. Unlike at other Top Journals such as the AER, we were not allowed to reply to our critics.
The sad irony is that our empirical results are robust. Our reduced form findings are still statistically significant but some of our point estimates certainly change but a secret RFS Committee declared that our results do not replicate and our paper was retracted! I claim that this is a “cruel and unusual punishment”.
Fannie Mae will be delighted by this outcome. The RFS also gains as it will now have a “tough guy” reputation for expecting empirical work to be perfect but Amine and I lose out.
NOW the Story. The good news for us is that this story will have a happy ending.
_______________________________________________________________________
Back in 2022, the Review of Financial Studies (RFS) published my paper with Amine Ouazad. This paper has been well cited and it has made the national news with coverage in Bloomberg and the New York Times.
Since August 21st 2024, Amine and I have known that our paper faced an Expression of Concern. For four months, we waited for a response. We submitted detailed letters to the anonymous committee presenting new evidence documenting the enduring quality of our 2022 paper’s empirical findings.
On January 30th 2025, the RFS retracted our paper as an anonymous committee concluded that our results “do not replicate” .
Our paper’s reduced form empirical results presented in Table 2 have faced 3 main criticisms;
#1 loan limit coding
#2 rounding of conforming loan limits
#3 An implicit contract regarding our econometric specification versus the code actually implemented to generate the estimates in the Tables.
Our critics claimed that our treatment effects were not statistically significant when these three issues were properly addressed.
Our Response
#1 Loan Limits in high cost areas —- we admit that we made a mistake here and in all subsequent coding of these variables we will fix this issue. Our coefficient estimates do change when we fix this issue and some of the original treatment effects are no longer statistically significant but many of them continue to be.
#2 rounding of conforming loan limits —- we reject this claim and when we release our revised NBER paper in Summer 2025, we will document why our original approach was correct and we will present new evidence supporting our point here. In fairness to the RFS, in their final judgment on our paper they dropped this critique of our paper.
#3 Our critics point out a valid mistake here. In our defense, when we present our core treatment effect results under many different panel econometric specifications, the reduced form coefficient estimates are positive and statistically significant in the vast majority of the cases. We did not “cherry pick” a specification that made us look good. In our revised NBER paper in Summer 2025, we will discuss this point in detail.
_______________________________________________________________________
The Editor of the RFS was polite to us and he acknowledged that this process was painful for us. The process dragged on for over 5 months as we interacted with an anonymous RFS Review Committee.
Permit me to make the following points;
#1 We suggested that we be allowed to write a Corrigendum. This request was denied.
#2 We also received no report from the anonymous committee that ruled on our paper.
On December 27th 2024, we were notified by the editor of RFS that our paper would soon be retracted. I responded to his email and here is part of what I wrote him;
As a professional courtesy, can I ask you to answer the following questions?
#1 Why haven’t we been allowed to issue a Corrigendum? Why is the retraction punishment merited here? To us, this borders on cruel and unusual punishment.
#2 We cannot agree to your proposed statement because we reject your claim that we have used the wrong rounding method. In our letter from October 4th 2024, we presented you with our new McDash “IsJumbo” results that scientifically document that our approach yields less measurement error than the LLPW (2024) proposed solution. Can you share evidence with us that the RFS Committee deliberated on this point. Why is the LLPW (2024) study’s approach taken as the right way to proceed here? What is the scientific basis for their approach?
#3 Will the RFS Committee release a report so that the public can see the deliberations and the debate? We believe that this will be the RFS’s first retraction ever. This precedent case merits special detail and documentation.
#4 Can you confirm that you have watched our analysis where we delve into the errors embedded in the LLPW (2024) study? www.youtube.com/watch?v=EcSQWSiIGao
#5 How does the RFS Committee define 'replication'? Our results do replicate. In the F-tests provided in our October 4, 2024, letter, we demonstrate that the vast majority of our specifications reject the null hypothesis, even when incorporating the corrected county-level limits. Furthermore, Table A17 documents that in 6 of the 9 reported regressions, our results are jointly statistically significant after implementing both LLPW (2024) suggestions, as evidenced by the F-statistics.
#6 Gregor Matvos is the editor at RFS who accepted the LLPW paper. Can you confirm that neither Gregor Matvos nor Stijn Van Nieuwerburgh (the editor on our RFS 2022 paper) served on our RFS Review Panel? We believe that neither should have participated.
The RFS Editor did not answer any of my questions. Several famous economists who I briefed about this episode were deeply surprised that we have faced this retraction. One was the Editor of the QJE for years and the other was the editor of the RFS for years.
______________________________________________________________________
Our Next Steps
Everyone acknowledges that our estimates published in the 2022 paper can be replicated if you run our code using our data archive.
We acknowledge that we miscoded the loan limits in high cost areas. We have re-estimated our key Table 2 making this data edit. Of course, the regression coefficients change and some are no longer statistically significant based on t-tests. In our 2022 paper, we did not report F-tests. F-tests report a joint hypothesis test and we will report these in our next draft.
We have results already produced and shared with the RFS Review Committee that document that our core findings are robust across many different reduced form econometric specifications when we correctly code high cost areas. This defense did not sway the jury!!
In late Spring 2025, we will release a new updated version of our original NBER Paper and incorporate the discussion above related to the two criticisms of our paper and present our estimates including the F-tests. We will show the world the robustness of our reduced form findings. We will find a publication outlet who is willing to give our work a fair hearing.
Here is a link to our 2022 paper. Please read it, it contains many insights. The reduced form estimates will be updated and revised coefficients, t-stats and f-tests will be reported.
At the end of the day, I truly believe that we have made an important contribution to financial economics. We studied the impact of 15 major U.S hurricanes during a particular window of time from 2004 to 2012. We documented that during these years, that the GSE’s “rules of the game” encouraged mortgage lenders to originate conforming loans and to sell these to Fannie Mae and Freddie Mac. These entities do not engage in climate risk pricing and for profit banks take advantage of these rules to share risks with them. Our paper is well cited and has nudged other researchers to study related topics.
When I want to feel less pain from this experience, I think to myself that a retraction is “just a public rejection letter”. This thought helps a little bit.
Here is Amine Ouazad’s substack on this painful episode.