What a Journal of Finance retraction gets right
In July 2021, the Journal of Finance published Paul Guest’s “Risk Management in Financial Institutions: A Replication”. Guest examined an earlier paper by Rampini, Viswanathan, and Vuillemey (RVV), published in the same journal, and identified six discrepancies in its reporting, coding, and data. The authors of RVV retracted their paper, acknowledging that its conclusions could not be relied upon.
It is tempting to treat a retraction as an embarrassment for a field. But the absence of retractions is not evidence that all is well. Errors can remain undetected, or they can be detected without the published record being corrected. In this case, a problem was investigated and the record was corrected. That part of the story is a positive sign.
A retraction does not, by itself, establish whether an unreliable result arose from an honest mistake, carelessness, or fraud. Those distinctions matter, but they do not change the need to correct the literature. Two aspects of this case are especially relevant to making correction more likely.
Lowering the cost of replication
Guest had access to data and code supplied with RVV’s paper. These materials made it possible to examine how the reported results had been produced and to investigate discrepancies.
The story is more informative than a simple failure to run the code. Guest reproduced many results, but found problems in the reporting, code, and data that affected the interpretation of the findings. Correcting those problems undermined the paper’s central conclusions. The retraction notice also reported that the replication code did not fully reproduce central published results and that the original materials that would resolve the problem could not be supplied.
This illustrates why a data-and-code policy should be judged by what the deposited materials allow someone to do. An archive containing a few scripts is not enough if no one can connect those scripts to the tables in the paper. Useful materials reduce the work required to reconstruct that connection and make it possible to investigate whether the analysis supports the claims made for it.
The cost of starting from scratch can be prohibitive. Anyone who has tried to replicate a paper from its prose alone will appreciate the difference.
Increasing the payoff to replication
Guest’s replication appeared in the Journal of Finance itself. This matters because replication takes time, and researchers respond to the professional rewards for spending that time.
Two accounting examples available when I wrote this note in 2021 offer a contrast. Alex Young’s 2018 critique of a paper in The Accounting Review appeared in Econ Journal Watch. The paper he criticised was subsequently retracted. Stephen Walker’s 2021 critique of the machine-learning paper by Bao et al. in the Journal of Accounting Research also appeared in Econ Journal Watch, alongside a response from the original authors in the same issue.1
There is nothing wrong with publishing a critique in another journal. But a journal’s willingness to publish serious replications of its own papers gives researchers a clearer reason to undertake them and puts the findings in front of the original audience. The Journal of Finance’s policy on replication papers provides one model.
These two changes work together. Usable data and code lower the cost of checking a paper. A credible prospect of publishing a careful replication increases the payoff. More effective data policies and a willingness to publish replications and corrections seem to be modest reforms that more journals should consider.
Footnotes
I discuss the Bao et al. paper in more detail in a later note.↩︎