But not all research is bad

“But not all research is bad.”

I have heard versions of this response in discussions of the quality of accounting research. Someone acknowledges that p-hacking is a problem, or that much of the published literature is unconvincing, but then offers a reassurance: some research is good. I have written elsewhere about p-hacking and accounting research. Here I want to consider why this particular reassurance is so unhelpful.

The problem is not whether good research exists. The problem is whether we can identify it.

Chocolate and other brown stuff

Consider a metaphor. Good research is high-quality chocolate. Research whose apparent evidence has been manufactured by undisclosed analytical choices is doggie doo-doo, or “DS” for short. The literature is a pile of brown stuff, and the “some research is good” argument tells us to eat the chocolate and ignore the DS.

I confess I don’t understand the advice. How do we distinguish the two? If the distinction is obvious, why is the DS being published in the first place?

In my experience, people offering this reassurance often struggle when asked to identify the chocolate, especially if they are constrained not to nominate their own research. The claim that some research is good can become an article of faith that avoids the harder question of how a reader should tell.

P-hacking makes that question particularly difficult. A paper can present a hypothesis, an empirical test, and a statistically significant result without revealing the order in which those things arose. Was the hypothesis specified before the analysis, or was it written after trying enough analyses to find a result? How many outcomes, samples, controls, and specifications were considered before the reported version was chosen? The finished paper may give us no way to know.

Exploration is a legitimate part of research. The problem is presenting a result selected through exploration as though it came from a test specified in advance. Nor does p-hacking establish that the conclusion must be false. It undermines the claimed evidential force of the result: a small reported p-value means less when the process that selected it is hidden.

Suppose, for illustration, that 90% of a literature were p-hacked and readers could not tell which 90%. That is a hypothetical, not an estimate of the prevalence of p-hacking in accounting. The existence of good work in the remaining 10% would offer little practical reassurance to someone trying to learn from the literature. A pile of DS studded with chocolate chips is still not an appealing meal.

Why does the distinction matter?

Some memorable demonstrations show how apparently persuasive statistical evidence can be produced for absurd conclusions. Bennett and colleagues found apparently significant activity in an fMRI scan of a dead salmon when they failed to correct for multiple comparisons. Simmons, Nelson, and Simonsohn (2011) used flexibility in analysis to produce a result suggesting that listening to music made people younger. And John Bohannon’s chocolate-and-weight-loss hoax showed how a flimsy statistical result could be packaged as a scientific finding and promoted in the media.

These were demonstrations or deliberate hoaxes, not three ordinary studies whose authors sincerely believed the advertised conclusions. Their value is to show what can go wrong when readers see the selected result without understanding the process that produced it. The salmon example concerns multiple testing; the others illustrate related opportunities to select and promote a favourable result.

One response is that we can use prior knowledge to reject implausible findings. We do not need an elaborate statistical argument to doubt that a dead salmon is responding to photographs. But the difficult cases are the plausible findings. A result that fits a familiar story can be produced by the same unreliable process. If our rule is simply to accept findings that agree with what we already believe, what are we learning from the research?

Prior knowledge is useful. It cannot substitute for evidence about how the study was conducted.

Who cares?

Another response is that p-hacking is less serious than outright fraud. There may be important moral differences between the two, but both can leave readers with evidence that is much less informative than it appears. The scientific problem does not disappear because the conduct falls short of fabrication.

What would make “some research is good” a useful response? We would need reasons to identify particular work as good: transparent reporting of analytical choices, a clear distinction between exploration and tests specified in advance, usable data and code, and findings that survive independent scrutiny. None of these is a guarantee. Together, they give readers something more useful than a journal’s prestige or a result’s plausibility.

Occasionally, the response is more cynical: research is not really about accumulating knowledge; it is about deciding who gets tenure at the good universities. But if the rewarded skills are collecting data, massaging them until results pop out, and telling a cute story about those results, the problem extends beyond the literature. Are those the skills we want selecting the people who teach MBAs and future accountants?