The Gino-Colada Affair

Abstract

This short note reproduces analysis found on Replicability-Index.

1 Study #1

We can reproduce the equivalent of the published results of an ANOVA with the three conditions as categorical predictor variable and deductions as outcome variable using linear regression. Results are reported in column (1) of Table 1. In addition, the original article reported that each difference between the experimental “signature-on-top” and the two control conditions (“signature-on-bottom”, “no signature”) was significant. This is confirmed in columns (2) and (3) of Table 1.

Next, we can repeat the analysis without rows 67 to 72. Results are reported in Table 2. Without the six contested cases, the results are no longer statistically significant, \(F( 2, 92) = 2.96\), \(p = 0.057\), as seen in column (1) of Table 2. The comparisons of the experimental group with the two control groups were also statistically significant (see columns (2) and (3) of Table 2). Combining the two control groups into one and comparing it to the experimental group and presenting the results as a planned contrast would also have produced a significant result (see column (4) of Table 2).

Of course, the accusation is that she switched rows with low values to the experimental condition and rows with high values to the control condition. To attempt to reverse this manipulation, we can recode the contested rows 67–69 as signature-at-the-bottom and 70–72 as signature-at-the-top and repeat the analysis. In this case, there was no evidence that the group means differed from each other, \(F( 2, 98) = 0.454\), \(p = 0.637\). Results are presented in Column (1) of Table 3. Neither comparison of the experimental group with each of the two control groups was statistically significant (see columns (2) and (3) of Table 3).

Table 1: Reproduction of results from the paper

	(1)	(2)	(3)
(Intercept)	8.445	5.271	8.445
	(0.966)	(0.906)	(0.906)
Cond1	-3.174		-3.174
	(1.346)		(1.263)
Cond2	1.179	4.353
	(1.366)	(1.300)
F	5.633	11.203	6.312
p	0.005	0.001	0.014

Table 2: Reproduction of results without disputed observations

	(1)	(2)	(3)	(4)
(Intercept)	8.445	5.703	8.445	8.417
	(0.895)	(0.827)	(0.909)	(0.645)
Cond1	-2.742		-2.742
	(1.276)		(1.296)
Cond2	-0.059	2.684
	(1.297)	(1.189)
Cond == 1TRUE				-2.714
				(1.110)
F	2.956	5.098	4.478	5.975
p	0.057	0.028	0.038	0.016

Table 3: Reproduction of results with corrected data

	(1)	(2)	(3)
(Intercept)	8.445	7.100	8.445
	(1.015)	(0.979)	(1.062)
Cond1	-1.345		-1.345
	(1.415)		(1.480)
Cond2	-0.761	0.585
	(1.436)	(1.405)
F	0.454	0.173	0.826
p	0.637	0.679	0.367

2 Study #2

Table 4: Study 2: Reproduction of results with OSF data

	Cheating	SumDeductions	SumEthicsWords
(Intercept)	3.567	7.063	0.867
	(5.491)	(7.266)	(4.720)
SignAtTop	-1.900	-3.830	0.533
	(-2.068)	(-2.786)	(2.054)
F	4.279	7.761	4.218
p	0.043	0.007	0.045

The original results from the paper are reported in Table 4.

From the calcChain.xml file, it appears that just three observations (P# values 1, 59, 61) have been moved “out of order” from “sign at the bottom” to “sign at the top”. These observations are (now) in rows 2, 60, and 61 of the OSF spreadsheet. It seems these changes involved moving a row from the bottom to the top and two rows from the top to the bottom.

<c r="I58" i="1"/>
<c r="K58" i="1"/>
<c r="I59" i="1"/>
<c r="K59" i="1"/>
<c r="I2" i="1"/>
<c r="K2" i="1"/>

and

<c r="I60" i="1"/>
<c r="K60" i="1"/>
<c r="I61" i="1"/>
<c r="K61" i="1"/>
<c r="I3" i="1"/>
<c r="K3" i="1"/>
<c r="I4" i="1"/>

Let’s see what happens if we move it back? Results are reported in Table 5.

Table 5: Study 2: Reproduction of results with corrected data

	Cheating	SumDeductions	SumEthicsWords
(Intercept)	3.375	6.950	0.906
	(5.314)	(7.390)	(5.066)
SignAtTop	-1.625	-3.861	0.487
	(-1.748)	(-2.804)	(1.858)
F	3.055	7.864	3.453
p	0.086	0.007	0.068