In my previous post I bemoaned the variability in definitions of reproducibility and replication.
The definition of reproducibility in particular concerns me as I finalise my thesis on ‘Reproducible Research Pipelines’. However I also have noticed a confusion of Repeatability and Replication that is worth noting. In various author’s definitions. In CASSEY, P., & BLACKBURN, T. M. (2006). Reproducibility and Repeatability in Ecology. BioScience, 56(12), 958. http://bioscience.oxfordjournals.org/content/56/12/958.full they agree with the definition of Peng 2011 that Reproducibility is to re-calculate the same result from the same data and they use Repeatability as a synonym for Peng’s Replication.
According to Cassey and Blackburn (2006), reproducibility is the case that:
from the information presented in the study, a third party could
replicate (sic) the reported results identically.
This definition distinguishes reproducibility from repeatability which is when
a third party must be able to perform a study using identical methodological protocols
and analyze the resulting data in an identical manner
Which is what Peng terms ‘replicability’. This leads me to conclude that:
Cassey and Blackburn conflate Repeatability with Replicability.
However another author gives a very confused and overlapping view Ellison, A. (2010). Repeatability and Transparency in Ecological Research. Ecology. https://dash.harvard.edu/bitstream/handle/1/3123279/Ellison_Repeatability.pdf?sequence=2 Accessed 12 Jan 2016.
To add further confusion I have dug up the latest dictionary of epidemiology and find that whilst this agrees with Reproducibility and Replication, it treates Repeatability as a synonym of Reproducibility!
- Porta, Miquel S. Dictionary of Epidemiology (6th Edition). New York, NY, USA: Oxford University Press, USA, 2014. ProQuest ebrary. Web. 24 January 2016. Copyright © 2014. Oxford University Press, USA. All rights reserved.
Repeatability (Syn: reproducibility): The value below which the
absolute difference between two single test results may be expected to
lie with a probability of 95%, when the results are obtained by the
same method and equipment from identical test material in the same
setting by the same operator within short intervals of time. A test or
measurement is repeatable if the results are identical or closely
similar each time it is conducted. 1-3,5-9,91 See also measurement,
terminology of; reliability.
Replication: The execution of an observational or experimental study
more than once so as to confirm the findings, increase precision, and
obtain a closer estimation of sampling error. Exact replication should
be distinguished from consistency of results on replication. Exact
replication is often possible in the physical sciences, but in the
health, life, and social sciences consistency of results on
replication is often the best that can be
attained. 1,2,6,25,39,42,91,206-208,270,273,533 Consistency of results
on replication is perhaps the most important consideration in
judgements of CAUSALITY.
Reproducibility: see REPEATABILITY.
Whatever happens, never cite Drummond 2009!
It is unclear whether Drummond’s self-published conference paper was peer reviewed (or reviewed by the conference committee) but the following quote is unsupported assertion and should be ignored.
Reproducibility requires changes; replicability avoids them.
Drummond, C. (2009). Replicability is not Reproducibility: Nor is it Good Science. Proceedings of the Evaluation Methods for Machine Learning Workshop 26th International Conference for Machine Learning. Retrieved January 25, 2016, from http://cogprints.org/7691/7/icmlws09.pdf
Indeed in 2012 Drummond (in another self-published, working paper without evidence of peer review) did a backflip and said:
Reproducibility requires that the experiment originally carried out be duplicated
as far as is reasonably possible. The aim is to minimize the difference from
the first experiment including its flaws, to produce
independent verification of the result as reported
Drummond, C. (2012). Reproducible research: A dissenting opinion. Unpublished draft. Retrieved October 9, 2015, from http://cogprints.org/8675/
