We had an excellent colloquium speaker today as Dan Sholler, a postdoctoral researcher at Berkeley, presented results from his work on open science initiatives and the challenges involved in ensuring reproducibility and replicability in practice. His work involved interviews with key players in journals and research programs and raised in my mind some serious questions about the almost blind rush to adopt open practices.
I think we can all accept that it is important for science to justify claims and allow appropriate data to be shared, especially where public funds have enabled the research, but I have to ask what exactly is it that we are trying to ensure? Outside of the very real concern about the narrowness of what constitutes ‘science’ in this movement (clue – it has to involved lots of numbers), and the effort involved in adding another layer of effort to support the further checking of results and data sharing, what would we gain here?
Many (most?) research studies are dull and let’s be honest, don’t make strong claims at advancing our knowledge. Should we insist that their data be provided anyway? Where claims are major, there’s already a process within normal science practice to require replications and further studies to support the claims. For such work, I think the values of reproducibility and replication are high and normally expected. But such work is not the norm. Applying the same standards for all work that we require for strong claims is likely more a burden than an enhancement.
More pernicious to me is the problem of fraud. I can engineer data to fit results if I want, and then sharing that data with another researcher who replicates my analysis will only add confirmation to a lie. I’d like open science to shine a light here. Right now, offering badges for good practice and letting the workload fall on reviewers and coders to check data deposits is likely to result only in slowing down all science.
Food for thought today, definitely, and another reminder that an ‘obviously good thing’ often comes with a price. Now, where can I find a good measure for data quality?