Is there a way to cope with the reproducibility crisis in science?
15/01/18 17:37
In the heart of scientific work is the strive to challenge and falsify existing hypothesis. Only by disproving old ideas new ways of thinking can be developed. To do so, scientists need in first step to be able to reproduce the results of other researchers: Same setups should give same results. However, many scientific disciplines are currently experiencing a “reproducibility crisis” because numerous scientific findings cannot be repeated consistently. Everybody certainly understands that reproducing results under real world settings is difficult. Imagine a forest ecosystem with dozens of tree species, hundreds of understory plant species and uncountable microbes; you do a measurement and the system and the environmental conditions are unique - you will never catch a point in time or space when the complex interplay is the same. That's why ecologists complement field work with experiments under controlled conditions. This is to reduce the complexity of the symphony of different biotic and abiotic players in order to make the system manageable. We like pots with defined soil, defined soil water availability and plants grown with defined distance among each others. We call these systems imitating parts of the real world "microcosms" let them grow in a climate controlled chamber and just manipulate one of multiple environmental conditions: Light or temperature or competition or…
This defined systems are claimed to produce results that are reproducible, but they are often not. They might be valid under given conditions (in Lab y with a climate chamber of company z, with an LED lighting system from supplier a) thus being local truths but not generalizable. Stringent levels of environmental and biotic standardization in experimental studies under controlled conditions might even reduce reproducibility among labs by amplifying impacts of location-specific environmental factors that are not (and often cannot be) accounted for in study designs.
We have now published a paper (Milcu et al. 2018) that aims at (at least partially) overcoming this problem by deliberately including variability in plant microcosm experiments in an ecological study. Such inclusion of controlled systematic variability has already been successfully adopted when assessing animal behavior.
Our results show that introducing genotypic controlled systematic variability (e.g. using not only one but different genotypes of a plant species) can increase reproducibility of results among different labs - the different genotypes of the grass species we used in our experiments might buffer the effects of the partially different and often unaccounted environmental conditions in different labs - i.e. genotype I might react to the slightly different light conditions between two labs intensively in one direction, the second genotype II in the other direction and genotype III might not react at all. On average, all three together won't be strongly affected. If you used only genotype I or II, inter-lab reproducibility would be lower. Even though counterintuitive at the first glimpse, deliberately including genetic variation may be a simple solution for increasing the reproducibility of ecological studies performed in controlled environments.
Milcu et al. 2018. Genotypic variability enhances the reproducibility of an ecological study. Nature Ecology & Evolution doi: 10.1038/s41559-017-0434-x
Preprint at bioRxiv
featured in Nature News: "Why 14 ecology labs teamed up to watch grass grow" Nature 548, 271 (17 August 2017) doi:10.1038/548271a 
This defined systems are claimed to produce results that are reproducible, but they are often not. They might be valid under given conditions (in Lab y with a climate chamber of company z, with an LED lighting system from supplier a) thus being local truths but not generalizable. Stringent levels of environmental and biotic standardization in experimental studies under controlled conditions might even reduce reproducibility among labs by amplifying impacts of location-specific environmental factors that are not (and often cannot be) accounted for in study designs.
We have now published a paper (Milcu et al. 2018) that aims at (at least partially) overcoming this problem by deliberately including variability in plant microcosm experiments in an ecological study. Such inclusion of controlled systematic variability has already been successfully adopted when assessing animal behavior.
Our results show that introducing genotypic controlled systematic variability (e.g. using not only one but different genotypes of a plant species) can increase reproducibility of results among different labs - the different genotypes of the grass species we used in our experiments might buffer the effects of the partially different and often unaccounted environmental conditions in different labs - i.e. genotype I might react to the slightly different light conditions between two labs intensively in one direction, the second genotype II in the other direction and genotype III might not react at all. On average, all three together won't be strongly affected. If you used only genotype I or II, inter-lab reproducibility would be lower. Even though counterintuitive at the first glimpse, deliberately including genetic variation may be a simple solution for increasing the reproducibility of ecological studies performed in controlled environments.
Milcu et al. 2018. Genotypic variability enhances the reproducibility of an ecological study. Nature Ecology & Evolution doi: 10.1038/s41559-017-0434-x
Preprint at bioRxiv
featured in Nature News: "Why 14 ecology labs teamed up to watch grass grow" Nature 548, 271 (17 August 2017) doi:10.1038/548271a