Sampling the literature: what’s gained and lost by taking a non-exhaustive approach to searching the evidence?
On this page:
Systematic assessments of evidence from the literature can drive research planning and inform environmental management. However, current approaches to systematic review are resource intensive, creating a barrier to wider uptake. One aspect where evidence assessment could be made more efficient is through using samples of the literature to test hypotheses, rather than a full census. We are using the data from an in-progress CEE systematic review to assess the ability of samples of the literature to reach the same conclusions as the full census. The review is characterizing the relationship between total nitrogen or total phosphorus concentrations and chlorophyll a in streams and rivers. Part of our approach uses Monte Carlo analyses to simulate large numbers of analyses on different randomly chosen samples from the full dataset. We assessed the proportion of effect size estimates that capture the true mean effect size to within several different percentages, and how the statistical power to detect true differences varies with sample size. We also estimated the approximate time savings by removing citations that were time-intensive to extract. Precise estimates of the true effect size require relatively large samples of the full body of literature, but statistical power to detect a true non-zero effect size is high even for quite small sample sizes. The suitability of sub-sampling therefore depends upon the nature of the question being asked by the review. Most scientific endeavours make inferences using samples because of an inability to undertake censuses of entire populations of data. Evidence synthesis methods that can reliably employ samples of the literature would be a major advancement. Together with other concurrent initiatives designed to improve efficiency, this has the potential to drive uptake of evidence synthesis methods in environmental science and management and thus drive the effectiveness revolution that we know is possible.