Comparison of Integrated Risk Information System and Navigation Guide tools for study evaluation in systematic reviews of environmental epidemiology studies
On this page:
Introduction There are several tools available for evaluating risk of bias in environmental epidemiology studies, but few case studies comparing these tools have been done. Methods The Integrated Risk Information System (IRIS) tool is a domain-based approach where each study is evaluated for risk of bias and study sensitivity. Eick et al. (2020) applied multiple tools, including IRIS, to studies from a Navigation Guide (NG) systematic review of polybrominated diphenyl ethers (PBDEs) and neurodevelopment (Lam et al., 2017). In the current analysis, trained IRIS staff applied the IRIS tool, including PBDE exposure and cognitive testing-specific considerations, to the 10 studies of IQ and compared the results to those using the NG tool. This analysis was restricted to domains comparable across the tools: Exposure and Outcome Assessment, Confounding, Population/Selection, and Selective Reporting (and NG domains considered within the broader IRIS Exposure and Outcome domains). The IRIS and NG tools have four rating levels that may not be strictly analogous but were interpreted as similar for identifying study deficiencies. Results We identified three studies with multiple “deficient” domain ratings (low confidence overall) using the IRIS tool. These studies also had multiple domains with “probably high risk of bias” in the NG systematic review. Among studies rated as high or medium confidence overall using the IRIS tool, there were differences in ratings between the tools, but generally within the top two rating levels (IRIS good/adequate; NG low/probably low). Conclusions The tools identified the same studies as concerning for risk of bias. Among higher quality studies, some inter-tool rating differences may be attributed to different expertise/interpretations of evaluators rather than the tools. Additional comparisons would be elucidating, ideally with collaboration to ensure application of the tools consistent with their intended use. Disclaimer: The views expressed in this abstract are those of the author and do not necessarily reflect the views or policies of the U.S. EPA.