This study conducted by researchers at the Icahn School of Medicine and the University of Michigan assessed the impact of implementing predictive models on the subsequent performance of those and other models. The study simulated critical care scenarios at two major health care institutions, the Mount Sinai Health System in New York and Beth Israel Deaconess Medical Center in Boston, analyzing 130,000 critical care admissions. The findings showed that using the models to adjust how care is delivered can alter the baseline assumptions that the models were “trained” on, often for worse. The study also found that retraining models to address performance degradation over time can paradoxically lead to further degradation by disrupting the learned relationships between presentation and outcome, and that following a model’s predictions can save patients from adverse outcomes such as sepsis.
