Medical Transformer For Multimodal Survival Prediction In Intensive Care: Integration Of Imaging And Non-imaging Data

In this work, we developed and evaluated the medical transformer architecture MeTra to integrate imaging and non-imaging data for survival predictions in patients in critical care. MeTra can predict the survival of critically ill patients when trained on clinical data or imaging data exclusively, and can combine both data sources for improved model predictions. Additionally, MeTra can deal with missing data and has a smooth transition between high diagnostic accuracy when all data is available to reduced diagnostic accuracy when data are missing. This study is the first to investigate the performance of a fully transformer-based architecture in the survival prediction of patients in intensive care and proves its viability when handling imaging and non-imaging data.