One key challenge is that the medical notes used to train and validate these models may differ greatly across hospitals, providers, and time. For example, doctors often use specialized templates, such as headings or tables, in their notes. The team also proposes using available auxiliary data (like timestamps, document types, and patient demographics) associated with but not included in these medical notes to create better approximations of counterfactual data. Through extensive experiments, the researchers demonstrate that using language models in a domain-informed manner improves an ML model's generalizability in challenging, safety-critical tasks like medical note analysis. This project is part of an effort led by Saria with collaborators at regulating agencies such as the FDA toward the development of an AI safety framework for health care applications.
