Dr. Andy Stein

Date: April 5, 2022

Speaker: Dr. Andy Stein

Affiliation: Novartis

Title: Assessing the uncertainty and predictive accuracy of mechanistic models: with applications in CML

Abstract: When studying a biological system, it can be useful to represent the system with a set of equations. Through mathematical analysis or computer simulation, one can explore the properties of these equations and gain insight into the biology. During this process, it’s usually best to start by avoiding the topic of model uncertainty, because starting simple is often the best way to approach a problem. However, when using mechanistic models to inform decisions, it is critical to account for uncertainty; often, we do not understand biology well enough to write down equations that make accurate predictions. To support model-informed decision-making in the face of uncertainty, we outline two approaches for validating mechanistic models. The first approach is to benchmark systems models against simpler, heuristic approaches to see if the more mechanistic models have improved predictive accuracy [1]. However, in many situations, there is not enough data available for benchmarking (e.g. predicting behavior of a new drug with a novel target). In these scenarios, we make use of the FDA’s Context of Use table [2] and our Uncertainty Pedigree Table [3], which was inspired by uncertainty assessments in other fields such as climate change, public health, and economics. We will then show retrospectively how the Uncertainty Pedigree Table could have been used to better frame the debate in the CML modeling literature on the effect of Gleevec on leukemic stem cells [4-5].

1. Stein AM, Looby M. Benchmarking QSP models against simple models: A path to improved comprehension and predictive performance. CPT:PSP, 7.8, 487, 2018.
2. Kuemmel, Colleen, et al. “Consideration of a credibility assessment framework in model‐informed drug development: potential application to physiologically‐based pharmacokinetic modeling and simulation.” CPT: Pharmacometrics & Systems Pharmacology 9.1 (2020): 21-28.
3. Stein, AM, et al. (2021). Cheat sheet for model uncertainty assessment. Zenodo. http://doi.org/10.5281/zenodo.4409236
4. Michor, F., et al. (2005). Dynamics of chronic myeloid leukaemia. Nature, 435(7046), 1267-1270.
5. Roeder, Ingo, et al. Dynamic modeling of imatinib-treated chronic myeloid leukemia: functional insights and clinical implications. Nature medicine 12.10 (2006): 1181-1184.