Chris Rackauckas, Yingbo Ma, Anand Jain, Ranjan Anantharaman, Vijay Ivaturi, Viral Shah
Julia Computing and Pumas-AI
Objectives: Quantitative systems pharmacology (QsP) may need to change in order to accommodate machine learning (ML), but ML may need to change to work for QsP. Here we investigate the use of neural network surrogates of stiff QsP models. This technique reduces and accelerates QsP models by training ML approximations on simulations. We describe how common neural network methodologies, such as residual neural networks, recurrent neural networks, and physics/biologically-informed neural networks, are fundamentally related to explicit solvers of ordinary differential equations (ODEs). Similar to how explicit ODE solvers are unstable on stiff QsP models, we demonstrate how these ML architectures see similar training instabilities. To address this issue, we showcase methods from scientific machine learning (SciML) which combine techniques from mechanistic modeling with traditional deep learning. We describe the continuous-time echo state network (CTESN) as the implicit analogue of ML architectures and showcase its ability to accurately train and predict on these stiff models where other methods fail. We demonstrate the CTESN’s ability to surrogatize a production QsP model from Pfizer Inc., a >1,000 ODE chemical reaction system from the SBML Biomodels repository, and a reaction-diffusion partial differential equation. We showcase the ability to accelerate QsP simulations by up to 56x against the optimized DifferentialEquations.jl solvers while achieving <5% relative error in all of the examples. This shows how incorporating the numerical properties of QsP methods into ML can improve the intersection, and thus presents a potential method for accelerating repeated calculations such as global sensitivity analysis and virtual populations.
Methods: We show that recurrent neural networks (RNNs), long-short term memory networks (LSTMs), and physics-informed neural networks (PINNs) are numerically unstable and fail to capture the dynamical properties of highly stiff dynamical systems, while the continuous-time echo state network’s (CTESN) implicit training allows for accurately surrogatizing highly stiff dynamical systems [1]. We demonstrate that this property theoretically by deriving standard machine learning methods from explicit discretizations of neural network-defined ODEs, and showcase the CTESN as the implicit analogue. We further improve the method’s prediction accuracy by deriving a nonlinear projector which further stabilizes the projective eigensystem and reduces the data requirements. The result is a technique which requires ~100 ODE simulations to learn to emulate stiff ODE models with a maximum error of <5% over the parameter range and time span.
Results: We built a system for reading SBML, CellML, and BioNetGen formats into Pumas through the underlying ModelingToolkit acausal modeling system [2]. Using symbolic simplification tooling along with the Julia differential equation solvers [3], we demonstrate that the baseline solving using the JuliaSim and Pumas tooling accelerates simulation of a 1122 ODE WUSCHEL model [4] by approximately 140x over SimBiology prior to surrogatization. Using the CTESNs, we achieve a 724x speedup over the simulation speed of SimBiology. We then used these links to surrogatize via CTESNs >100 models from the CellML physiome library to demonstrate consistent stability of across a large array of pharmacometric models. On the larger side of models (models with >1000 ODEs) and on reaction-diffusion partial differential equations, we demonstrate that the accelerations of the CTESN transfer to the pharmacometrics domain, reaching speedups of 56x over the non-ML differential equation solving in JuliaSim and Pumas.
Conclusions: We demonstrated the viability of new machine learning architectures for accelerating pharmacometric simulation and have developed the first platform which seamlessly integrates surrogatization into a pharmacometrics software.
References:
[1] Anantharaman, R., Ma, Y., Gowda, S., Laughman, C., Shah, V., Edelman, A., & Rackauckas, C. (2020). Accelerating Simulation of Stiff Nonlinear Systems using Continuous-Time Echo State Networks. arXiv preprint arXiv:2010.04004.
[2] Ma, Y., Gowda, S., Anantharaman, R., Laughman, C., Shah, V., & Rackauckas, C. (2021). ModelingToolkit: A Composable Graph Transformation System For Equation-Based Modeling. arXiv preprint arXiv:2103.05244.
[3] Rackauckas, C., & Nie, Q. (2017). Differentialequations. jl–a performant and feature-rich ecosystem for solving differential equations in julia. Journal of Open Research Software, 5(1).
[4] Jönsson, Henrik, et al. “Modeling the organization of the WUSCHEL expression domain in the shoot apical meristem.” Bioinformatics 21.suppl_1 (2005): i232-i240.
Reference: PAGE 29 (2021) Abstr 9756 [www.page-meeting.org/?abstract=9756]
Poster: Methodology - Other topics