Stefano Giampiccolo (1,2), Giovanni Iacca (2), Luca Marchetti (1,3)
(1) Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI), Italy; (2) University of Trento, Department of Information Engineering and Computer Science – DISI, Italy (3) University of Trento, Department of Cellular, Computational and Integrative Biology – CIBIO, Italy
Introduction: Neural Ordinary Differential Equations (NODEs) [1,2] constitute a promising data-driven method for modeling dynamical systems in computational biology. Recent works have demonstrated that NODEs can be integrated with mechanistic models in a hybrid setting [L1] [3,4]. These hybrid models have been used in pharmacometrics to estimate parameters related to drug distribution and elimination without requiring complete mechanistic knowledge of the underlying biology, replacing portions of the model for which a mechanistic description is unavailable (e.g., absorption processes [3]) with neural networks. Despite its potential, the use of hybrid NODEs to estimate mechanistic parameters is still largely unexplored. Moreover, best practices regarding how to globally explore the search space for parameter estimation are currently lacking. Finally, to the best of our knowledge, no method has been proposed for analyzing parameter identifiability, a crucial requirement in pharmacometrics applications [6].
Objectives: We aim to introduce an end-to-end computational approach to assist the development of pharmacometrics models when only partial mechanistic knowledge of the system is available, leveraging NODE hybrid models and tackling the following challenges:
- selection of an appropriate hybrid NODE model;
- estimation of the model parameters by ensuring a global exploration of the parameter space;
- assessment of the parameters’ identifiability.
Methods: Building on the Julia SciML software framework [7], we have developed an end-to-end pipeline that has been tested by leveraging in silico-generated datasets obtained by simulating the models reported in the Results section. To create more realistic scenarios beyond the case without noise, we considered the case where Gaussian noise is added to each time series (with SD equal to 5% of its min-max variation).
The proposed pipeline is composed of the following steps:
Step 1. The initial step involves tuning the hyperparameters of the hybrid NODE model, which extends the incomplete mechanistic model, with the Tree-structured Parzen Estimator [8]. The hyperparameters encompass the neural network architecture, along with other training-specific configurations (such as the learning rate, the L2 regularization constant, etc.). This step also performs a global exploration of the parameter search space by treating the initial points as hyperparameters for optimizing the mechanistic parameters.
Step 2. The hybrid model is then fully trained with the Adam algorithm. This training process optimizes both the mechanistic and the neural network parameters simultaneously.
Step 3. Local identifiability of the mechanistic parameters is assessed a posteriori using an ad-hoc variant of the eigenvalue method [9].
Step 4. The asymptotic confidence intervals for mechanistic parameter estimates are finally computed using the Fisher Information Matrix.
Results: The pipeline has been tested in various in-silico scenarios, derived from three mechanistic models considered benchmark models in Computational Biology: the Lotka-Volterra model [10], a model describing cell apoptosis [11], and a model describing the oscillations in yeast glycolysis [12]. In each case, we assumed a lack of knowledge of specific portions of the model (the two inter-species interaction terms in Lotka-Volterra and one complete missing equation for the remaining two models, selected to maximize the challenge of parameter estimation, according to the results reported in [13]).
We successfully assessed the non-identifiability of parameters in the Lotka-Volterra test case, where the neural network can compensate for changes in the estimates of mechanistic parameters. In the cell-apoptosis test case, we assessed the non-identifiability of three parameters while estimating four others (mean precision among parameters less than 1% in the case without noise, less than 5% with noise). In the glycolytic test case, we estimated 13 identifiable model parameters (mean precision less than 1% in the case without noise, less than 25% with noise).
Conclusions: The proposed pipeline can be effectively applied to define hybrid NODE models and estimate relevant parameters when the system’s complete mechanistic knowledge is unavailable. Our results indicate that the proposed end-to-end approach can be used in realistic conditions of pharmacometrics studies, with small-sized datasets containing noisy observations of a limited number of system variables.
References:
[1] Stankeviciute, Kamile, et al. “Bridging the worlds of pharmacometrics and machine learning.” Clinical Pharmacokinetics 62.11 (2023): 1551-1565.
[2] Chen, Ricky TQ, et al. “Neural ordinary differential equations.” Advances in neural information processing systems 31 (2018).
[3] Valderrama, Diego, et al. “Integrating machine learning with pharmacokinetic models: Benefits of scientific machine learning in adding neural networks components to existing PK models.” CPT: Pharmacometrics & Systems Pharmacology 13.1 (2024): 41-53.
[4] Bräm, Dominic Stefan, et al. “Low-dimensional neural ODEs and their application in pharmacokinetics.” Journal of Pharmacokinetics and Pharmacodynamics (2023): 1-18.
[5] Yin, Yuan, et al. “Augmenting physical models with deep networks for complex dynamics forecasting.” Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021): 124012
[6] Shivva, Vittal, et al. “An approach for identifiability of population pharmacokinetic–pharmacodynamic models.” CPT: pharmacometrics & systems pharmacology 2.6 (2013): 1-9
[7] Rackauckas, Christopher, et al. “Universal differential equations for scientific machine learning.” arXiv preprint arXiv:2001.04385 (2020).
[8] Bergstra, James, et al. “Algorithms for hyper-parameter optimization.” Advances in neural information processing systems 24 (2011).
[9] Krivorotko, Olga Igorevna, Dar’ya Vladimirovna Andornaya, and Sergey Igorevich Kabanikhin. “Sensitivity analysis and practical identifiability of some mathematical models in biology.” Journal of Applied and Industrial Mathematics 14 (2020): 115-130.
[10] Wangersky, Peter J. “Lotka-Volterra population models.” Annual Review of Ecology and Systematics 9.1 (1978): 189-218.
Reference: PAGE 32 (2024) Abstr 10917 [www.page-meeting.org/?abstract=10917]
Poster: Methodology - New Modelling Approaches