Beyond the RSE: Improving accuracy of in-host population modeling
Adquate Mhlanga1, Ori Wasserman1, Louis Shekhtman1,2, Ashish Goyal1, Elisabetta Degasperi3, Maria Paola Anolli4, Sara Colonia Uceda Renteria5, Dana Sambarino3, Marta Borghi3, Riccardo Perbellini3, Floriana Facchetti3, Annapaola Callegaro5, Scott Cotler1, Pietro Lampertico3,4, Harel Dahari1
1Program for Experimental & Theoretical Modeling, Division of Hepatology, Department of Medicine, Stritch School of Medicine, Loyola University Chicago, 2Department of Information Science, Bar-Ilan University, 3Division of Gastroenterology and Hepatology, Foundation IRCCS Ca’ Granda Ospedale Maggiore Policlinico, 4CRC “A. M. and A. Migliavacca” Center for Liver Disease, Department of Pathophysiology and Transplantation, University of Milan, 5Foundation IRCCS Ca’ Granda Ospedale Maggiore Policlinico, Virology Unit
Introduction: Reliable parameter estimation for in-host models is based on understanding the interplay between the data quality and model complexity, a balance primarily reflected in uncertainty metrics. The relative standard error (RSE) in non-linear mixed effects models is a measure that expresses the uncertainty of parameter estimates as a percentage of their estimated values, commonly used with a threshold of <50% considered ‘precisely estimated’ [1-5]. Objectives: We aim to evaluate the limitations of relying on RSE for model accuracy. Methods: Thirty-eight patients with hepatitis D virus (HDV)-related compensated cirrhosis and clinically significant portal hypertension were treated with bulevirtide (BLV) 2 mg/day monotherapy for up to 96 weeks [6]. All patients received either tenofovir disoproxil fumarate or entecavir to treat hepatitis B. Blood samples were collected at treatment initiation, weeks 4, 8, 16, 24, 32, 40, 48 and every 12 weeks thereafter. To recapitulate the observed HDV kinetics, we employed a model proposed by El Messaoudi et al. [7], which was calibrated with measured kinetic data from two French multicenter cohort studies. The model (Eq. 1) is given as follows: dI/dt=ß(1-?)VT_0-dI dV/dt=pI-cV dA/dt=s+adI-c_a A where I denotes HDV-infected cells, V represents serum HDV viral load in blood, and A represents serum alanine aminotransferase (ALT, a surrogate marker of liver damage). The virus, V, infects HDV-free susceptible cells, T_0, at a rate constant ß, generating productively HDV-infected cells, I, which produce new virions at rate p per infected cell. I are lost at rate constant d, and V are assumed to be cleared from blood at a rate constant c. The entry inhibitor BLV’s blocks infection with efficacy ?, with 0 =? =1. ALT release into blood due to HDV-infected cell death and other (hepatic and extrahepatic) cells are denoted by rate constants a, and s, respectively. Parameter c_a denotes ALT clearance from blood. Based on pre-treatment steady state condition parameters T0=cd/ßp , I0 =(c V_0)/p, and A0=(s+adI_0)/c_a . Model parameters were estimated using the maximum-likelihood method implemented in Monolix version 2023R1. We assessed the precision of parameter estimates (pre-treatment HDV viral load, V_0; infected cells loss rate, d; blocking efficacy of bulevirtide, ?), using RSE values in Monolix. To further evaluate the goodness of fit for the model (Eq. 1), we utilized three methods: Root mean square error (RMS), Coefficient of determination (R2), and the Durbin-Watson (DW) test. All computations were performed using Python 3.11, with RMS and R2 computed using NumPy (version 1.23.5), and the DW test using statsmodels (version 0.14.0). Results: Despite achieving RSE <50% in Monolix for all parameter estimates, we observed significant discrepancies between the model (Eq. 1), outputs and actual patient data, indicating limitations in the model’s ability to accurately explain HDV kinetics (see also [8]). We found (i) a strong inverse correlation between the BLV efficacy and the loss rate of HDV-infected cells, indicating identifiability issues, (ii) the model failed to accurately capture HDV kinetics in 61% patients due to poor fits, and (iii) the model failed to accurately recover certain key parameters such as pretreatment serum HDV RNA levels, resulting in unreliability of parameter estimates. Further examination of the covariance matrix confirmed a high negative correlation between BLV efficacy and the infected cell loss rate, reinforcing concerns about identifiability issues. Moreover, the goodness-of-fit metrics (RMS, R2, DW test), indicated poorer fits for patients with non-monophasic HDV decline, showing systematic bias in model performance. Confidence intervals analysis also revealed some uncertainty in key parameters, particularly the estimated BLV efficacy. Conclusion: Reliance on RSE may overlook critical issues such as parameter identifiability, inadequacy of model structures, goodness of fit, and overall model accuracy.
1. Baaz, M., T. Cardilin, and M. Jirstrand, Model-based prediction of progression-free survival for combination therapies in oncology. CPT Pharmacometrics Syst Pharmacol, 2023. 12(9): p. 1227-1237. 2. Faggionato, E., M. Schiavon, and C. Dalla Man, Modeling Between-Subject Variability in Subcutaneous Absorption of a Fast-Acting Insulin Analogue by a Nonlinear Mixed Effects Approach. Metabolites, 2021. 11(4). 3. El Messaoudi, S., et al., A Semi-mechanistic Model to Characterize the Long-Term Dynamics of Hepatitis B Virus Markers During Treatment With Lamivudine and Pegylated Interferon. Clin Pharmacol Ther, 2023. 113(2): p. 390-400. 4. Del Amo, E.M., et al., Towards a population pharmacokinetic/pharmacodynamic model of anti-VEGF therapy in patients with age-related macular degeneration. Eur J Pharm Biopharm, 2023. 188: p. 78-88. 5. Quality, A.f.H.R.a. Precision Standards Guidelines for Reporting MEPS-HC Descriptive Statistics. n.d. [cited December 2, 2024; Available from: https://meps.ahrq.gov/survey_comp/precision_guidelines.shtml. 6. Wasserman, O., et al., Bulevirtide monotherapy in patients with compensated cirrhosis and CSPH: a 96-week interim kinetic analysis of real-life setting. 2024. 80 (Suppl. 1): p. S309-S310. 7. El Messaoudi, S., et al., Effect of Peg-IFN on the viral kinetics of patients with HDV infection treated with bulevirtide. JHEP Rep, 2024. 6(8): p. 101070. 8. Mhlanga, A., et al., Modeling challenges of hepatitis D virus kinetics during bulevirtide-based therapy. JHEP Rep, 2025. 7(2): p. 101211.
