Ya-Han Hsu1, Xiaomei Chen1, Mats O. Karlsson1
1Department of Pharmacy, Uppsala University
Introduction/Objectives The model-building process involves a series of decisions, for example on the search scope, the model selection criteria, and the development sequence of model components. However, the performance of different strategies is rarely evaluated. The automated model development (AMD) tool¹ mimics manual model building and provides a convenient tool for performing such investigations. This work aims to apply the AMD tool to (A) explore and optimize model-building strategies, (B) study model properties (e.g., covariate identification power, parameter uncertainty) in modeling applications, and (C) optimize the settings within the AMD tool for performance improvement. Methods The computer-intensive modeling work covered three aspects: (A) model-building strategy (Task 1-4; i.e., T1-4), (B) modeling application analysis (T5-6), and (C) AMD improvement (T7-8). It was done by applying the AMD tool² (Pharmpy/pharmr version 1.3.0 to 1.6.0) using an oral moxonidine pharmacokinetic (PK) example, either a real dataset (T1-2) with 73 subjects and 1,006 observations³,4 or simulated datasets (T3-6) from a one-compartment PK model with three transit compartments, and a covariate of renal function on the clearance. The computer-intensive explorations include 8 tasks: T1-compare modeling sequences (SIR, SRI, RSI; S: structure; I: inter-individual variability (IIV); R: residual unexplained variability (RUV) models); T2– compare selection criteria: Bayesian information criterion (BIC) for mixed effect models5 and modified BIC (mBIC) which adds additional penalty for multiple testing6 ; T3-compare cross validation OFV (pOFV)7, mBIC, BIC, and Akaike information criterion (AIC); T4-subsequent validation with a large simulated dataset (9,990 subjects) to evaluate the selected final models; T5-evaluate the power of detecting covariates using 100 stochastic simulated datasets (SSE)8 with the true model and the AMD selected models; T6-evaluate parameter uncertainty using 10 bootstrapped datasets?; T7-a 20-iteration AMD loop run in which each iteration started with a dataset simulated from the final model of the previous iteration; T8-compare different compilers (GCC 8.3.0 and 4.6.3) and NONMEM versions (7.4.4, 7.5.0, and 7.5.1). Results (A) Model building strategy With the clinical dataset, all sequences selected the same structural model, yet slightly different IIV and RUV. SRI (BIC=-2563) performs the best, followed by RSI (BIC=-2527) and SIR (BIC=-2523) (T1). When ranked with mBIC for the SIR sequence, a simpler model was selected only when the expected model complexity value was =0.1, showing a limited impact of multiple testing due to the small search scope in this case (T2). In a simulation study evaluating four selection criteria (pOFV, AIC, BIC, mBIC), all Spearman’s rank correlation coefficients were above 0.91, indicating high consistency (T3). The above results were further confirmed through prediction into a large dataset, where final models with SRI in all selections (AIC, BIC, mBIC) had the lowest OFV (T4). (B) Modeling application analysis In SSE analysis with AMD (T5), the power to identify the renal function effect was 94% using the simulation model, versus 73% with the AMD selected models. In the bootstrap analysis (T6), the coefficient of variability (CV) of clearance, for instance, was 5.2% and 2.7% with the individual AMD selected models and the true model, respectively. (C) AMD improvement The loop run result (T7) illustrated a consistent structural model selection, with covariates identified occasionally across iterations. 60% of the final models had a drop in BIC compared to the true model, showing that AMD tool tends to identify parsimonious models. The results from the AMD tool are reproducible by design under the same settings. Under different compilers and versions, BIC of final models varied from -3453 to -3437 (T8). Conclusions The AMD tool offers efficiency and flexibility to explore the model building strategies, individual applications, and AMD tool improvement under different circumstances.
1. Chen X, Nordgren R, Belin S, Hamdan A, Wang S, Yang T, et al. A fully automatic tool for development of population pharmacokinetic models. CPT: Pharmacometrics & Systems Pharmacology 2024;13:1784–97. 2. Nordgren R, Belin S, Yngman G, et al. Pharmpy: a versatile open-source library for pharmacometrics. PAGE 30 (2022) Abstr 10029 [www.page-meeting.org/?abstract=10029] 3. Karlsson MO, Jonsson EN, Wiltse CG, Wade JR. Assumption testing in population pharmacokinetic models: illustrated with an analysis of moxonidine data from congestive heart failure patients. J Pharmacokinet Biopharm. 1998;26:207-246. 4. Savic RM, Jonker DM, Kerbusch T, Karlsson MO. Implementation of a transit compartment model for describing drug absorption in pharmacokinetic studies. J Pharmacokinet Pharmacodyn. 2007;34:711-726. 5. Delattre M, Lavielle M, Poursat MA. A note on BIC in mixed-effects models. Electron J Stat. 2014;8(1):456-475. 6. Bogdan M, Ghosh JK, Doerge RW. Modifying the Schwarz Bayesian information criterion to locate multiple interacting quantitative trait loci. Genetics. 2004;167(2):989-999. 7. Katsube T, Khandelwal A, Harling K, Hooker AC, Karlsson MO. Evaluation of Stepwise Covariate Model Building Combined with Cross-Validation. PAGE 20 (2011) Abstr 2111 [www.page-meeting.org/?abstract=2111] 8. Vong C, Bergstrand M, Nyberg J, Karlsson MO. Rapid sample size calculations for a defined likelihood ratio test-based power in mixed-effects models. AAPS J. 2012;14(2):176–186. 9. Mirasbekov Y, Chen X, Hooker AC, Karlsson MO. Characterizing Bootstrap Model Selection Variability Through an Automated Model Building Approach. PAGE 31 (2023) Abstr 10690 [www.page-meeting.org/?abstract=10690]
Reference: PAGE 33 (2025) Abstr 11369 [www.page-meeting.org/?abstract=11369]
Poster: Methodology - New Modelling Approaches