Thanakorn Vongjarudech1, Mohona Khandokar1, Ya-Han Hsu1, Mats O Karlsson1
1Department of Pharmacy, Uppsala University
Introduction/Objectives: Cross-validation (XV) is a widely used technique for evaluating a model‘s predictive performance on unseen data [1]. Although XV has been applied in covariate model building in pharmacometrics [2,3], its potential for guiding structural model selection remains underexplored. Automatic Model Development (AMD) [4] tools provide an efficient and fully automated approach to building population pharmacokinetic (PK) models, utilizing metrics including the likelihood ratio test (LRT), Akaike information criterion (AIC), Bayesian Information Criterion (BIC) [5], and modified BIC (mBIC) [6] for model evaluation. This study aimed to investigate the utility of XV for structural model selection in pharmacometrics, focusing on comparing XV-based and other selection strategies. Method: A simulated PK dataset was generated based on a published population PK model for moxonidine [7], comprising 74 individuals and 1166 observations. AMD tool was used to fit 16 different structural models into both the full dataset and split dataset for XV. The structural models included one- or two-compartment disposition models, with or without an absorption delay (modeled by one or three transit compartments), and first-order absorption with a stepwise order. To evaluate model performance using XV, five-fold XV was implemented by randomly partitioning the data into five equal-size subsets. In each fold, four subsets (80% of the data) served as the training set for fitting candidate models, while the remaining subset (20% of the data) served as the test set. After fitting each model to the training subsets, models were evaluated on the test set without re-estimation (MAXEVAL = 0 in NONMEM). The prediction-based objective function value (pOFV) obtained from each fold was summed up (SpOFV) for model ranking and selection. The XV process was repeated five times to obtain the mean and the standard deviation (SD) of the SpOFV. The ranking based on mean SpOFV was further compared with AIC, BIC, mBIC, and LRT (if nested) using Spearman’s rank correlation coefficients. The “One Standard Error Rule” (1SE rule) [8] was also implemented to select the parsimonious model. NONMEM 7.5 with first-order conditional estimation with interaction (FOCE-I), aided by PsN 5.3.0 [9] and R 4.2.2 with the Pharmr 1.6.0 package (an R wrapper for Pharmpy), was used for this analysis. Results: Using the full dataset, the one-compartment model with one depot and three transit compartments (T(3,D);P(0)) was the best-performing structure (BIC = -3458.34, AIC = -3484.01, mBIC = -3456.15), followed closely by a two-compartment model with one depot and three transit compartments (T(3,D);P(1)) (BIC = -3444.48, AIC = -3480.01, mBIC = -3440.09). In the XV analysis, the two-compartment model with one depot and three transit compartments (T(3,D);P(1)) was the best (mean SpOFV = -3429.82, SD = ±252.64), followed by one depot and three transit compartments (T(3,D);P(0)) (mean SpOFV = -3427.2). However, based on the 1SE rule the one depot and three transit compartments (T(3,D);P(0)) model is considered the best-performing model. Model rankings were compared across AIC, BIC, and mBIC, with Spearman’s rank correlation coefficients showing the highest correlation between mBIC and BIC (0.999), followed by AIC and BIC (0.944), while correlations between AIC, BIC, mBIC, and SpOFV ranged from 0.91 to 0.93. Ranking with LRT provided the same outcome as other criteria, where a more complicated model did not significantly improve the performance. Conclusion: XV-based model selection is a useful and informative approach for structural model selection in pharmacometrics. In this simulation study, the XV-based approach successfully identified the best-performing model, aligning closely with AIC, BIC, mBIC, and LRT. XV provides a more direct measure of predictive performance by assessing models on independent test data.
[1] Bates S, Hastie T, Tibshirani R. Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association 2024;119:1434–45. https://doi.org/10.1080/01621459.2023.2197686. [2] Katsube T, Khandelwal A, Harling K, Hooker AC, Karlsson MO. Evaluation of Stepwise Covariate Model Building Combined with Cross-Validation n.d. https://www.page-meeting.org/?abstract=2111 (accessed March 2, 2025). [3] Hutmacher MM, Kowalski KG. Covariate selection in pharmacometric analyses: a review of methods. British Journal of Clinical Pharmacology 2015;79:132–47. https://doi.org/10.1111/bcp.12451. [4] Chen X, Nordgren R, Belin S, Hamdan A, Wang S, Yang T, et al. A fully automatic tool for development of population pharmacokinetic models. CPT: Pharmacometrics & Systems Pharmacology 2024;13:1784–97. https://doi.org/10.1002/psp4.13222. [5] Delattre M, Lavielle M, Poursat MA. A note on BIC in mixed-effects models. Electron J Stat. 2014;8(1):456-475. (https://doi.org/110.1214/14-EJS890) [6] Bogdan M, Ghosh JK, Doerge RW. Modifying the Schwarz Bayesian information criterion to locate multiple interacting quantitative trait loci. Genetics. 2004;167(2):989-999. (https://doi.org/10.1534/genetics.103.021683) [7] Karlsson MO, Jonsson EN, Wiltse CG, Wade JR. Assumption Testing in Population Pharmacokinetic Models: Illustrated with an Analysis of Moxonidine Data from Congestive Heart Failure Patients. J Pharmacokinet Pharmacodyn 1998;26:207–46. https://doi.org/10.1023/A:1020561807903. [8] Chen Y, Yang Y. The One Standard Error Rule for Model Selection: Does It Work? Stats. 2021; 4(4):868-892. https://doi.org/10.3390/stats4040051 [9] Lindbom L, Pihlgren P, Jonsson EN. PsN-Toolkit–a collection of computer intensive statistical methods for non-linear mixed effect modeling using NONMEM. Comput Methods Programs Biomed 2005;79:241–57. https://doi.org/10.1016/j.cmpb.2005.04.005.
Reference: PAGE 33 (2025) Abstr 11736 [www.page-meeting.org/?abstract=11736]
Poster: Methodology - Model Evaluation