A modified Bayesian information criterion (mBIC) with multiple testing correction for population pharmacokinetic model building
Xiaomei Chen1, Ya-Han Hsu1, Rikard Nordgren1, Stella Belin1, Emilie Schindler2, Andrew C. Hooker1, Mats O. Karlsson1
1Department of Pharmacy, Uppsala University, 2Roche Pharma Research and Early Development, Pharmaceutical Sciences, Roche Innovation Center Basel
Multiple testing (or multiplicity) is a well-recognized problem in statistics, which occurs when multiple hypothesis tests are performed simultaneously on a single dataset. This problem is especially prominent when evaluating a large pool of predictors or models. The process of pharmacometric model building involves estimations of various models and multiple comparisons among the estimated models. Without correction for multiple testing, a more complicated model tends to be selected due to the increased type I error. However, few studies have been done to address this problem. In the field of quantitative trait loci mapping, a modified Bayesian information criterion (mBIC) was previously proposed1,2 for linear multiple regression considering multiple testing correction. A fully automatic model development (AMD) tool for population pharmacokinetic models, implemented in the open source software package Pharmpy/pharmr, has been developed by our group3, which consists of multiple modules for model building of different components of nonlinear mixed effect models (NLME). The modules for selecting structural models and random-effect models use the default mixed-effect BIC4 and random-effect BIC5, respectively.
Objectives:
• Expand and further modify the published mBIC for NLME and model development of population pharmacokinetic models.
• Compare BIC and mBIC in the selected final models by using the AMD tools for model structure development and inter-individual variability (IIV) model development.
Methods:
In the current work, two mBIC equations were proposed for selecting the pharmacokinetic structural model and random effect model, respectively. An additional penalty term for multiple testing ( is added to the mixed-effect BIC for model structure selection. The size of the penalty depends on the size of the search space (p), the a priori value related to the expected model complexity (E), and the model elements in the currently evaluated model (. For random effect models, two terms of similar penalty are added to the random-effect BIC equation accounting for diagonal and off-diagonal terms of the OMEGA matrix. The proposed mBIC equations were applied to structural model selection and IIV model selection when using the AMD tool on 10 real datasets (5 i.v. drugs and 5 oral drugs; subject numbers of 32-210; and observation numbers of 112-1177). The final selected models were compared to those using the default BIC criteria.
Results:
The additional penalty for multiple testing increases with decreasing E. Numerical results presented below are for E values of 0.1, 0.5, 1, and 2, respectively. During a structural model search with the default search space of the AMD tool (p=6), the multiple-testing penalty was 8.2, 5.0, 3.6, and 2.2 for one additional level of model complexity (e.g., adding a peripheral compartment). For a dataset with 74 subjects and 476 observations, the application of mBIC corresponded to a significance level of 0.0002, 0.0011, 0.0023, and 0.0045 using the likelihood ratio test (LRT), for which 0.05 is commonly used as the significance level. During a search for IIV models with four pharmacokinetic parameters, the multiple-testing penalty was 6.8, 3.6, 2.2, and 0.8 for selecting an additional diagonal OMEGA. Those corresponded to LRT-derived significance levels of 0.0009, 0.005, 0.011, and 0.024, respectively. Through the AMD procedure on the 10 datasets, the mBIC provided more parsimonious models compared to the BIC in one case for model structure building (E=0.1) and in 5 datasets for IIV model building (E=0.5 for diagonal OMEGA terms and E=1 for off-diagonal terms).
Conclusions:
The proposed mBIC is a promising selection criterion that accounts for multiple testing in the case of extensive pharmacokinetic model building and can avoid an overly complicated final model, especially for the selection of random-effect models.
1. Bogdan M, Ghosh JK, Doerge RW. Modifying the Schwarz Bayesian information criterion to locate multiple interacting quantitative trait loci. Genetics. 2004;167(2):989-999. doi:10.1534/GENETICS.103.021683
2. Cui X, Dickhaus T, Ding Y, Hsu JC. Handbook of Multiple Comparisons. Handb Mult Comp. Published online October 27, 2021.
3. Chen X, Nordgren R, Belin S, et al. A fully automatic tool for development of population pharmacokinetic models. CPT Pharmacomet Syst Pharmacol. 2024;13(10):1784-1797. doi:10.1002/psp4.13222
4. Delattre M, Lavielle M, Poursat MA. A note on BIC in mixed-effects models. Electron J Stat. 2014;8(1):456-475. doi:10.1214/14-EJS890
5. Delattre M, Poursat MA. An iterative algorithm for joint covariate and random effect selection in mixed effects models. Int J Biostat. 2016;16(2):75-83. doi:10.48550/arxiv.1612.02405