Franziska Schaedeli Stark (1), Sylvie Retout (1), Valérie Cosson (1), Sreenath Krishnan (1), Orwa Albitar (1), João A. Abrantes (1), Emilie Schindler (1)
Roche Pharma Research and Early Development, Pharmaceutical Sciences, Roche Innovation Center Basel, Switzerland
Introduction: The Automatic Model Development (AMD) tool in Pharmpy is an open source tool for automatic building of population pharmacokinetic (PopPK) models, using a systematic stepwise process and covering building of all components of common popPK models [1,2,3]. The performance of the AMD tool was recently assessed using simulated rich PopPK datasets, highlighting the potential of such a tool in early drug development, producing reliable PK models while saving time for pharmacometricians to be invested in more complex modeling activities [4].
Objectives: To evaluate the AMD tool in a real drug development context by (i) assessing its performance in identifying models with appropriate descriptive and predictive properties; (ii) identifying opportunities for its enhancements .
Methods: Nine PopPK datasets from Roche drug development projects were used to evaluate the AMD tool. These datasets included PK data collected after repeated IV (N=2) or non-IV (N=7) administrations from clinical phase 1-3 studies. They were of various sizes from 344 to 3533 PK observations, collected in 20 to 288 subjects, with a mixture of sparse and rich PK sampling. For each dataset, the final original popPK model developed without AMD was available. The 9 original popPK models covered a variety of model features, including 2 or 3 mammillary compartment disposition, linear and/or nonlinear elimination, various degrees of absorption complexities, covariate effects, and various complexities in the inter-individual (IIV) and residual unexplained (RUV) variability model structures.
The AMD tool (modelsearch, IIV, and RUV search subtools) was run with each dataset. The structural search space included 1-3 compartments with, linear, Michaelis-Menten or combined elimination. For non-IV drugs, first-order absorption without or with a delay, modeled either with a lag time, with 1, 3, or 10 transit compartments, or with sequential zero-first order absorption. For 6 datasets with available patient baseline characteristics, covariate search was performed subsequently using the COVsearch subtool. The pharmpy R-wrapper (pharmr) 0.88.0 [5] and NONMEM 7.5.1 were run in the model building environment Improve [6].
The final AMD models were assessed and compared to the original ones in terms of overall complexity (structural and statistical models, parameter estimate precisions), data description (BIC, goodness-of-fit [GOF] plots), predictive performance (VPC), detected covariates and impact on clinical decisions (Forest plots).
Results: For 8 out of 9 datasets a final AMD model was obtained, following 100 to 920 model building steps. For one dataset the process stopped during iivsearch possibly due to computational issues. Overall, AMD models provided lower BIC values compared to the original models, although the former were often associated with a more complex compartmental structure (7/8 models) and more complex IIV structure (4/8 models) and RUV models (7/8 models). Forest plots indicated that selected covariate effects were meaningful and aligned with the original covariate models, although not systematically on the same parameters, due to differences in the detected IIV structure. AMD models showed similar GOF and VPC to the original models, qualifying them for simulation tasks. Through this evaluation, potential improvements of the tool were identified, namely reducing the risk of having over-parameterized models via a modified BIC calculation with additional penalty for multiple testing [2], stricter p-values during RUV search, and additional strictness criteria (e.g. number of significant digits) that were implemented in newer versions of pharmpy/pharmr. More recent runs using pharmr 0.106.0/0.107.0 provided final AMD models for all 9 datasets, with overall less complex structural, IIV, and/or RUV model parts, without affecting the descriptive and predictive model performances.
Conclusions: The AMD tool was able to generate reasonable descriptive and predictive models for 8/9 investigated real datasets, confirming results generated previously [1]. However, the AMD performance evaluation highlighted a trend to provide more complex models than the ones developed without AMD. The assessment allowed identifying improvements of the AMD tool and sub-tools that have been implemented. This work further showcases the usefulness of the AMD tool towards an automated modeling environment in drug development.
Acknowledgments: The authors would like to thank the Uppsala Pharmacometrics Research Group for their collaboration, Scinteco for their technical assistance, and Nicolas Frey for his valuable input.
References:
[1] Chen X, et al. Development of a tool for fully automatic model development (AMD). PAGE 30 (2022) Abstr 10091 [www.page-meeting.org/?abstract=10091]
[2] Pharmpy/PharmR AMD tool documentation [Internet]. https://pharmpy.github.io. [accessed March 2024]
[3] Abrantes J, et. al. ADaMO: End-to-end automation of Pharmacometric modelling in drug development, from dataset building to output generation. PAGE 30 (2022) Abstr 10051 [www.page-meeting.org/?abstract=10051]
[4] Duvnjak et al. Performance evaluation of the full Automatic Model Development (AMD) tool when the true model is known. PAGE 31 (2023) Abstr 10529 [www.page-meeting.org/?abstract=10529]
[5] Nordgren R, et al. Pharmpy: a versatile open-source library for pharmacometrics. PAGE 31 (2023) Abstr 10508 [https://www.page-meeting.org/default.asp?abstract=10508]
Reference: PAGE 32 (2024) Abstr 11013 [www.page-meeting.org/?abstract=11013]
Poster: Methodology - New Modelling Approaches