Benjamin Guiastrennec (1), Anne Kümmel (1), Henning Schmidt (1)
(1) IntiQuan GmbH
Introduction/Objectives: Pharmacometric analyses are commonly used to support critical decision. Therefore, a high degree of traceability and reproducibility must be ensured from the source data to the report generation. Only then could collaborators, sponsors and regulatory agencies properly evaluate the quality and validity of the analysis [1]. However, with NONMEM-based pharmacometric analyses this can be challenging due to the array of tools used throughout data processing (e.g., SAS, R), data analysis (e.g., R, NONMEM, PsN), and reporting which themselves have additional dependencies (e.g., compilers, libraries) [2]. Thanks to the development of dedicated tools to facilitated workflow-based analysis the traceability aspect has mostly been addressed. However, the reproducibility aspect is often overlooked. Due to differences in operating systems, program versions/configurations and R libraries between workstations, or within a given workstation taken at different timepoint, breaking changes can occur and yield to a lack of confidence in the capacity to reproduce given results in the future. Herein we illustrate common reproducibility issues and suggest solutions to address them.
Methods: A pharmacokinetic (PK) [3] and a pharmacodynamic model (PD) [4] were incorporated into a fully traceable analysis workflow developed using the IQRtools R package [5]. For each model the reproducibility of the NONMEM results using the first-order conditional estimation (FOCE) or the stochastic approximation expectation maximization (SAEM) was evaluated with and without parallelization under 3 different setups: a Linux server running a docker image (Ubuntu 18.04 LTS, NONMEM 7.4.3, R 3.6.3 with CRAN snapshot), a MacBook pro (OSX 10.15.7) with a local NONMEM installation (NONMEM 7.4.3, R 3.6.3 with CRAN snapshot) or the MacBook pro running the docker image [6]. Furthermore, to assess the repeatability of the NONMEM estimation methods, each scenario was run twice in a row.
Results: IQdesktop a docker image containing a validated pharmacometric analysis environment was developed to facilitate this analysis [7]. For both the PK and PD models, the results of the NONMEM FOCE estimation were repeatable and not impacted by the parallelization. Minor differences were seen on some model parameter between the docker and the OSX environments. All results were identical between the docker environments. The results of the SAEM were only repeatable without parallelization. An update to the default NONMEM parallelization configuration file enabled repeatable results for SAEM on a given number of nodes. Results of the SAEM were different between the docker and OSX runs but were near identical between docker images.
Conclusions: Computational reproducibility in pharmacometric analyses is often overlooked. Here we showed that the NONMEM FOCE estimation was robust to parallelization unlike SAEM for which the default NONMEM parallelization file systematically led to non-repeatable results. Integrating the standard pharmacometric toolbox within a docker image along with improvements to the default NONMEM parallelization file enabled near identical results for both evaluated models with FOCE and SAEM estimations between a Linux and MacBook computer. In the future one could imagine the computing environment being provided alongside the analysis workflow for full traceability and reproducibility.
References:
[1] Y.C. Ou et al. Integration of Biostatistics and Pharmacometrics Computing Platforms for Efficient and Reproducible PK/PD Analysis: A Case Study. The Journal of Clinical Pharmacology. 53(11), 1112–1120, 2013.
[2] J.J. Wilkins, N. Jonsson. Reproducible Pharmacometrics, Using Reproducible Research methodologies to improve pharmacometric analyses. PAGE meeting, 2013 [www.page-meeting.org/?abstract=2774].
[3] M.O. Karlsson et al. J Pharmacokinetics and Biopharmaceutics. 26(2):207–246 (1998)
[4] L.E. Friberg et al. J Clin Oncol. 2002 Dec 15;20(24):4713-21.
[5] IQRtools. Modeling & Simulation in R. iqrtools.intiquan.com.
[6] C. Boettiger. An introduction to Docker for reproducible research, with examples from the R environment, ACM SIGOPS Operating Systems Review, Special Issue on Repeatability and Sharing of Experimental Artifacts. 49(1), 71-79, 2015.
[7] IQdesktop. A Qualified Virtual Modeling & Simulation Environment Supporting Efficient Model Informed Drug Development. iqdesktop.intiquan.com.
Reference: PAGE 29 (2021) Abstr 9782 [www.page-meeting.org/?abstract=9782]
Poster: Methodology - Other topics