Stella Belin1, Rikard Nordgren1
1Department of Pharmacy, Uppsala University
Objectives: For a computation to be reproducible it needs to generate exactly the same or similar results given the same inputs and methods. In this work, we explore some problems that can arise when trying to reproduce computational results and suggest solutions in different tiers of completeness. Using examples, we want to highlight different challenges of reproducibility rather than fully trying to reproduce any particular result. The aim is to give concrete suggestions to address the challenges of reproducibility.
Methods: We have explored the reproduction of some of the results in one study on comparison of machine learning (ML) systems [1] and one study of the pharmacokinetics of phenobarbital [2].
In the former, the authors wrote a Python package with other Python dependencies. All code had been published (both in the supplementary materials and on GitHub), all dependencies were in Python, and the study was relatively recent (2021). Neither of the sources of the code could run on the first try, and modifications were made to attempt to solve this (e.g. inferring a requirements.txt-file from provided materials).
In the latter, we attempted to reproduce the values of the PK parameters and their standard errors using a newer NONMEM version.
In light of the examples we explored various best practices and solutions to the problem of reproducibility for general computations to apply them in the field of pharmacometrics.
Results: In the first example, even with modifications we could not get it to run on our set-up. In the second example we could reproduce similar results (using NONMEM 7.6). The largest difference in results was for the half-life where we got 145 h and the original paper got 141 h (a 3.1% difference).
We propose three different tiers of solutions for reproducibility:
Tier 1: Scripting and using basic package management systems
To make reproducibility possible in practice the computations need to be scripted since manual changes or steps require detailed documentation to reproduce, which is prone to mistakes. The dependencies of the script need to be recorded and included, with full lists with versions of all direct and indirect dependencies. The package managers of the programming languages could be used to simplify this, for example pip/requirements.txt for Python and renv/lock-files in R. This will give a basic level of reproducibility, but often a full workflow in pharmacometrics includes software written in various languages and their compilers. Although you cannot use the same approach as with R or Python dependencies, a list of the software used with their version should still be created. Limiting the number of dependencies can also be useful (e.g. by doing simple calculations yourself in the script or using the same package for more tasks). Finally, we recommend deciding at the beginning whether to freeze your dependencies at the start or rerunning the project at the end and then recording the dependency versions. [3, 4, 5, 6]
Tier 2: Containers
Containers [7, 8] are isolated systems that can run inside of another system. After building a container the computations can be run inside it with the versions of all software being frozen. The container file can then be given to someone attempting to reproduce the computations. This approach requires more setup than the previous tier but also captures more of the complex relationship between dependencies.
Tier 3: Fully reproducible build systems: guix and nix
The big weakness of containers is management of the compile time dependencies, i.e. all software needed to build the containers. This means that it can be difficult to reproduce the container itself. With the advent of the nix and guix functional package managers [9, 10] complete solutions to the problem of reproducible computations exist. These will also take care of transitive compile time dependencies and should also be able to make NONMEM runs exactly reproducible for each version.
Conclusion: Reproducibility of computations is a hard problem, but there currently exist solutions that can solve the problem. We propose some solutions with different levels of effort, as reproducibility is a scale (i.e. doing something is better than nothing). From the proposed solutions, it follows that reproducibility is something that is important to keep in mind in the beginning of a project and follow throughout, since doing so later in the process can be hard, if not impossible.
[1] G. Liu, D. Lu, and J. Lu, “Pharm-AutoML: An open-source, end-to-end automated machine learning package for clinical outcome prediction,” CPT Pharmacomet. Syst. Pharmacol., vol. 10, no. 5, pp. 478–488, May 2021, doi: 10.1002/psp4.12621. [2] T. H. Grasela and S. M. Donn, “Neonatal population pharmacokinetics of phenobarbital derived from routine clinical data,” Dev. Pharmacol. Ther., vol. 8, no. 6, pp. 374–383, 1985, doi: 10.1159/000457062. [3] M. Ziemann, P. Poulain, and A. Bora, “The five pillars of computational reproducibility: bioinformatics and beyond,” Brief. Bioinform., vol. 24, no. 6, p. bbad375, Nov. 2023, doi: 10.1093/bib/bbad375. [4] A. H. C. van Kampen et al., “ENCORE: a practical implementation to improve reproducibility and transparency of computational research,” Nat. Commun., vol. 15, no. 1, p. 8117, Sep. 2024, doi: 10.1038/s41467-024-52446-8. [5] Nature computational tools reporting guidelines. Accessed: Mar. 07, 2025. [Online]. Available: https://www.nature.com/documents/Computational_tools_reporting_guidelines.pdf [6] G. K. Sandve, A. Nekrutenko, J. Taylor, and E. Hovig, “Ten Simple Rules for Reproducible Computational Research,” PLOS Comput. Biol., vol. 9, no. 10, p. e1003285, Oct. 2013, doi: 10.1371/journal.pcbi.1003285. [7] G. M. Kurtzer, cclerget, M. Bauer, I. Kaneshiro, D. Trudgian, and D. Godlove, hpcng/singularity: Singularity 3.7.3. (Apr. 06, 2021). Zenodo. doi: 10.5281/zenodo.4667718. [8] D. Merkel, “Docker: lightweight Linux containers for consistent development and deployment,” Linux J, vol. 2014, no. 239, p. 2:2, Mar. 2014. [9] E. Dolstra, The purely functional software deployment model. PhD thesis, ISBN 90-393-4130-3, 2006. [10] L. Courtès and R. Wurmus, “Reproducible and User-Controlled Software Environments in HPC with Guix,” Jul. 26, 2015, arXiv: arXiv:1506.02822. doi: 10.48550/arXiv.1506.02822.
Reference: PAGE 33 (2025) Abstr 11341 [www.page-meeting.org/?abstract=11341]
Poster: Methodology - Other topics