Andreas Noack 1, David Müller-Widmann 1, Vijay Ivaturi 1,2
1 PumasAI (Dover, United States of America), 2 Manipal Academy of Higher Education (Manipal, India)
Introduction:
The First-Order Conditional Estimation (FOCE) method remains central to the application of non-linear mixed-effects models in pharmacometrics. Datasets of several thousand subjects are now commonplace, as are computationally expensive inference methods such as the bootstrap and sampling-importance-resampling (SIR). Computational efficiency is therefore an important consideration when choosing between available implementations of the FOCE method.
Recent changes to the gradient evaluation of the FOCE objective in the Pumas software package have led to meaningful gains in efficiency, through both the introduction of specialized directional derivatives for use during line search and the elimination of redundant computations in existing gradient code.
Objectives:
The goal of this analysis is to quantify the impact of these recent changes to gradient evaluation in Pumas, using NONMEM as a reference.
Methods:
Since Pumas was first released in 2019, all derivative calculations for the FOCE objective function have been based on the forward-mode automatic differentiation implementation ForwardDiff [1]. The calculation relies on a decomposition of the FOCE gradient [2], which involves second-order derivatives with respect to population and random effect parameters through an ODE solver [3]. A direct application of automatic differentiation to this problem is consequently quite inefficient. Recent changes to Pumas eliminate redundancy in the second-order derivative calculation, reducing the cost of gradient evaluation by roughly half.
A related but separate change concerns derivative calculation during the line search of the outer optimization. Many line search methods require derivative evaluations along the search direction, typically expressed as a functional of the full gradient. A specialized directional derivative that bypasses full gradient evaluation is substantially faster, and such an approach has now been implemented in Pumas, enabling more efficient line search methods.
The new Pumas implementation is benchmarked against NONMEM 7.6.0 across a range of pharmacokinetic (PK) models, from simple single-dose linear PK models to computationally intensive target-mediated drug disposition (TMDD) models in quasi-steady state. All runs use four cores in parallel on the JuliaHub cloud platform, with MPI-based parallelism in NONMEM and multi-threading in Pumas. Default configurations are used in both packages.
Results:
Pumas is substantially faster than NONMEM across all models tested. For linear PK models, Pumas runs five to six times faster, while for TMDD models the speedup ranges from two to four times. For the quasi-steady-state model specifically, Pumas completes in approximately 400 seconds compared to around 800 seconds for NONMEM. The number of iterations required for convergence is generally lower in Pumas, suggesting that the Hager-Zhang line search algorithm contributes meaningfully to the performance gains. Parameter estimates were virtually identical between the two packages.
Conclusions:
When implemented efficiently, forward-mode automatic differentiation can substantially accelerate the estimation of NLME models under the FOCE approximation relative to finite-difference-based approaches. Although automatic differentiation has long been standard practice in machine learning, it only entered pharmacometrics software in 2019 with the release of Pumas, followed more recently by Phoenix NLME [4]. The late adoption likely reflects the historical dependence on specialized libraries or compilers to support automatic differentiation. Building Pumas on Julia, a modern programming language designed for numerical computing, provided straightforward access to capable automatic differentiation libraries, and the performance benefits are evident in the results presented here.
References:
[1] Revels, Jarrett, Miles Lubin, and Theodore Papamarkou. “Forward-mode automatic differentiation in Julia.” arXiv preprint arXiv:1607.07892 (2016).
[2] Almquist, Joachim, Jacob Leander, and Mats Jirstrand. “Using sensitivity equations for computing gradients of the FOCE and FOCEI approximations to the population likelihood.” Journal of pharmacokinetics and pharmacodynamics 42, no. 3 (2015): 191-209.
[3] Rackauckas, Christopher, and Qing Nie. “Differentialequations.jl – a performant and feature-rich ecosystem for solving differential equations in Julia.” (2017).
[4] Chen, Rong, Mark Sale, Alex Mazur, Michael Tomashevskiy, Shuhua Hu, James Craig, Mike Dunlavey, Robert Leary, and Keith Nieforth. “ADPO: automatic-differentiation-assisted parametric optimization.” Journal of Pharmacokinetics and Pharmacodynamics 52, no. 5 (2025): 53.
Reference: PAGE 34 (2026) Abstr 12016 [www.page-meeting.org/?abstract=12016]
Poster: Methodology - Estimation Methods