SAEM in Pumas – A systematic evaluation and benchmarking
Chris Elrod(1,2), Andreas Noack(1,2), Patrick Mogensen(1,2), Vaibhav Dixit(1,2), Vijay Ivaturi(1,3)
Pumas-AI, Julia Computing, University of Maryland
Objectives: Stochastic Approximation Expectation Maximization (SAEM) is a robust algorithm for fitting Non-Linear Mixed Effects (NLME) models. Pumas 2.0 introduces SAEM support through a new model type, the PumasEMModel. Here, we fit a menagerie of pharmacokinetic models in Pumas, NONMEM[1], and Monolix[2], assessing their agreement in estimated parameter values and standard errors, and benchmarking runtime performance.
Methods: To facilitate an efficient implementation of the SAEM algorithm, Pumas 2.0 introduces the PumasEMModel. This model type is more structured than the standard PumasModel, with variances of the random effects and dispersion parameters of the error model being defined implicitly by the random effects and error model definitions, rather than explicitly as in the PumasModel. While making these parameters internally managed limits the scope of models possible to express, this structure enables optimizations such as using sufficient statistics for the maximization step of SAEM -- critical for high performance -- and reducing the number of partial derivatives taken through the model dynamics when fit using a Laplace approximation. Sampling for the SAEM expectation step is done via random walk Metropolis-Hastings with adaptive step size and thinning rate. The Laplace fit of the PumasEMModel allows us to compare linearized maximum likelihood fits with NONMEM in addition to comparing the SAEM fits.
We tested models featuring a variety of dynamical models with one and four compartments, solving the dynamics numerically as well as analytically when possible. We repeated each fit three times and recorded the minimum runtime. We used a single thread while fitting to increase consistency. We estimated standard errors through linearization. z
Results: Estimated parameter values and standard errors produced by Pumas, NONMEM, and Monolix were generally in agreement, while Pumas’ runtime performance was comparable to Monolix and significantly better than NONMEM, given similar stopping criteria and differential equations solver and tolerance settings. The PumasEMModel supports most of the existing PumasModel utilities and diagnostics, such as Latexify.jl integration and plots of observations vs individual predictions, or visual predictive checks of simulated populations.
Conclusions: Pumas’s new SAEM implementation is compelling and competitive with the existing state of the art, offering solid performance, a reproducible scripting workflow, and rich diagnostics.
References:
[1] Beal S, Sheiner L. The NONMEM system. Am Stat. 1980;34(2):118–9.
[2] Monolix version 2020R1. Antony, France: Lixoft SAS, 2020. http://lixoft.com/products/monolix/