Parameter uncertainty in small datasets – evaluation approaches at their limit
Astrid Broeker (1), Sebastian G. Wicha (1)
(1) Dept. of Clinical Pharmacy, Institute of Pharmacy, University of Hamburg, Hamburg, Germany
Objectives: Small patient or subject numbers in pharmacometric analyses possess various restrictions such as higher uncertainty of population parameter estimates, in particular for interindividual variability. Yet, ‘small-n’ studies appear regularly in pilot studies or when specific research questions are addressed and therefore conclusions based on small datasets need to be reliable. This requests assessment of parameter uncertainty, for which various approaches are known, but many come with restrictions, especially regarding small datasets [1]. The aim of this study was (i) to compare techniques of parameter uncertainty evaluation in small datasets, and (ii) to provide guidance on how to assess parameter uncertainty in small datasets.
Methods: Simulation scenarios based on two-compartment pharmacokinetic models were implemented in NONMEM® 7.4.1 with different dataset sizes (n=5-100 subjects, n=10 samples per subject). Parameter uncertainty was determined by bootstrap, sampling importance resampling (SIR), log-likelihood profiling (LLP) and standard errors derived from the variance covariance matrix (SE) using R 3.5.2 and PsN 4.7.0. Stochastic simulations and estimations (SSE) were used to define a reference parameter uncertainty of the simulation examples. The 0-95% confidence intervals (CI) (median and 90% CI of all CIs across n=100 simulations) and the coverage were compared. A real data example (n = 11 subjects, [2]) was evaluated using bootstrap, SIR, LLP and SE and the 0-95% CI of all methods were compared.
Results: Parameter uncertainty of the simulation examples was assessed by bootstrap, SIR, LLP and SE and compared to SSE results. The 95% CI’s of all methods were in good alignment with the SSE for the structural parameters and provided similar results even in very small datasets (n=5 subjects). However, uncertainty of interindividual variabilities (IIVs) was captured much worse, especially in these very small datasets. Bootstrap and SE underestimated the 95% CI for small datasets, while LLP in median overestimated the 95% CI. SIR results were sensitive to the proposal distribution and tended to underestimate parameter uncertainties of the IIVs in case the variance covariance matrix was used as proposal, while (arbitrarily) inflated proposal distributions led to overestimation.
In a second step, we evaluated providing LLP results as input to SIR, which resulted in best alignment of median 95% CI and SSE 95% CI. For example, 95% coverage of IIV of clearance was 85% for SIR and 91% for LLP in a small dataset scenario (n=10 subjects) and 82% and 90% in the respective very small dataset scenario (n=5 subjects), respectively. Bootstrap and SE coverage was lower and also poor for several structural parameters in small datasets, whereas LLP showed coverage rates closest to 95% across the investigated scenarios. A similar pattern of parameter uncertainties assessed by all methods for structural parameters was observed in the real data example.
Conclusions: Bootstrap and SE were least appropriate to evaluate parameter uncertainty, especially regarding IIVs in small datasets. LLP provided robust and conservative parameter uncertainty estimation but tended to overestimate the uncertainty of IIV parameters in very small datasets. SIR can benefit from rational proposal distributions, which might be provided by LLPs and led to most accurate estimations of parameter uncertainties in small datasets.
References:
[1] Dosne A-G, Bergstrand M, Harling K, Karlsson MO. Improving the estimation of parameter uncertainty distributions in nonlinear mixed effects models using sampling importance resampling. J Pharmacokinet Pharmacodyn 2016;43:583–96. doi:10.1007/s10928-016-9487-8.
[2] Broeker A, Wicha SG, Dorn C, Kratzer A, Schleibinger M, Kees F, et al. Tigecycline in critically ill patients on continuous renal replacement therapy: a population pharmacokinetic study. Crit Care 2018;22:341. doi:10.1186/s13054-018-2278-4.