Evaluation of four distinct weighting schemes in a model averaging and selection approach in model-informed precision dosing of continuously infused vancomycin
David W. Uster (1), Astrid Heus (2,3), Annemie Somers (2), Veerle Grootaert (3), Nele Vermeulen (4), Diana Huis in’t Veld (5), Pieter De Cock (2,6), Sebastian G. Wicha (1)
(1) Dept. of Clinical Pharmacy, Institute of Pharmacy, University of Hamburg, Hamburg, Germany (2) Department of Pharmacy, Ghent University Hospital, Ghent, Belgium (3) Department of Pharmacy, AZ Sint-Jan Brugge Oostende av, Bruges, Belgium (4) Department of Pharmacy, OLV Aalst, Aalst, Belgium (5) Department of General Internal Medicine, Ghent University Hospital, Belgium (6) Heymans Institute of Pharmacology, Ghent University, Ghent, Belgium
Objectives: Optimal dosing of the antibiotic vancomycin in potentially life-threatening infections can be supported by population pharmacokinetic (PK) models. Although the process, termed model-informed precision dosing (MIPD), is recommended in vancomycin therapy,[1] it requires an adequate model. Choosing the model manually is prone to errors, potentially costly and requires pharmacometric knowledge.[2], [3] Two recently published algorithms, which are automatically selecting the best (model selection algorithm; MSA) or an set of models (model averaging algorithm; MAA) for an individual patient amongst a set of candidate models, have been proven to be superior over using a single model for MIPD.[4] Thereby, the algorithms are individually weighting the predictions of the different candidate models. The present study aimed to derive and evaluate four different weighting schemes and to compare the algorithms’ predictive performance in a case study with vancomycin.
Methods: The MAA/MSA comprised two essential steps: First, the individual PK parameters were estimated based on observed data with a set of 7 population PK models. Subsequently, the forecasted data were weighted model-wise, according to the following criteria: i) the individual objective function value (W_OFV); ii) the Akaike criterion (W_AIC); iii) the summed squared residuals (W_SSE); or iv) the extended least squares (W_ELS). The MAA averaged the predictions at each forecasted time point using the set of models jointly. In contrast, the MSA algorithm selected the best fitting, i.e. highest weighted, model.
The exactness of these algorithms was evaluated in vancomycin PK data of 181 adult patients receiving continuous infusions. In detail, the predictive performance of the four weighting schemes were compared using NONMEM 7.4.3 [5] and R [6] in two scenarios: supplying measurements from one dosing interval to forecast concentration-time data in a subsequent dosing interval blinded to the Bayesian estimation (Bayesian forecasting, BF) or by providing all concentration-time data (Bayesian estimation, BE). The bias and root mean square error (RMSE) were used to evaluate accuracy and (im-)precision, respectively.
Results: The predictions from the individual models varied greatly (bias\RMSE: -4.5 – 7.9 mg/L \ 5.8 – 11.0 mg/L in BF; -0.7 – -1.1 mg/L \ 3.2 – 4.8 mg/L in BE). For MAA in BF, the predictions were constantly precise (RMSE: 6.0 mg/L, 6.0 mg/L, 6.4 mg/L, 5.9 mg/L in W_OFV, W_AIC, W_SSE, W_ELS, respectively), while the accuracy was always high (0.2 mg/L, -0.8 mg/L, 0.3 mg/L, -0.4 mg/L). In BE, an improved bias between -0.3 and -0.4 mg/L and lower RMSE between 2.7 and 3.1 mg/L across the 4 MAA weighting schemes were observed.
Comparing the MAA weightings across the scenarios, the W_SSE resulted in the worst RMSE in BF (6.4 mg/L), but displayed the best precision in BE (2.7 mg/L). W_OFV resulted in the overall best metrics (bias\RMSE: 0.2\6.0 mg/L in BF; -0.3\3.0 mg/L in BE).
Although the performance of the 4 weighting schemes in the MSA reached the same level as the MAA in BE, the performance was overall slightly worse in BF (bias -0.6 mg/L, -1.4 mg/L, 0.5 mg/L, -1.5 mg/L; RMSE 6.4 mg/L, 6.9 mg/L, 7.1 mg/L, 6.5 mg/L). In detail, the confidence interval of the bias values included 0 in W_OFV and W_SSE only, and the RMSE was between 0.4 and 0.9 mg/L higher compared to the MAA weighting schemes.
Interestingly, the subtle differences across the 4 MSA schemes were not reflected by the patient-wise selection frequency of the models: although the Thomson model [7] was selected in most of the patients using W_OFV (BF: 46%; BE: 56%) or W_ELS (BF: 65%; BE: 83%), W_AIC and W_SSE led to a more heterogenous selection of models. W_AIC, for example, was dominated by the Adane, Medellin-Garibay and Revilla models [8]–[10], each being selected in 17% – 27% of the patients.
Conclusions: In conclusion, although different models were dominating the MAA and MSA for different weighting schemes, the differences in the resulting predictive performance were small. Nonetheless, W_OFV and W_ELS were outperforming W_AIC and W_SSE in the present analysis. W_AIC was overly penalizing more complex models when supplying only a few samples, while W_SSE was neglecting the prior information included in the model structure and thus might overfit the data. Based on this study, we recommend the W_OFV in MAA/MSA when used in MIPD of vancomycin, as currently implemented in the MIPD software TDMx.[11]
References:
[1] M. J. Rybak et al., “Therapeutic monitoring of vancomycin for serious methicillin-resistant Staphylococcus aureus infections: A revised consensus guideline and review by the American Society of Health-System Pharmacists, the Infectious Diseases Society of America, the Pediatr,” Am. J. Heal. Pharm., vol. 77, no. 11, pp. 835–864, May 2020.
[2] R. ter Heine et al., “Prospective validation of a model-informed precision dosing tool for vancomycin in intensive care patients,” Br. J. Clin. Pharmacol., vol. n/a, no. n/a, p. bcp.14360, Jun. 2020.
[3] A. Broeker et al., “Towards precision dosing of vancomycin: a systematic evaluation of pharmacometric models for Bayesian forecasting,” Clin. Microbiol. Infect., vol. 25, no. 10, pp. 1286.e1-1286.e7, Oct. 2019.
[4] D. W. Uster et al., “A Model Averaging/Selection Approach Improves the Predictive Performance of Model-Informed Precision Dosing: Vancomycin as a Case Study,” Clin. Pharmacol. Ther., vol. 109, no. 1, pp. 175–183, Jan. 2021.
[5] R. J. Bauer, “NONMEM Tutorial Part I: Description of Commands and Options, With Simple Examples of Population Analysis,” CPT Pharmacometrics Syst. Pharmacol., vol. 8, no. 8, p. psp4.12404, Jun. 2019.
[6] R Core Team, “A Language and Environment for Statistical Computing,” R Foundation for Statistical Computing, vol. 2. Vienna, Austria, 2020.
[7] A. H. Thomson, C. E. Staatz, C. M. Tobin, M. Gall, and A. M. Lovering, “Development and evaluation of vancomycin dosage guidelines designed to achieve new target concentrations,” J. Antimicrob. Chemother., vol. 63, no. 5, pp. 1050–1057, Mar. 2009.
[8] E. D. Adane, M. Herald, and F. Koura, “Pharmacokinetics of Vancomycin in Extremely Obese Patients with Suspected or Confirmed Staphylococcus aureus Infections,” Pharmacother. J. Hum. Pharmacol. Drug Ther., vol. 35, no. 2, pp. 127–139, Feb. 2015.
[9] S. E. Medellín-Garibay, B. Ortiz-Martín, A. Rueda-Naharro, B. García, S. Romano-Moreno, and E. Barcia, “Pharmacokinetics of vancomycin and dosing recommendations for trauma patients,” J. Antimicrob. Chemother., vol. 71, no. 2, pp. 471–479, Feb. 2016.
[10] N. Revilla, A. Martín-Suárez, M. P. Pérez, F. M. González, and M. del M. Fernández de Gatta, “Vancomycin dosing assessment in intensive care unit patients based on a population pharmacokinetic/pharmacodynamic simulation,” Br. J. Clin. Pharmacol., vol. 70, no. 2, pp. 201–212, Aug. 2010.
[11] S. G. Wicha, M. G. Kees, A. Solms, I. K. Minichmayr, A. Kratzer, and C. Kloft, “TDMx: A novel web-based open-access support tool for optimising antimicrobial dosing regimens in clinical routine,” Int. J. Antimicrob. Agents, vol. 45, no. 4, pp. 442–444, Apr. 2015.