II-066 Christos Kaikousidis

Quantification of model misspecification of population pharmacokinetic models by a machine learning postprocessing correction of the residual variability

Christos Kaikousidis, Robert R Bies, Aristides Dokoumetzidis

Department of Pharmacy, University of Athens / Department of Pharmaceutical Sciences, State University of New York at Buffalo, New York.

Objectives:

In Population Pharmacokinetics (PopPK) residual unexplained variability (RUV) characterizes the remaining observed variability having taken into account the variability that is attributed to the between subject variability of the pharmacokinetic parameters, and it is handled as random noise on the predictions. While this is partly justified, since measurement errors and other random factors can indeed appear as noise, one of the main sources of RUV is model misspecification. Efforts have been made to adress this issue [1-3]. In this work we address the problem of model misspecification, by modelling RUV by Machine learning (ML) methods in a postprocessing step after conventional model building. The practical purpose of the method is two-fold (a) the generation of realistic virtual patients (VP) and (b) the quantification of the extent of model misspecification, by introducing an appropriate metric, to be used as an additional diagnostic of model quality.  

Methods: 

In the proposed methodology, after developing a PopPK model, the individual residual errors IRES=DV-IPRED, were computed, where DV are the observations and IPRED the individual predictions. Our main proposition is that besides the random part of the residual error which explains the noise of the data, i.e. sampling errors and other unexplained factors, there is a significant part of residual error that arises from model misspecification. A Machine Learning model was used to detect any trends and patterns in the residual error and provide a prediction of the residual error, which was denoted as IRESML. In simulation mode, the corrected responses, DVsim, were generated from the model by the equation: DVsim=IPRED+IRESML. The latter allows the simulation of corrected VP profiles. 

For the second step of the methodology, we assume that the portion of the IRES explained by the ML algorithm offers a magnitude of model misspecification of the original mathematical model and is therefore a measure of the quality of the model particularly regarding its capability to produce realistic VP profiles. Therefore, a goodness of fit metric was computed between the “observed” IRES and IRESML to quantify this magnitude. The rationale is that the higher the value of the metric, and therefore the largest the portion of IRES described by the ML, the higher the model mispecification. The metric appropriate for the purpose was chosen to be, the R2, of the IRES vs IRESML since it is properly scaled (from 0 to 1, or 1 to 100%) and is a familiar evaluation metric in modeling procedures. It is denoted as R2ML and was calculated from the following formula: R2ML=100(1-Σ(IRESi – IRESML)2 / Σ(IRESi – IRESavg)2) where Σ sums i from 1 to N, the total number of observations and IRESavg is the average value of IRES.

The methodology was tested in several available datasets: First, IRES was modelled using ML on a PK study of ropinirole, for which a PopPK model was developed while a second deliberately misspecified model was also considered for comparison. Virtual patients were also created for both cases using a leave-one-out cross validation (LOOCV) rationale and were evaluated visually by comparing the IRES vs time plots and VPC plots, before and after correction to determine the improvement in model mispesification and also concentration – time profiles of individual simulated patients before and after correction. Also, in order to explore the behavior of R2ML, the ropinirole models were used while also several other datasets for which PopPK models were already available were examined namely for Busulfan and MMF. Lastly, to showcase the utility of the metric and in order to test it in a completely controlled example, a simulated case study was performed. Two simulated datasets were created using a 2-compartment model, one with a mild and one with a stronger biphasic shape and both were fitted to a 1 compartment model to demonstrate two levels of model mispecification. The R2ML metric was calculated in both cases while the exercise was repeated 100 times to statistically validate the results. 

Results: 

In the ropinirole case, with the optimal PopPK model (model 1) the application of the ML methodology significantly improved the individual predictions. In terms of mean squared error (MSE), the initial model had an MSEuncor= 10783 while the corrected MSEcor=7858. For the poor model (model 2) the improvement was greater since it decreased the MSE from MSEuncor=42527 to MSEcor =24526. This shows that for the more misspecified model, the ML algorithm can achieve more pronounced corrections to the predictions. IRES vs time plots for both models were created using the corrected predictions and the trends and bias which were present in the initial models, were corrected. The individual profiles of virtual patients in both cases looked more realistic and more similar to the corresponding real, observed profiles after the correction, compared to the jagged uncorrected profiles. Corrected VPC plots were also generated and were superior in both cases to the uncorrected ones. It is worth noting that in the case of model 2, while the initial VPC failed to describe the second peak present in the data, the ML correction fixed the discrepancy.

Regarding the study of the quantification metric R2ML, in the Ropinirole case the result for model 1 was R2ML=22% while for model 2 was R2ML=41%, consistent with the fact that model 2 is a heavily misspecified model. In the case study of Busulfan which is an IV model with simple kinetics, and the final 2-CMT model describes the data very accurately, R2ML was 0.08%, refelcting the fact that the ML algorithm was unable to find any trends in IRES, as expected. In the case study of MMF in which the model was not perfect, the ML algorithm had to perform some corrections and the R2ML was 30%. Finally in the simulated scenario, the 1-CMT model perrformed better on the mildly biphasic dataset and much worse on the strongly biphasic, which was reflected in the values of R2ML, being 25% and 47%, respectively. A 2-CMT model was fitted to each of the datasets perfectly as expected, and the value of R2ML was 2%. 

Conclusions: 

We present a novel application of ML in pharmacometrics to address model misspecification by modelling RUV, with the purpose of generating corrected VPs, as well as reporting a metric, R2ML, that can qunatify the magnitude of the mispecifiation. Misspecification is always present in models and in fact the more informative the avaialble dataset, the more pronounsed it usually is. We envisage this new method to be useful for simulation excesises where individual predictions are considered, such as for example for model informed bioequivalence studies [6], while the R2ML is a novel quality measure for models that could have many applications.

References:

  1. Ibrahim MMA, Ueckert S, Freiberga S, Kjellsson MC, Karlsson MO. Model-Based Conditional Weighted Residuals Analysis for Structural Model Assessment. AAPS J. 2019 Feb 27;21(3):34. doi: 10.1208/s12248-019-0305-2.
  2. Karlsson MO, Beal SL, Sheiner LB. Three new residual error models for population PK/PD analyses. J Pharmacokinet Biopharm. 1995 Dec;23(6):651-72. doi: 10.1007/BF02353466.
  3. Ibrahim MMA, Nordgren R, Kjellsson MC, Karlsson MO. Model-Based Residual Post-Processing for Residual Model Identification. AAPS J. 2018 Jul 2;20(5):81. doi: 10.1208/s12248-018-0240-7.
  4. McComb M, Bies R, Ramanathan M. Machine learning in pharmacometrics: Opportunities and challenges. Br J Clin Pharmacol. 2022 Feb;88(4):1482-1499. doi: 10.1111/bcp.14801.
  5. Keutzer L, You H, Farnoud A, Nyberg J, Wicha SG, Maher-Edwards G, Vlasakakis G, Moghaddam GK, Svensson EM, Menden MP, Simonsson USH, On Behalf Of The Unite Tb Consortium. Machine Learning and Pharmacometrics for Prediction of Pharmacokinetic Data: Differences, Similarities and Challenges Illustrated with Rifampicin. Pharmaceutics. 2022 Jul 22;14(8):1530. doi:10.3390/pharmaceutics14081530.
  6. Song E, Lee W, Kim BH. Model-Based Approach for Designing an Efficient Bioequivalence Study for Highly Variable Drugs. Pharmaceuticals (Basel). 2021 Oct 28;14(11):1101. doi: 10.3390/ph14111101.

Reference: PAGE 32 (2024) Abstr 11223 [www.page-meeting.org/?abstract=11223]

Poster: Methodology - New Modelling Approaches

PDF poster / presentation (click to open)