Ron Keizer 1, Martin Bergstrand, Jasmine Hughes
1 Insightrx (San Francisco, United States)
Background: It is known that visual predictive checks (VPC) cannot be used reliably when real world data (RWD) or data from adaptive trials are analyzed, specifically in the cases of modifications in treatment or sampling times in response to measured concentrations or outcomes.[1] While prediction-correction (pcVPC) may help in some cases (e.g. changes only in dose amount), it still provides an incorrect diagnosis for most other cases like adaptation of dosing interval, sampling times and/or follow-up duration. Here we present a potential solution to this problem, in the form of propensity-score-matched VPCs (pmVPC).
The pmVPC requires a modification in the simulation step of the VPC: instead of randomly assigning drawn parameter sets to individual dataset designs from observed subjects, they are now matched based on the individual maximum-likelihood parameter estimates obtained from the observed data. Matching is done using propensity-score matching (on single or multiple model parameters). After matching, simulations are performed as normal and used to construct the VPC.
Objectives: To investigate the ability of pmVPCs to overcome the limitations of VPCs and pcVPCs for RWD and adaptive trials.
Methods: Four cases of VPCs for RWD were investigated based on simulated data from a known true model. All cases were based on Bergstrand et al.[2] in which a treatment adaptation was introduced in response to measured drug concentrations. The original case implemented only dose adaptation (case 0), but here additional scenarios were implemented that studied interval adaptation (case 1) and sampling-schedule adaptation (case 2 and 3). All cases erroneously indicated model misspecification using the regular VPC, and in all but case 0 also indicated misspecification using the pcVPC. Separately, for these same cases also several control scenarios were run, now using misspecified models (scenario 1: population CL +20%, scenario 2: linear clearance when the true model included Michaelis-Menten kinetics) to investigate whether the pmVPC was still able to correctly diagnose model misspecification when present. Propensity-score matching was performed using the MatchIt package in R [3], based on individual estimates for CL and V, and VPCs were made using the vpc package [4].
Results: In all four cases, the pmVPC was able to correctly diagnose that the true model did not suffer from misspecification. In all control cases, and for both model misspecification scenarios, the pmVPC was able to correctly diagnose that the model was misspecified.
Discussion: In the presence of treatment adaptation as seen in RWD, the pmVPC is able to correctly diagnose true and misspecified models, in contrast to regular VPCs and pcVPCs. An additional benefit over pcVPCs is that it does not apply any transformation of the observed data. Future work should explore more complex models and additional real world scenarios. In combination with other novel VPC methodology [5,6] the pmVPC will offer an improved toolbox for efficient, reliable and interpretable model evaluation for complex datasets.
References:
[1] Hughes JHH, Bergstrand M, Keizer RJ. The visual predictive check and real-world data. CPT (In press)
[2] Bergstrand M and Karlsson MO. Prediction-Corrected Visual Predictive Checks for Diagnosing Nonlinear Mixed-Effects Models. AAPSJ 2011
[3] https://github.com/kosukeimai/MatchIt
[4] https://github.com/ronkeizer/vpc
[5] Ibrahim MMA, Jonsson EN, Bergstrand M. The Reference-Corrected Visual Predictive Check: A More Intuitive Diagnostic for Non-Linear Mixed Effects Models. AAPS J. 2025 Apr 29;27(4):86.
[6] Ibrahim MMA, Hughes JHH, Keizer RJ, Bergstrand M. Reference-Corrected VPCs: Addressing Model Evaluation Challenges with Real-World Data and Adaptive Designs. PAGE 2026
Reference: PAGE 34 (2026) Abstr 12128 [www.page-meeting.org/?abstract=12128]
Poster: Methodology - Model Evaluation