Shamin M Saffian, Daniel F.B. Wright, Stephen B. Duffull
University of Otago
Objectives:
An assessment of predictive performance is required to demonstrate that a model is fit for purpose and has potential utility in the intended setting. In the setting of non-repeated measures data (e.g. assessment of model-based dose predictions) we expect a linear relationship between the predicted and observed data. When plotted on an x,y plot, the predictions should cluster close to the line of identity. Deviations from the line of identity may suggest bias in the model predictions. A method for assessing predictive performance was described by Sheiner and Beal [1] where bias is estimated using mean prediction error (MPE). MPE provides a measure of the magnitude and direction of bias across the range of observed data while the 95% confidence interval (CI) provides a statistical criterion for the presence of bias. Using this method, the data are assumed to arise from a single bin. We propose that MPE will be insensitive to bias when the relationship between prediction error and the observed data displays a systematic trend over the range of the observations. The objective of this study is to illustrate a method for detecting systematically biased predictions over the range of observations.
Methods:
Overall Approach: The proposed method represents a generalised form of the MPE (single bin) approach proposed by Sheiner and Beal [1]. Here, multiple bins were created across the observed data. The method was developed and evaluated in three steps.
Observed data were binned and the slope of the MPEs vs bin was determined.
- Starting with n=2 bins, n bins of equal width were created across the x-axis.
- The mean prediction error (MPE) of each of n bins was calculated.
- The MPE value was regressed against the bin number.
- The steps above were repeated until n=the maximum number of bins.
The asymptomatic slope at infinite bins was determined.
- The slope values were regressed against the number of bins using an exponential model to determine the asymptotic slope at infinite number of bins.
- If the 95% CI of the asymptotic slope included zero, then no systematic bias was concluded.
The method was evaluated for seven scenarios, illustrating different patterns of systematic bias; (1) unbiased predictions (control), (2) linear, off-set, bias (predictions are shifted below the line of identity), (3) linear, bidirectional, bias (slope of the regression line <45 degrees), (4) non-linear, unidirectional bias at larger observations, (5) non-linear, unidirectional bias at lower observations, (6) combined off-set (linear) and unidirectional (non-linear) bias, and, (7) curvilinear, non-monotonic, bias.
All simulations were conducted in MATLAB and data sets included 50 predicted and observed doses. Random noise was simulated assuming a normal distribution with a mean of zero. A variance 0.25 was used as this value was sufficient to provide random variability but still maintain the shape of bias. MPE (95% CI) using a single bin and the asymptotic slope (95% CI) at infinite bins were determined for each simulated scenario.
Results:
In the control scenario, both the single bin MPE and asymptotic slope method correctly specified the absence of bias. For scenario 2 (off-set bias), the MPE method identified the presence of a bias while the asymptotic slope method correctly identified that the bias was not dependent on the size of the observations. The single bin method did not identify bias in scenarios 3-6. In contrast, the asymptotic slope method correctly detected systematic bias in these scenarios, all of which were characterised by a monotonic deviation from the line of identity (scenarios 3, 4, 5, and 6). Neither method could detect curvilinear bias (scenario 7).
Conclusions:
A method for numerically detecting systematic deviation in model-predictions has been proposed. It provides an additional interpretation to the standard single bin MPE approach. The method has similarities to visualizing a LOESS regression, which is useful for visualising deviation from the line of identity, but differs in that it provides numerical and statistical quantities associated with the deviation. The method could be automated and implemented in statistics software.
References:
[1] Sheiner LB and Beal SL J Pharmacokinet Biopharm (1981) 9,503-512.
Reference: PAGE 27 (2018) Abstr 8536 [www.page-meeting.org/?abstract=8536]
Poster: Methodology - Model Evaluation