Estelle Chasseloup (1), Mats O. Karlsson (1)
(1) Department of Pharmacy, Uppsala University, Uppsala, Sweden.
Background
Model averaging across drug models[1] (MAD), individual model averaging[2] (IMA) and corrected LRT[3] (cLRT) has shown to have a good control of the type I error in different context of drug effect assessment using real or simulated data.
Objectives
Assess MAD, IMA and cLRT in the same real data framework, together with other approaches: standard (STD), relative standard (RSTD), randomized cLRT (rcLRT), and model averaging across drug and placebo (MADP) models.
Methods
The 7 approaches were assessed using real placebo data on type I error first and when controlled, on power and drug effect estimates in the context of drug effect assessment using balanced 2 arms designs. All simulations and estimations were performed with NONMEM 7.4 using FOCE and PsN.
Data ADAS-cog scores from 817 untreated patients modelled as a continuous data using a published model[4]. Four simulated drug effect scenarios were generated in addition to the raw data consisting in a no drug effect scenario: S1 offset of +2 (S2 with 30% IIV), S3 disease modifying of +2 at the last observation (S4 with 30% IIV).
Approaches For all the approaches (1) the Akaike information criteria (AIC) selects an alternative hypothesis (H1) among a predefined set of models, and (2) a statistical test discriminates between H1 and the null hypothesis (H0).
In STD H0 consists of a placebo model applied to all subjects and H1 adds a drug model to the treated subjects. In RSTD the drug model is fitted to all subjects in H0, allowing different estimates for treated individuals in H1. MAD (MAPD) selects only H1 (both H0 and H1) within a set of predefined models built according to STD. In IMA all subjects have, through a mixture feature, a probability of being described by the drug model. This probability θ is estimated (H1), or not (H0), depending on the treatment allocation.
cLRT and rcLRT are an extension of STD where the LRT is modified: the cut-off value is set to the 95th percentile of the dOFV distribution obtained using the same procedure over 100 simulations from H0 for cLRT, or over 100 permutations for rcLRT.
Assessment The type I error was assessed on real placebo data where the patients were randomized (1:1) to artificial placebo or drug treatment. Power and drug effect estimates were assessed on the placebo data modified by the addition of a simulated drug effect to the individuals allocated to treatment. n=100 randomizations were performed to mimic n controlled trials with or without drug effect. Four drug models were investigated: offset or disease modifying with or without IIV.
The type I error (power) rate was the frequency with which the best H1, according to AIC, was significant (a=0.05) relative to H0 when fitting the placebo data (modified with drug effect). Accuracy in drug estimates was computed using the RMSE.
Results
The type I error rate was controlled for the rcLRT and the IMA approaches, equal to 6% for both. Other approaches had a type I error of 17% for RSTD, and of 100% for STD, cLRT, MAD and MAPD.
The power was 95, 92, 91 and 91% (39, 37, 8 and 12% ) for rcLRT (IMA) for S1, S2, S3 and S4 respectively.
The RMSE was 0.24, 0.44, 1.48 and 1.48 (1.41, 1.44, 2.00 and 1.98) for rcLRT (IMA) for S1, S2, S3 and S4 respectively.
Conclusions
This work compares 7 NLMEM approaches to test for drug effect in the same framework using real placebo data. All approaches but IMA and rcLRT had inflated type I error which can be explained by the misspecification of the placebo model, arising from the use of real placebo data, absent from the previous assessments for cLRT and MAD. rcLRT handles the placebo misspecification by computing the cut-off values for the statistical test via a permutation test. rcLRT had a good power (4 scenarios > 90%) contrary to IMA (4 scenarios < 40%). The RMSE was between 1.5 and 2 for all scenarios except for rcLRT when simulating with an offset drug effect. Both approaches suffered from the greedy behavior of AIC in the selection step, often dismissing the “true” simulated drug model.
References
[1] Aoki, Y., Röshammar, D., Hamrén, B. & Hooker, A. C. Model selection and averaging of nonlinear mixed-effect models for robust phase III dose selection. J. Pharmacokinet. Pharmacodyn. 44, 581–597 (2017).
[2] Chasseloup, E., Tessier, A. & Karlsson, M. O. Assessing Treatment Effects with Pharmacometric Models: A New Method that Addresses Problems with Standard Assessments. AAPS J. 23, 63 (2021).
[3] Buatois, S., Ueckert, S., Frey, N., Retout, S. & Mentré, F. cLRT-Mod: An efficient methodology for pharmacometric model-based analysis of longitudinal phase II dose finding studies under model uncertainty. Stat. Med. 40, 2435–2451 (2021).
[4] Ito, K. et al. Disease progression model for cognitive deterioration from Alzheimer’s Disease Neuroimaging Initiative database. Alzheimers Dement. J. Alzheimers Assoc. 7, 151–160 (2011).
Reference: PAGE 30 (2022) Abstr 10137 [www.page-meeting.org/?abstract=10137]
Poster: Methodology - New Modelling Approaches