Mohamed Gewily, Gustaf J. Wellhagen, Mats O. Karlsson
Department of Pharmacy, Uppsala University, Uppsala, Sweden
Objectives: Non-parametric Mixed Models for Repeated Measures (MMRM) have evolved to be the gold standard for disease progression data analysis due to their ability to handle patient dropouts (1). Parametric Non-Linear Mixed Effects Models (NLMEM) on the other hand can markedly increase the power in efficacy analyses (2). There exists a spectrum of models from parsimonious NLMEM to parameter rich MMRM, that may be used for longitudinal data depending on the assumptions made. NLMEM can provide precise and unbiased effect size estimates with the fewest number of parameters if the initial assumptions are met, while MMRM can almost always have unbiased estimates, but are less precise (3). In this work, we investigate the difference between MMRM and NLMEM in terms of bias and precision of treatment effect size estimation, as well as in terms of model fit. In the comparison, we include a standard NLMEM, a NLMEM with individual mixture assignment probability, IMA (4), and a parameter rich MMRM. We also outline some ideas to include NLMEM and MMRM in the same analysis.
Methods: We used two real clinical trial data sets with associated published NLMEM: The Movement Disorder Society’s Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) placebo disease progression (5), and the Alzheimer’s Disease Assessment Scale-Cognitive subscale (ADAS-Cog) placebo disease progression (6). Four time points were chosen in both datasets to standardize the evaluation framework; months 0, 3, 6 & 9, and months 1, 6, 12 & 24 respectively. 100 different datasets per study were then generated by randomizing placebo – treatment allocation. In one setting, no treatment effect was added after the randomized allocation. In another setting, an offset effect with a magnitude of 6 points was added in the treatment arm. Three models were then fitted to the generated datasets (model, number of parameters for MDS-UPDRS, ADAS-Cog): i) standard NLMEM (NLMEM 16, 12) ii) mixture model NLMEM – Individual Model Averaging (IMA, 17, 13) iii) MMRM with an unconstrained residual error correlation structure (MMRM, 28, 30). The trial endpoint was chosen to be the placebo-adjusted end-of-treatment effect size. Model averaging was then performed based on weighted hybrid Bayesian Information Criterion (BIC) (BIC-AVG) (7), and weighted Akaike Information Criterion (AIC) (AIC-AVG).
Results: i) for MDS-UPDRS data, IMA showed a superior model fit in terms of OFV, but not BIC (mean BIC): NLMEM (6541), IMA (6548) and MMRM (6658). The treatment effect predictions were relatively biased for NLMEM, but with a precision advantage as seen in mean bias and standard deviation of effect estimates (mean bias, SD effect, accuracy): NLMEM (0.13, 0.39, 0.52), IMA (0.04, 0.46, 0.50), MMRM (0.02, 1.53, 1.55), AIC-AVG (0.09, 0.41, 0.50), BIC-AVG (0.12, 0.39, 0.51) ii) for ADAS-Cog data, MMRM-UN had a better model fit compared to NLMEM on the OFV scale, but not on BIC, with a mean BIC of: NLMEM (11749), IMA (11751) and MMRM (11871). The effect size predictions were unbiased for MMRM, but slightly biased for NLMEM and IMA. Precision was best for NLMEM followed by IMA, with (mean bias, SD effect, accuracy): NLMEM (-0.22, 0.16, 0.38), IMA (-0.14, 0.23, 0.37), MMRM-UN (-0.06, 0.73, 0.78), AIC-AVG (-0.11, 0.69, 0.80) and BIC-AVG (0.22, 0.18, 0.40).
Conclusions: Our results show that NLMEM have superior effect size prediction precision and IMA the best accuracy, while MMRM appears to be the least biased. The choice of MMRM as a standard for longitudinal data analysis is understandable, but can come at the cost of effect estimation accuracy. For the MDS-UPDRS data, none of the applied models contained any covariates. For the ADAS-Cog data, we used a published NLMEM as basis for the comparison (6). We tried to apply the same covariate parametrization to MMRM to render it comparable. We suspect that the chosen covariate model was not optimal and might have favoured MMRM. Model averaging results were different when based on AIC or BIC. The BIC averaging estimates did not benefit simultaneously from NLMEM/IMA and MMRM because NLMEM/IMA always dominated the BIC weight. We could incorporate models with flexibility in between NLMEM and MMRM to enhance the model averaging. Such models could be MMRM with more constrained variability or NLMEM with more flexible time course capturing functions. Alternatively, another method would be to come up with an exclusion protocol for NLMEM to favour MMRM when needed.
References:
[1] Mallinckrodt CH, Lane PW, Schnell D, Peng Y, Mancuso JP. Recommendations for the Primary Analysis of Continuous Endpoints in Longitudinal Clinical Trials. Drug Inf J. 2008 Jul;42(4):303–19.
[2] Karlsson KE, Vong C, Bergstrand M, Jonsson EN, Karlsson MO. Comparisons of Analysis Methods for Proof-of-Concept Trials. CPT Pharmacomet Syst Pharmacol. 2013 Jan 16;2:e23.
[3] Wellhagen GJ, Karlsson MO, Kjellsson MC. Comparison of Precision and Accuracy of Five Methods to Analyse Total Score Data. AAPS J. 2020 Dec 17;23(1):9.
[4] Chasseloup E, Tessier A, Karlsson MO. Assessing Treatment Effects with Pharmacometric Models: A New Method that Addresses Problems with Standard Assessments. AAPS J. 2021 May 3;23(3):63.
[5] Access Data | Parkinson’s Progression Markers Initiative [Internet]. [cited 2022 Apr 5]. Available from: https://www.ppmi-info.org/access-data-specimens/download-data
[6] Ito K, Corrigan B, Zhao Q, French J, Miller R, Soares H, et al. Disease progression model for cognitive deterioration from Alzheimer’s Disease Neuroimaging Initiative database. Alzheimers Dement J Alzheimers Assoc. 2011 Mar;7(2):151–60.
[7] Delattre M, Lavielle M, Poursat M-A. A note on BIC in mixed-effects models. Electron J Stat Electron Only. 2014 Jan 1;8.
Reference: PAGE 30 (2022) Abstr 10118 [www.page-meeting.org/?abstract=10118]
Poster: Methodology - New Modelling Approaches