Claire Kong , John Maringwa, Chandni Valiathan
Clinical Pharmacology & Pharmacometrics, Janssen Research & Development, LLC
Objectives:
Model-based meta-analysis (MBMA) uses reported clinical trials data and pharmacologic principles to integrate a wider spectrum of data across dose levels, observation times, and covariates to compare treatments.
A continuous outcome aggregate-level data point is ideally a pair; mean value and precision measure (standard error/deviation, SE/SD). When reported SEs are fixed weights in the model, no estimation of residual variance is required [1]. Measures of precision are often unreported in sources, henceforth “missing data”. A pragmatic approach weights outcomes by sample size [2, 3], assuming a common between-subject variability (BSV) across studies. Utilizing reported precision values is preferred since these optimally represent variability in each arm.
Through simulations, the impact on parameter estimation of different extents of missing data combined with multiple techniques to address the missingness issue in MBMA was investigated. The aim was to provide suggestions on approaches to apply in the face of missing precision data.
Methods:
Literature data from 16 placebo-controlled trials of antidepression treatments venlafaxine (ven) (10;1289 pts), and fluoxetine (flu) (8;982 pts), and 1161 placebo-treated pts were used. Ven doses ranged from 25 to 375 mg/day, while flu had doses of 20, 40, and 60 mg/day, enabling assessment of dose-response. Change from baseline in Hamilton Depression Rating scale, the clinical endpoint of interest, was analyzed at the primary timepoint of each study.
The dose-response model structure was:
Yij =eoi+Emaxk*dkij/(ED50k+dkij)+ eij
Yij: outcome in arm j of trial i, eoi: non-parametric trial-specific placebo response, Emaxk: drug k maximal effect, dkij: dose for drug k with potency ED50k, and eij:residuals with mean 0 and variance var(eij )=σ2ij/Nij, σij: BSV, Nij :sample size. When dose-response was not supported an overall drug effect combining all dose levels versus placebo was used.
Six different approaches to address the missing data issue were explored: (1) weighting by sample size (2) replace all or (3) only missing values with weighted mean of reported SDs, (4) replace all or (5) only missing values with the maximum reported SD, and (6) impute missing values using random forest (RF) [4]. Approach (4) reflected the “worst-case scenario” for benchmarking. For a baseline for comparison, estimates in approach (6) and the corresponding measures of precision were considered as the “true values” for simulating data.
Dose levels and number of trials (20) were set comparable to the example. Sample sizes of 50, 100, and 150 were used. About 49% of study arms in the example had missing data. A grid of % values of 5, 10, 20, 50, 70, and 90 reflected the simulated extent of missing data. BSV was based on resampling SDs from approach (6). The “ideal case” was when the resampled SDs were used as fixed weights in the model. Model uncertainty was incorporated by resampling from the variance-covariance matrix of the “true model”.
Precision on parameter estimates was reflected by the % relative SE, %RSE=100*SE/estimate; the lower the better. Bias, total variance, and mean squared error (MSE) of estimates were also examined.
Results:
The model included an additive overall drug effect for flu and an E-max type dose-response for ven. No covariates were tested since this was not the focus.
Fixed effects parameter estimates were mostly comparable across all six approaches although those for the “worst-case scenario” appeared to differ somewhat from other approaches. Notable differences were on precision. %RSE were generally lowest in approach (6) and (2). Unsurprisingly, the largest %RSE were in approach (4). Interestingly, these were comparable to approach (1).
Simulations suggested that next to the “worst-case scenario”, approach (1) was associated with lowest precision. Approach (5) %RSE increased with increasing extent of missing data. At around 70% missingness, %RSE were comparable to approach (1), suggesting the latter may result in uncertain parameter estimates. Bias, variance, and MSE results did not provide conclusive patterns, further investigations are warranted.
Conclusions:
When some measures of precision are reported, it may be recommended to impute the missing data using the weighted mean of available data or by RF and fix these as weights in the model. Routine weighting by sample size may result in uncertain estimates, culminating in wide confidence intervals that provide limited insights.
References:
[1] Heisterkamp, S.H., van Willigen, E., Diderichsen, P.M. & Maringwa, J. Update of the nlme package to allow a fixed standard deviation of the residual error. The R Journal. 9, 239-251 (2017).
[2] Daniele et.al (2023). Overall and complete response rates as potential surrogates for overall survival in relapsed/refractory multiple myeloma. Future oncology (London, England), 19(6), 463–471. https://doi.org/10.2217/fon-2022-0932.
[3] Daniele et.al (2022). Response rates and minimal residual disease outcomes as potential surrogates for progression-free survival in newly diagnosed multiple myeloma. PloS one, 17(5), e0267979. https://doi.org/10.1371/journal.pone.0267979.
[4] Shah, A. D., Bartlett, J. W., Carpenter, J., Nicholas, O., & Hemingway, H. (2014). Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study. American journal of epidemiology, 179(6), 764–774. https://doi.org/10.1093/aje/kwt312.
Reference: PAGE 32 (2024) Abstr 10817 [www.page-meeting.org/?abstract=10817]
Poster: Methodology - Other topics