Huifang you , Lina Keutzer , Ulrika Simonsson
Department of Pharmaceutical Biosciences, Uppsala University, Uppsala, Sweden
Introduction:
In clinical tuberculosis (TB) drug development, phase 2a trials assess early bactericidal activity over two weeks using the time-to-positivity (TTP) biomarker [1]. A standardized methodology has earlier been developed for the analysis of Phase 2a trials in TB [2]. This is followed by a phase 2b study which usually is 8 weeks of treatment with the endpoint culture conversion. This endpoint can be analyzed using semi-parametric or parametric models for time-to-event data (TTE). In this work, we investigated the informativeness of phase 2a TTP biomarker efficacy for the prediction of TTE phase 2b culture conversion in TB using machine learning (ML). Additionally, we wanted to explore the informativeness of different Phase 2a study lengths for the prediction of culture conversion.
Methods:
Time-to-event culture conversion data from a randomized, double-blind, placebo-controlled, phase 3 trial was utilized. The trial evaluated the noninferiority of two moxifloxacin-containing regimens compared to the control regimen for uncomplicated, smear-positive pulmonary tuberculosis [3]. A standardized pharmacometrics model-based early bactericidal activity analysis (EBA) was applied where different linear and nonlinear models were evaluated using nonlinear mixed effects modeling in NONMEM (version 7.5.1, ICON Development Solutions, Hanover, MD, USA) and PsN (version 5.3.1, Department of Pharmacy, Uppsala University) to describe the decline in TTP over time. Stepwise covariate modeling (SCM) was applied to covariate model selection. Inter-individual variability was evaluated in all model parameters. Model evaluations were done using visual predictive checks (VPC), parameter precision, and objective function value (OFV). The difference in TTP from 0 to 14 days (TTP0-14 days) and 0 to 28 days (TTP0-28 days) were used in the ML analysis.
The pharmacometrics analysis was followed by the ML analysis where the TTE culture conversion data was divided into training and testing datasets with four scenarios: (a) an 80% training and 20% testing split of the entire dataset; (b) training on up to 2 weeks of data for testing on up to 2 months; (c) training on up to 4 weeks for testing on up to 2 months; and (d) training on up to 2 weeks for testing on up to 4 weeks. As predictors of TTE of culture conversion the following features were used; height, weight, regimen, cavity, smoking, time since start of treatment, baseline TTP, and TTP0-14 days and TTP0-28 days. Four ML models, extreme gradient boosting (Xgboost), Random forest (RF), K-nearest-neighbors algorithm (KNN), and C-support vector classification (SVC), were applied using Python (version 3.9.1) to predict the TTE of culture conversion at 2 months. Bayesian optimization was used to optimize model hyperparameters. Five-fold cross-validation was applied to the training phase, while the optimal threshold F1 score method was utilized to the imbalanced data. Models were evaluated using ROC-AUC and C-index metrics with Kaplan-Meier (KM) plots to guide the model selection.
Results:
In the ML analysis, scenario A used all the phase 3 TTE datasets with an 80% training and 20% testing split. In this scenario, Xgboost predicted the TTE data with a ROC-AUC of 0.94 and a C-index of 0.86. Scenario B involved training on data for up to 2 weeks and testing on data for up to 2 months. Here, Xgboost was selected as the best model with a ROC-AUC of 0.63 and a C-index of 0.55. Scenario C focused on training on data for up to 4 weeks and testing on data for up to 2 months. The SVC model was the best model with a ROC-AUC of 0.76 and a C-index of 0.64. Scenario D consisted of training on data for up to 2 weeks and testing on data for up to 4 weeks. Again, Xgboost showed better performance with ROC-AUC (0.73) and C-index (0.64). Final ML models for all scenarios showed a good fit to the TTE data as indicated by the KM plots stratified for regimen as the model observation event lines were within the 90% confidence interval. However, a better prediction of TTE culture conversion was seen for scenario C compared to scenario B from week 6 to week 8.
Conclusions:
Machine learning is a useful tool for the analysis of TTE data and can utilize pharmacometrics-based predicted biomarker information as part of the explanatory variables. Phase 2a TTP information predicted the phase 2b outcome and was even better when an extended Phase 2a trial of 4 weeks was used to predict the phase 2b culture conversion.
References:
[1] A. H. Diacon et al., “Time to liquid culture positivity can substitute for colony counting on agar plates in early bactericidal activity studies of antituberculosis agents,” Clinical Microbiology and Infection, vol. 18, no. 7, pp. 711–717, Jul. 2012, doi: 10.1111/j.1469-0691.2011.03626.x.
[2] L. Mockeliunas et al., “Standards for model-based early bactericidal activity analysis and sample size determination in tuberculosis drug development,” Front. Pharmacol., vol. 14, p. 1150243, Apr. 2023, doi: 10.3389/fphar.2023.1150243.
[3] S. H. Gillespie et al., “Four-month moxifloxacin-based regimens for drug-sensitive tuberculosis,” N Engl J Med, vol. 371, no. 17, pp. 1577–1587, Oct. 2014, doi: 10.1056/NEJMoa1407426.
Reference: PAGE 32 (2024) Abstr 11253 [www.page-meeting.org/?abstract=11253]
Poster: Methodology – AI/Machine Learning