Shadi Ghiasi1, Sanjeev Roy2, Lena Klopp-Schulze3, Josiah Ryman4, Ying Li4, Flavie Moreau4, Florence Casset-Semanaz4, Eduardo Vianna5, Karthik Venkatakrishnan4, Nadia Terranova1
1Quantitative Pharmacology, Ares Trading S.A., an affiliate of Merck KGaA, Darmstadt, Germany, 2Ares Trading SA, an affiliate of Merck KGaA, Darmstadt, Germany, 3Quantitative Pharmacology, Merck KGaA, 4EMD Serono Research and Development Institute, Inc., , 5Development Unit Neurology and Immunology, Merck KGaA,
Objectives: Systemic Lupus Erythematosus (SLE) is a systemic autoimmune disease characterized by significant heterogeneity in immunological features and clinical manifestations, impacting multiple organ systems. SLE is characterized by recurring bouts of increased disease activity (flares), mild to life-threatening in severity. Measuring SLE disease activity accurately remains a challenging and demanding task given the complex, multi-system nature of the disease. SLE flares derived from the British Isles Lupus Assessment Group (BILAG) are a key component of response assessment [1]. However, disease heterogeneity and complexity make the early prediction of flares challenging. In this study, we leverage a large longitudinal, multivariable dataset of SLE patients using state-of-the-art Machine Learning (ML) methods to forecast SLE flares. Methodology: The dataset coming from the TransCelerate program, an initiative for sharing historical data across biopharma companies, includes patient-level data in the placebo plus standard of care arms of 7 randomized controlled trials involving 1933 patients (NCT02446899 (ANIFROLUMAB), NCT01262365 (EMBODY1), NCT01261793 (EMBODY2), NCT00424476 (BLISS-52), NCT00410384 (BLISS-76), NCT01196091 (ILLUMINATE-1) and NCT01205438 (ILLUMINATE-2)). A set of 290 constant and time varying covariates including demographic data, disease specific biomarkers, immunology variables, measurements from urinalysis, vital signs, variables from Electrocardiograms (ECG) signal, and hematological variables are collected up to 52 weeks. BILAG score across three main involved organs (mucocutaneous, musculoskeletal and renal) were used to derive the SLE flare status defined as moderate, severe and no flare across all organs [2]. We set up a supervised binary classification ML pipeline to classify those patients experiencing moderate or severe flares from those patients who never experienced any flare over the course of the study. Covariate engineering was performed considering the baseline value, statistical features and longitudinal metrics derived from feature engineering techniques. Three tree-based ML models (Xgboost [3], Random Forests [4] and Gradient Boosting Trees [5]) have been implemented for model training with hyperparameter tuning specific to each model resulting into a set of over 100 experiments per each model. The hidden test set was derived after randomly splitting the data into 80% of the training set and 20% for the hidden test making sure the same stratification factor of positive versus negative class in the training set is preserved. Results: A total number of 455 patients considered as positive class (those who experienced moderate and severe flare) and 1478 patients were considered as negative class (those who didn’t experience moderate or severe flares). The best model performance was achieved considering covariate values at the prior visit to the event along with the rate of their time-varying change derived as slope using the Xgboost model. The performance of this model resulted into 93% of precision, 98% of recall and 95% of F1-Score for prediction of flares, 93% of precision, 75% of recall and 83% of F1-Score for prediction of no flares, a weighted average of 93% of precision, 93% of recall and 92% of F1-score for prediction of both flares and no flares and an area-under-the-precision-recall curve (AUCPRC) of 0.91 on a hidden test set. As an external validation, leave-one-study-out cross validation technique was performed, resulting into performance metrics of an average of 0.76 of AUCPRC with the standard deviation of 0.19 across all 7 studies which shows the consistency of performance results across all studies. Conclusions: The ML based flare prediction algorithm achieved promising results in prediction of flares (AUCPRC of 0.91 with consistent performance across all 7 clinical trial data) including 1933 patients across 290 covariates. The ML flare detection pipeline could provide valuable insights for enhancing clinical trial design and interpretation. Additionally, such a model can assist physicians in predicting a patient’s likelihood of experiencing a flare.
[1] Ceccarelli F, Perricone C, Massaro L, Cipriano E, Alessandri C, Spinelli FR, Valesini G, Conti F. Assessment of disease activity in Systemic Lupus Erythematosus: Lights and shadows. Autoimmunity reviews. 2015 Jul 1;14(7):601-8. [2] Isenberg DA, Allen E, Farewell V, D’Cruz D, Alarcon GS, Aranow C, Bruce IN, Dooley MA, Fortin PR, Ginzler EM, Gladman DD. An assessment of disease flare in patients with systemic lupus erythematosus: a comparison of BILAG 2004 and the flare version of SELENA. Annals of the rheumatic diseases. 2011 Jan 1;70(1):54-9. [3] Chen T, Guestrin C. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining 2016 Aug 13 (pp. 785-794). [4] Cutler A, Cutler DR, Stevens JR. Random forests. Ensemble machine learning: Methods and applications. 2012:157-75. [5] Friedman JH. Greedy function approximation: a gradient boosting machine. Annals of statistics. 2001 Oct 1:1189-232.
Reference: PAGE 33 (2025) Abstr 11443 [www.page-meeting.org/?abstract=11443]
Poster: Methodology – AI/Machine Learning