Sebastiaan C. Goulooze (1), Elke H.J. Krekels (1), Erwin Ista (2), Monique van Dijk (2), Dick Tibboel (2), Thomas Hankemeier (1), Catherijne A.J. Knibbe (1,3)
(1) Department of Systems Biomedicine and Pharmacology, Leiden Academic Centre for Drug Research, Leiden University, Leiden, The Netherlands (2) Intensive Care and Department of Pediatric Surgery, Erasmus MC-Sophia Children’s Hospital, Rotterdam, The Netherlands (3) Department of Clinical Pharmacy, St. Antonius Hospital, Nieuwegein, The Netherlands.
Objectives:
Item Response Theory (IRT) modelling is a multivariate data analysis technique, which is increasingly used in pharmacometrics to analyse data from questionnaire-based clinical assessments [1]. Because IRT is essentially an unsupervised technique, the derived latent variable will not selectively quantify the condition of interest in all situations. For example, the clinical assessment of iatrogenic withdrawal syndrome (IWS) in children includes many items that are also affected by other conditions, such as disease, pain and undersedation. The latent variable obtained by IRT modelling of such data cannot be expected to selectively quantify IWS severity [2]. We therefore developed a supervised implementation of IRT [3] to analyse item-level data from IWS assessments in children [4].
Methods:
Data were obtained during the clinical validation of the Sophia Observation withdrawal Symptoms (SOS) scale, which is a 15-item scale to assess IWS in critically ill children [4]. In addition to SOS scores, the IWS severity was judged by trained nurses on a numerical (integer) rating scale called NRS withdrawal, which ranges from 0 (no IWS) to 10 (worst IWS possible). The NRS score represents the expert opinion, taking into account all contextual factors of the patient and was used as the gold standard during the validation of the SOS scale. With supervised IRT (sIRT), the item characteristic curves (ICCs) of each item were estimated by setting the latent variable to equal the NRS withdrawal score.
Subsequently, these ICCs were fixed and individual posthoc estimates of the latent variable were estimated from the item-level data in the absence of NRS score. We compared the ability to predict NRS withdrawal scores using either the posthoc estimated sIRT latent variable, the total SOS score, or the latent variable estimated with the unsupervised IRT model.
Results:
In the sIRT model, two-parameter item characteristic curves (ICCs) were used for 9 of the 15 items. For five items, a three-parameter ICC was used that adds parameter for the maximum probability of observing the symptom: 61.3% for Agitation, 46.7% for Inconsolable Crying, 19.9% for Grimacing, 68.2% for Sleeplessness, and 21.7% for Diarrhea. The probability of experiencing Sweating was best described with a five parameter biphasic ICC. The contribution of each item to the total test informativeness ranged from 1.5% for the Tremor and Diarrhea items to 17.1% for the Motor Disturbance item.
Local minima were often encountered when fitting an unsupervised IRT model to the withdrawal data, often with unrealistic parameter estimates as a result. The final unsupervised IRT model contained two-parameter ICCs for all items, as more complicated models failed to adequately converge. In linear models to predict the NRS score, the sIRT latent variable performed better than the total SOS score (AIC of 5636.4 versus 5789.8). The unsupervised latent variable performed worse as a predictor of the NRS score, with an AIC of 5792.0.
Conclusions:
As many of the behavioural and symptomatic items of paediatric questionnaire-based assessments can be caused by different conditions, an unsupervised IRT model of the item-level data of such assessments might inadequately quantify the condition of interest. In our example, we leveraged context-specific information in a sIRT approach to guide the estimated latent variable from the SOS scores towards IWS. Compared to the unsupervised IRT, the sIRT approach resulted in increased model stability and an improved prediction of iatrogenic withdrawal from SOS score data.
References:
[1] Ueckert S. Tutorial: Modeling composite assessment data using item response theory. CPT:PSP (2018). Epub ahead of print. doi: 10.1002/psp4.12280
[2] Harris J, Ramelet AS, van Dijk M, Pokorna P, Wielenga J, Tume L, Tibboel D, Ista E. Clinical recommendations for pain, sedation, withdrawal and delirium assessment in critically ill infants and children: an ESPNIC position statement for healthcare professionals. Intensive Care Med (2016) 42: 972-986.
[3] Idé T, Dhurandhar A. Supervised item response models for informative prediction. Knowl Inf Syst (2017) 51(1): 235-257.
[4] Ista E, de Hoog M, Tibboel D, Duivenvoorden HJ, van Dijk M. Psychometric evaluation of the Sophia Observation Withdrawal Symptoms Scale in Critically Ill Children. Pediatr Crit Care Med (2013) 14(8): 761-769.
Reference: PAGE 27 (2018) Abstr 8592 [www.page-meeting.org/?abstract=8592]
Poster: Methodology - Other topics