Mikael Sunnåker and Angelica Quartino
Clinical Pharmacology and Quantitative Pharmacology, Clinical Pharmacology and Safety Sciences, R&D, AstraZeneca, Gothenburg, Sweden
Objectives: Neural networks (NNs) constitute a class of artificial intelligence methods inspired by the human brain to teach computers to process data. NNs have recently had an enormous impact in a wide range of applications ranging from image analysis to weather forecasting. Generative pre-trained transformers (GPTs) constitute a particular type of NNs based on the self-attention mechanism, which relates positions of data sequences to compute sequence representations. GPTs have recently increased in popularity due to impactful implementations of chat bots, such as ChatGPT, but have also been explored for analysis of time series [1].
Although NNs have been used for PK/PD predictions (e.g., neuralODEs [2]), we are not aware of any direct applications of GPTs in this domain. The objectives of this work are to investigate the applicability of GPTs for (1) simulation of dose regimens over time, (2) simulation of dose-response relationships, and (3) to investigate what type of data is required for GPTs to perform well.
Methods: We implemented a GPT following the approach described in [3,4], which uses the open-source neural network library Keras (TensorFlow) in Python. The input architecture is defined such that vectors of the training data, including time after the first- and last dose and the dose amount, are used to train the GPT.
Our GPT-implementation is applied to data that has been simulated from a one compartment PK model connected to an indirect dose-response model [5]. The simulated dataset describes the PD response to four consecutive doses with a dosing interval of 24 time units and including the subsequent return to baseline period. A GPT architecture is identified by manual tuning of the number of network layers, which is sufficiently complex to the describe the data, but sparse enough to avoid the curse of dimensionality (using a cross-entropy loss function for 150 epochs). The performance of the GPT is then evaluated for its ability to predict unobserved time points as well as new dosing regimens (3 and 7 doses) by RSME (between average simulations and predictions) and visual inspection. We also investigate the performance of the GPT when the attention mechanism is removed.
Results: Our analysis reveals several important principles for how GPTs should be implemented for analyses of PD data, and the type of observations required in real-world applications. We first show that for sufficiently informative data, the implemented GPT can well describe the training dataset (4 doses), but also make realistic predictions of new dosing regimens (3 doses and 7 doses) over time including the return to baseline period. Given the non-biased nature of GPTs (no shape of the response is assumed), we demonstrate that pre-dose observations are not sufficient for realistic interpolations of the response. It is also shown that the self-attention mechanism contributes to the separation of temporal effects (e.g. resulting in improved predictions of the 7 doses regimen), which gives GPTs an advantage to alternative NN implementations for PD predictions.
Conclusions: We show that GPTs, when successfully implemented and trained on informative PD data, can be used for realistic simulations of PD time-courses and dose-response relationships. This work therefore constitutes a first step towards GPT-based NNs for PD predictions.
References:
[1] Wen, Qingsong, et al. “Transformers in time series: A survey.” arXiv preprint arXiv:2202.07125 (2022).
[2] Lu, James, et al. “Neural-ODE for pharmacokinetics modeling and its advantage to alternative machine learning models in predicting new dosing regimens.” Iscience 24.7 (2021).
[3] Vaswani et al., Attention is all you need, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA 2017
[4] https://keras.io/examples/timeseries/timeseries_classification_transformer/
[5] https://metrumrg.com/wp-content/uploads/2017/06/Baron_ACOP_2015.10.pdf
Reference: PAGE 32 (2024) Abstr 10828 [www.page-meeting.org/?abstract=10828]
Poster: Methodology – AI/Machine Learning