Nikhil Pillai1, Panos Macheras2, 5,6, Sorell L. Schwartz3, Thang Ho4, Robert Bies1,5
1: Computational and Data Enabled Science, State University of New York (SUNY), Buffalo, USA 2: Department of Pharmacy National and Kapodistrian, University of Athens, Athens, Greece 3: Department of Pharmacology & Physiology, Georgetown University Medical Center 4: CytomX Therapeutics, San Francisco, USA 5: Department of Pharmaceutical Sciences, State University of New York (SUNY), Buffalo, USA 6: PharmaInformatics Unit, Research Center ATHENA, Athens, Greece
Objectives
The current scientific publishing landscape has been plagued with problems concerning reproducibility, recently discussed by Munafo et. al in [1]. A simplistic definition of reproducibility in this context would relate to the ability to duplicate a set of results, whether computational or experimental within a similar experimental environment by a different individual at another site. Specifically focusing on pharmacometrics and in particular quantitative systems pharmacology modeling, the issue relates to a failure to identify and differentiate between stochastic and deterministic systems, consequently risking the replacement of causality with probability. This affects all aspects of the problem from the development of the model itself, to the interconnection of different models qualitatively and ultimately to the reproducibility of the outcomes of these models. This is demonstrated especially remarkably in chaotic non-linear dynamic models [2], where initial solutions and model behavior are intertwined. In the current study, we depict these concepts within reproducibility using specific chaotic non-linear dynamic models. We perform simulation studies and statistical tests to demonstrate significant variability in final states (tumor burden) due to slight changes in critical controlling parameters.
Methods
We examined three models of tumor immune interactions:
- The Kuznetsov’s model (a mathematical representation of cytotoxic T lymphocyte response to the growth of immunogenic tumor) [3],
- The Kirschner-Panetta [4] model (model which explores the role of cytokines in the disease dynamics as well as addresses the topics of long-term tumor recurrence and short-term tumor oscillations) and
- The Mehmet model (which models the interactive nature between tumor cells, healthy tissue and activated immune system cells) [5].
Two analyses were performed to demonstrate the issue of irreproducibility of results due to the inherent nature of the model system. In the first analysis, for each model, two virtual samples (for number of patients, n=200 and 1000) of tumor burden were generated of different patients. The tumor burden was in turn calculated by creating a virtual sample of parameter values such that mean of this sample is equal to the nominal value provided in [3-5] and %CV is equal to 30%. The two samples only differed in the value of seed number. These two samples were then compared using Wilcoxon rank sum test. In the second analysis, for all three models, the first analysis (for n = 200) was repeated 100 times and the number of times the calculated p-value was below the level of significance was noted.
Results
We compared the tumor burden of two samples generated. Since the sample of tumor burden produced by the parameters generated from a normal distribution were not normally distributed the Wilcoxon rank sum test was used to compare the two. A p value of 0.0039, 0.011 and 0.0220 was obtained for Kuznetsov, Kirschner Panetta and Mehmet model respectively, for an unpaired two-sided test for the null hypothesis that the difference between the two samples arises from a distribution with zero median against the alternate that the difference arises from a distribution with a non-zero median. Based on the result of the test, at the default 5% level of significance the p value indicates that the test rejects null hypothesis, indicating that the two samples might be significantly different.
Conclusions
Multiple pharmacodynamic systems exhibit chaotic behavior (cardiovascular, CNS, metabolic). Hence, we should exercise caution in analytical tools and summary statistics when handling these kind of systems since they may be prone to irreproducibility due to inherent nature of the model system. If a nonlinear dynamic model exhibits chaotic behavior, even for a densely sampled data a small change in parameter values or initial cell count can cause significant change; this leads to failure of experiment or irreproducibility. Many irreproducibility problems are encountered in various disciplines e.g. social psychology; it is advisable to consider if nonlinear dynamical analysis can provide a plausible interpretation.
References:
[1]. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, Simonsohn U, Wagenmakers E-J, Ware JJ, Ioannidis JPA (2017) A manifesto for reproducible science. Nature Human Behaviour 1:0021. doi:10.1038/s41562-016-0021
[2]. Macheras P, Iliadis A (2016) Modeling in biopharmaceutics, pharmacokinetics and pharmacodynamics: homogeneous and heterogeneous approaches. Chapter 3. 2nd edition
[3]. Kuznetsov VA, Makalkin IA, Taylor MA, Perelson AS (1994) Nonlinear dynamics of immunogenic tumors: Parameter estimation and global bifurcation analysis. Bulletin of Mathematical Biology 56 (2):295-321. doi:10.1007/bf02460644
[4]. Kirschner D, Panetta JC (1998) Modeling immunotherapy of the tumor-immune interaction. Journal of mathematical biology 37 (3):235-252
[5]. Itik M, Banks SP (2010) Chaos in a Three-Dimensional Cancer Model. International Journal of Bifurcation and Chaos 20 (01):71-79. doi:10.1142/s0218127410025417
Reference: PAGE 28 (2019) Abstr 9006 [www.page-meeting.org/?abstract=9006]
Poster: Methodology - Other topics