**Group comparison with fused lasso penalized likelihood: an alternative to test based methods**

Ollier E. (1), Delavenne X. (1,2), Basset T. (1) and Samson A. (3)

(1) Laboratoire de Pharmacologie et Toxicologie, CHU Saint Etienne, Saint Etienne, France ;(2) Groupe de Recherche sur la Thrombose, EA3065, Université Jean Monnet, Saint-Etienne, France (3) Université Grenoble-Alpes, Laboratoire Jean Kuntzmann, UMR CNRS 5224, Grenoble, France

**Context: **Group comparaison methods for Non-linear mixed effects models (NLMEM) enable to identify parameters whose estimations significantly vary across the groups. They are classicaly based on statistical tests [1]. The main drawback of test-based methods could be their lack of sensitivity for studies with low statistical power. An alternative would be to choose the model that minimize a criteria like the Bayesian Information Criterion (BIC) [2]. The principal issue of this strategy is that the BIC of all the possible sub-models have to be evaluated. A solution is then to preselect only relevant sub-models with a penalized likelyhood method where the penalty enforces parameters to be equal across the groups. This is the case of the fused lasso penalty [3].

**Objectives: **Our objective was to develop a version of the SAEM algorithm that maximizes a likelihood penalized by a fused lasso penalty on both fixed effects and variance of random effects.** **

**Methods: **The fused lasso penalty was applied on fixed effects and random effect variances to the joint estimation problem’s likelihood which corresponds to the sum of the likelihoods of each group. Without the penalty term, maximizing this likelihood is equivalent to estimate a NLMEM independently in each group. The penalty strength is tuned by the sparsity parameter. When it is infinite, the penalty forces all the groups to have equal parameters (models are exactly the same in each group). A grid of possible values (user defined) for the sparsity parameter was tested and the optimal one is chosen using the BIC criterion.

We implemented a modified SAEM algorithm to solve the penalized likelihood problem. It follows the principle described by Bertrand et al [4].

The performance of the proposed method has been compared to a classical test based method on simulated data sets. Finaly we applied it to the analysis of a two way cross over trial studying the drug-durg interaction in 10 healthy subjects [5].

**Results/Conclusions: **The SAEM algorithm for fused lasso penalized maximum likelihood problems has been succefully implemented. Optimal model selection with the BIC criterion tend to be more powerful than the standard test based method on the simulated data sets, especially in design with small number of subjects. Results on real data were coherent with previously published results. The principal drawback of this method is that the grid of sparsity parameters values as to be user defined.

**References:**

[1] Samson, A., Lavielle, M., and Mentré, F. (2007). The SAEM algorithm for group comparison tests in longitudinal data analysis based on non-linear mixed-effects model . *Statistics in Medicine*, *26*(27), 4860–75.

[2] Delattre, M., Lavielle, M., and Poursat, M. A. (2012). BIC selection procedures in mixed effects models. http://hal.inria.fr/hal-00696435

[3] Gertheiss, J., and Tutz, G. (2012). Regularization and Model Selection with Categorial Predictors and Effect Modifiers in Generalized Linear Models. Statistica Sinica, 22, 957-982.

[4] Bertrand, J., and Balding, D. J. (2013) Multiple single nucleotide polymorphism analysis using penalized regression in nonlinear mixed-effect pharmacokinetic models. Pharmacogenetics and genomics, 23(3), 167-174.

[5] Delavenne, X., Ollier, E., Basset, T., Bertoletti, L., Accassat, S., Garcin, A., et al. (2013). A semi-mechanistic absorption model to evaluate drug-drug interaction with dabigatran: application with clarithromycin. British Journal of Clinical Pharmacology, 76(1), 107113. doi:10.1111/bcp.12055