Elastic net regularization

Last updated

In statistics and, in particular, in the fitting of linear or logistic regression models, the elastic net is a regularized regression method that linearly combines the L1 and L2 penalties of the lasso and ridge methods. Nevertheless, elastic net regularization is typically more accurate than both methods with regard to reconstruction. [1]

Contents

Specification

The elastic net method overcomes the limitations of the LASSO (least absolute shrinkage and selection operator) method which uses a penalty function based on

Use of this penalty function has several limitations. [2] For example, in the "large p, small n" case (high-dimensional data with few examples), the LASSO selects at most n variables before it saturates. Also if there is a group of highly correlated variables, then the LASSO tends to select one variable from a group and ignore the others. To overcome these limitations, the elastic net adds a quadratic part () to the penalty, which when used alone is ridge regression (known also as Tikhonov regularization). The estimates from the elastic net method are defined by

The quadratic penalty term makes the loss function strongly convex, and it therefore has a unique minimum. The elastic net method includes the LASSO and ridge regression: in other words, each of them is a special case where or . Meanwhile, the naive version of elastic net method finds an estimator in a two-stage procedure : first for each fixed it finds the ridge regression coefficients, and then does a LASSO type shrinkage. This kind of estimation incurs a double amount of shrinkage, which leads to increased bias and poor predictions. To improve the prediction performance, sometimes the coefficients of the naive version of elastic net is rescaled by multiplying the estimated coefficients by . [2]

Examples of where the elastic net method has been applied are:

Reduction to support vector machine

It was proven in 2014 that the elastic net can be reduced to the linear support vector machine. [7] A similar reduction was previously proven for the LASSO in 2014. [8] The authors showed that for every instance of the elastic net, an artificial binary classification problem can be constructed such that the hyper-plane solution of a linear support vector machine (SVM) is identical to the solution (after re-scaling). The reduction immediately enables the use of highly optimized SVM solvers for elastic net problems. It also enables the use of GPU acceleration, which is often already used for large-scale SVM solvers. [9] The reduction is a simple transformation of the original data and regularization constants

into new artificial data instances and a regularization constant that specify a binary classification problem and the SVM regularization constant

Here, consists of binary labels . When it is typically faster to solve the linear SVM in the primal, whereas otherwise the dual formulation is faster. Some authors have referred to the transformation as Support Vector Elastic Net (SVEN), and provided the following MATLAB pseudo-code:

functionβ=SVEN(X, y, t, λ2);[n,p]=size(X);X2=[bsxfun(@minus,X,y./t);bsxfun(@plus,X,y./t)];Y2=[ones(p,1);-ones(p,1)];if2p>nthenw=SVMPrimal(X2,Y2,C=1/(2*λ2));α=C*max(1-Y2.*(X2*w),0);elseα=SVMDual(X2,Y2,C=1/(2*λ2));endifβ=t*(α(1:p)-α(p+1:2p))/sum(α);

Software

References

  1. Huang, Yunfei.; et al. (2019). "Traction force microscopy with optimized regularization and automated Bayesian parameter selection for comparing cells". Scientific Reports. 9 (1): 537. arXiv: 1810.05848 . Bibcode:2019NatSR...9..539H. doi: 10.1038/s41598-018-36896-x . PMC   6345967 . PMID   30679578.
  2. 1 2 Zou, Hui; Hastie, Trevor (2005). "Regularization and Variable Selection via the Elastic Net". Journal of the Royal Statistical Society, Series B. 67 (2): 301–320. CiteSeerX   10.1.1.124.4696 . doi:10.1111/j.1467-9868.2005.00503.x. S2CID   122419596.
  3. Wang, Li; Zhu, Ji; Zou, Hui (2006). "The doubly regularized support vector machine" (PDF). Statistica Sinica. 16: 589–615.
  4. Liu, Meizhu; Vemuri, Baba (2012). "A robust and efficient doubly regularized metric learning approach". Proceedings of the 12th European Conference on Computer Vision. Lecture Notes in Computer Science. Vol. Part IV. pp. 646–659. doi:10.1007/978-3-642-33765-9_46. ISBN   978-3-642-33764-2. PMC   3761969 . PMID   24013160.
  5. Shen, Weiwei; Wang, Jun; Ma, Shiqian (2014). "Doubly Regularized Portfolio with Risk Minimization". Proceedings of the AAAI Conference on Artificial Intelligence. 28: 1286–1292. doi: 10.1609/aaai.v28i1.8906 . S2CID   11017740.
  6. Milanez-Almeida, Pedro; Martins, Andrew J.; Germain, Ronald N.; Tsang, John S. (2020-02-10). "Cancer prognosis with shallow tumor RNA sequencing" . Nature Medicine. 26 (2): 188–192. doi:10.1038/s41591-019-0729-3. ISSN   1546-170X. PMID   32042193. S2CID   211074147.
  7. Zhou, Quan; Chen, Wenlin; Song, Shiji; Gardner, Jacob; Weinberger, Kilian; Chen, Yixin. A Reduction of the Elastic Net to Support Vector Machines with an Application to GPU Computing. Association for the Advancement of Artificial Intelligence.
  8. Jaggi, Martin (2014). Suykens, Johan; Signoretto, Marco; Argyriou, Andreas (eds.). An Equivalence between the Lasso and Support Vector Machines. Chapman and Hall/CRC. arXiv: 1303.1152 .
  9. "GTSVM". uchicago.edu.
  10. Friedman, Jerome; Trevor Hastie; Rob Tibshirani (2010). "Regularization Paths for Generalized Linear Models via Coordinate Descent". Journal of Statistical Software. 33 (1): 1–22. doi: 10.18637/jss.v033.i01 . PMC   2929880 . PMID   20808728.
  11. "CRAN - Package glmnet". r-project.org. 22 August 2023.
  12. Waldron, L.; Pintilie, M.; Tsao, M. -S.; Shepherd, F. A.; Huttenhower, C.; Jurisica, I. (2011). "Optimized application of penalized regression methods to diverse genomic data". Bioinformatics. 27 (24): 3399–3406. doi:10.1093/bioinformatics/btr591. PMC   3232376 . PMID   22156367.
  13. "CRAN - Package pensim". r-project.org. 9 December 2022.
  14. "mlcircus / SVEN — Bitbucket". bitbucket.org.
  15. Sjöstrand, Karl; Clemmensen, Line; Einarsson, Gudmundur; Larsen, Rasmus; Ersbøll, Bjarne (2 February 2016). "SpaSM: A Matlab Toolbox for Sparse Statistical Modeling" (PDF). Journal of Statistical Software.
  16. "pyspark.ml package — PySpark 1.6.1 documentation". spark.apache.org. Retrieved 2019-04-17.
  17. "Proc Glmselect" . Retrieved 2019-05-09.
  18. "A Survey of Methods in Variable Selection and Penalized Regression" (PDF).

Further reading