Skip to contents

Penalties

Patrick Breheny

Source: vignettes/web/penalties.rmd
penalties.rmd
Under construction

ncvreg fits models that fall into the penalized likelihood framework. Rather than estimating β\boldsymbol{\beta} by maximizing the likelihood, in this framework we estimate β\boldsymbol{\beta} by minimizing the objective function

Q(β∣X,y)=L(β∣X,y)+Pλ(β), Q(\boldsymbol{\beta}|\mathbf{X}, \mathbf{y}) = L(\boldsymbol{\beta}|\mathbf{X},\mathbf{y})+ P_\lambda(\boldsymbol{\beta}),

where L(β∣X,y)L(\boldsymbol{\beta}|\mathbf{X},\mathbf{y}) is the loss (deviance) and Pλ(β)P_\lambda(\boldsymbol{\beta}) is the penalty. This article describes the different penalties available in ncvreg; see models for more information on the different loss functions available. Throughout, linear regression and the Prostate data set is used

data(Prostate)
X <- Prostate$X
y <- Prostate$y

MCP

This is the default penalty in ncvreg.

EQUATION HERE.

PICTURE OF PENALTY.

The following figure illustrates the effect of changing γ\gamma:

par(mfrow=c(1,3))
fit <- ncvreg(X, y, gamma=1.5)
plot(fit, main=expression(paste(gamma,"=",1.5)))
fit <- ncvreg(X, y)
plot(fit, main=expression(paste(gamma,"=",3)))
fit <- ncvreg(X, y, gamma=10)
plot(fit, main=expression(paste(gamma,"=",10)))

At smaller γ\gamma values, the estimates transition rapidly from 0 to their unpenalized solutions; this transition happens more slowly and gradually at larger γ\gamma values. Note that one consquence of these rapid transitions at low γ\gamma values is that the solutions are less stable (the gray region depicting the region of the solution path that is not locally convex is larger).

SCAD

EQUATION HERE.

PICTURE OF PENALTY.

Lasso

EQUATION HERE.

PICTURE OF PENALTY.

par(mfrow=c(1,3))
fit <- ncvreg(X, y)
plot(fit, main="MCP")
fit <- ncvreg(X, y, penalty="SCAD")
plot(fit, main="SCAD")
fit <- ncvreg(X, y, penalty="lasso")
plot(fit, main="Lasso")

Elastic Net and MNet

EQUATION HERE.

par(mfrow=c(1,3))
fit <- ncvreg(X, y)
plot(fit, main=expression(paste(alpha,"=",1)))
fit <- ncvreg(X, y, alpha=0.5)
plot(fit, main=expression(paste(alpha,"=",0.5)))
fit <- ncvreg(X, y, alpha=0.1)
plot(fit, main=expression(paste(alpha,"=",0.1)))

MENTION ELASTIC NET