This article presents the following formulation of the best subset selection problem
$$\min_{\|\beta\|_0\leq k}\frac{1}{2}\|y-X\beta\|^2_2$$
I wonder where the $1/2$ comes from. Help appreciated.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
The factor of $\frac{1}{2}$ is just for convenience, so that the Hessian of the objective does not have a factor of two.
The argmin (optimal value of $\beta$) is the same, whether or not the factor $\frac{1}{2}$ is included.
Note that rather than $\frac{1}{2}$, some "authors" use $\frac{1}{2n}$, This also does not affect the argmin.
answered Jan 15, 2020 at 17:45
-
4$\begingroup$ Well, the square on the norm is also optional in same sense as the 0.5. Needed if you want a QP though. $\endgroup$ Commented Jan 15, 2020 at 18:28
-
1$\begingroup$ @ ErlingMOSEK Yes that is true. And as a true conic man, I know you appreciate the potential advantages, including not squaring the condition number. Nevertheless, least squares, and to a large extent, nonlinear least squares, is the genesis for the 1/2. $\endgroup$ Commented Jan 15, 2020 at 18:49
-
$\begingroup$ Thanks @MarkL.Stone, could you please clarify in more details so that the Hessian of the objective does not have a factor of two (the rest is rather clear) $\endgroup$– k88074Commented Jan 16, 2020 at 8:28
-
$\begingroup$ @k88074 Without the factor of 1/2 the Hessian of the objective would be $2X^TX$. With the factor of 1/2, the Hessian is $X^TX$. $\endgroup$ Commented Jan 16, 2020 at 12:50
-
$\begingroup$ I the following reasoning correct? If want to minimize the objective (forget the constraint), I set the first derivative of the objective to zero, that is $-2Xy+2X^\top~X\beta=0$. Why does the $2X^\top~X$ bother us? We still get the same $\beta$ $\endgroup$– k88074Commented Jan 18, 2020 at 8:00