5
$\begingroup$

I have the following question on "Regularization vs. Constrained Optimization" :

In the context of statistical modelling, we are often taught about "Regularization" as a method of dealing with the "Bias-Variance Tradeoff" (i.e. stabilizing the inconsistent performance of complicated models). When a L1-Norm or L2-Norm Penalty Term is added to the estimation function (corresponding to the statistical model) being optimized, some of the model parameters will either "shrink" in size towards 0, thus producing a "sparser" model that is more likely to retain its "low bias" but possible reduce its "high variance":

enter image description here

I have often heard of functions containing these L1-Norm and L2-Norm "Penalty Terms" being referred to as "optimization constraints" (i.e. the "feasible region" from which valid choices of model parameters can belong to has now been "altered" due to these "norm penalty constraints"):

enter image description here

My Question: When we estimate some statistical model's parameters and the estimation equation contains some "regularization penalty term," would it be incorrect to refer to this as an example of "constrained optimization"?

Is regularized optimization in Machine Learning and Statistical Modelling fundamentally any different (with the exception of usually being more difficult and solved using approximate stochastic iterative methods) from Constrained Optimization in Linear Programming?

enter image description here

References:

$\endgroup$
2
  • 2
    $\begingroup$ I believe a multi-objective model is a more useful paradigm for regularization. $\endgroup$ Feb 7, 2022 at 16:34
  • $\begingroup$ The only thing i can tell you is that it is not a constraint because it doesn't satisfy the definition of a constraint "condition of an optimization problem that the solution must satisfy". Just go over the definitions. $\endgroup$ Feb 12, 2022 at 23:01

2 Answers 2

5
$\begingroup$

Yes it is incorrect to refer to a unconstrained optimization problem as a constrained optimization problem. The idea of putting constraints into the objective is a often used technique. Example one example is using Lagrange multipliers. However i wouldn't call an optimization problem which puts all constraints into the objective a constrained problem.

In addition i think that sparsity constraints is an odd way to put it. I see the tradeoff between performance and sparsity as a pareto front which arise in multi objective optimization.

The fundamental difference is the set of legal states for which the objective is defined, constraints reduce the size of that set while putting the constraint into the objective does not.

$\endgroup$
1
  • $\begingroup$ +1, a better terminology match for L1/L2 regularization in stats models would be "soft-constraints". (Although no doubt you could construct hard constraints that would have similar ends in terms of reducing variance for undetermined systems.) $\endgroup$
    – Andy W
    Feb 13, 2022 at 14:19
3
+25
$\begingroup$

I think regularization is a kind of constrained optimization problem. Of course, in LASSO they eliminate l-1 constraint term and get some equivalent Lagrangian objective formulation. But the nature of enforcing sparsity is putting some l-1, l-2 norm constraints (and move them up to objective term)

So in LASSO, concepts used in constrained optimization - (Fenchel) Duality, Proximal gradient, etc. are important. I highly recommend this amazing book.

(Statistical Learning with Sparsity) https://hastie.su.domains/StatLearnSparsity_files/SLS.pdf

For more optimization-oriented paper, this is great.

(Optimization with Sparsity-Inducing Penalties) https://arxiv.org/abs/1108.0775

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.