Regularization in Machine Learning : Constraint Optimization in Disguise?

Question

I have the following question on "Regularization vs. Constrained Optimization" :

In the context of statistical modelling, we are often taught about "Regularization" as a method of dealing with the "Bias-Variance Tradeoff" (i.e. stabilizing the inconsistent performance of complicated models). When a L1-Norm or L2-Norm Penalty Term is added to the estimation function (corresponding to the statistical model) being optimized, some of the model parameters will either "shrink" in size towards 0, thus producing a "sparser" model that is more likely to retain its "low bias" but possible reduce its "high variance":

I have often heard of functions containing these L1-Norm and L2-Norm "Penalty Terms" being referred to as "optimization constraints" (i.e. the "feasible region" from which valid choices of model parameters can belong to has now been "altered" due to these "norm penalty constraints"):

My Question: When we estimate some statistical model's parameters and the estimation equation contains some "regularization penalty term," would it be incorrect to refer to this as an example of "constrained optimization"?

Is regularized optimization in Machine Learning and Statistical Modelling fundamentally any different (with the exception of usually being more difficult and solved using approximate stochastic iterative methods) from Constrained Optimization in Linear Programming?

References:

• https://en.wikipedia.org/wiki/Regularization_(mathematics)
• http://ab-initio.mit.edu/wiki/index.php?title=NLopt_Tutorial

Answer 1

Yes it is incorrect to refer to a unconstrained optimization problem as a constrained optimization problem. The idea of putting constraints into the objective is a often used technique. Example one example is using Lagrange multipliers. However i wouldn't call an optimization problem which puts all constraints into the objective a constrained problem.

In addition i think that sparsity constraints is an odd way to put it. I see the tradeoff between performance and sparsity as a pareto front which arise in multi objective optimization.

The fundamental difference is the set of legal states for which the objective is defined, constraints reduce the size of that set while putting the constraint into the objective does not.

Answer 2

I think regularization is a kind of constrained optimization problem. Of course, in LASSO they eliminate l-1 constraint term and get some equivalent Lagrangian objective formulation. But the nature of enforcing sparsity is putting some l-1, l-2 norm constraints (and move them up to objective term)

So in LASSO, concepts used in constrained optimization - (Fenchel) Duality, Proximal gradient, etc. are important. I highly recommend this amazing book.

(Statistical Learning with Sparsity) https://hastie.su.domains/StatLearnSparsity_files/SLS.pdf

For more optimization-oriented paper, this is great.

(Optimization with Sparsity-Inducing Penalties) https://arxiv.org/abs/1108.0775