Which ML algorithms work by solving constrained optimisation problems?

Question

As far as I know, most machine learning algorithms solve unconstrained optimisation problems, i.e., if we were to unroll all the neurons into symbolic expressions we would end up with a massive objective function and no constraints.

I read somewhere that SVM algorithms solve constrained problems instead, so I am wondering:

which machine learning algorithms actually solve constrained optimisation problems?

Just to clarify, my question is with respect to using constrained optimisation as part of the algorithm, not feeding constrained problems in.

Answer 1

As the borders between statistics and machine learning are diffuse, I take freedom to include statistics. Some links:

• Non-negative least squares, posts on cross validated. But often restrictions can be implemented just by reformulation.

• Order-restricted statistical inference should be better known and more used. There is a new R package being developed here.

• Shape-restricted splines, a very simple example.

• A little known problem: A paper on optimization under unknown constraints.

• A paper on Constrained Optimization Methods in Health Services Research.

• A paper showing how to parametrize covariance matrices (that is, positive definite (posdef) matrices. That is, a trick to avoid to use nonlinear restrictions. After all, generally the best method for restricted optimization is to transform to a nonrestricted problem ...