Are Neural Networks Technically "Non-Linear Programs"?

Question

If you look at the Loss Function for a Neural Network:

Can we consider the above problem as a "Non-Linear Optimization Problem"?

The way I see it, the Loss Function of real world Neural Networks usually contains higher order terms that likely result in this function being Non-Linear. Seeing as the goal in Neural Networks is to effectively optimize this Non-Linear Loss Function, would it be technically incorrect to refer to this optimization problem as a "Non-Linear Optimization Problem" in the traditional sense?

For example, algorithms such as the "Interior Point Method" are often used to optimize typical "Non Linear Problems." Would the Loss Function of a Neural Network fall under a similar designation?

Answer 1

With the advent of Deep Learning, most modern machine learning is about minimising a nonlinear regression function. They even use similar techniques as we do in nonlinear optimisation, e.g., gradient descent.

However, there are a few practical differences in the approach:

• We typically (but not always) care about proving optimality, ML simply cares about finding a point in the neighbourhood of the minimum. This is because:
• The scale of computations is orders of magnitude different, and
• Proving optimality in ML is kind of pointless since it's a parameter fitting problem in the first place, i.e., it's not a first principles model. Since there is inherent noise in the model, the minimum itself has little physical meaning. All ML needs to achieve is to find a good enough fit.
• This is compounded by the fact that, unlike conventional optimisation, ML will keep changing the model dynamically until its predictive power is good enough when tested on a blind test set, which in classical nonlinear optimisation is something a human would do. In other words, while we iterate to optimise the model we have, ML iterates to find a good model.

Answer 2

I guess it really depends on what exactly are you asking. From a mathematical point of view, a neural network is nothing more than a function that maps inputs to outputs. Now, if you are thinking about the process of training a network--that is, given the network topology and a set of inputs and outputs, finding a set of parameters that minimize some function of the error between the neural network prediction and the observed output---the problem is indeed a 'traditional' mathematical program. Whenever your objective (loss) function as well as your constraints are most likely nonlinear, then your mathematical program is nonlinear.

Of course, this is not the only way of looking at the process of training your network, you may be interested in approaching the training from a bayesian perspective, then you can look at this process from a perspective other than optimization.

To summarize, the trained network is a function. The problem of training a network can be cast as a 'traditional' nonlinear optimization problem.

Answer 3

Maybe these will help : https://www.dynamic-ideas.com/books/machine-learning-under-a-modern-optimization-lens

https://arxiv.org/pdf/2112.09279.pdf (Robust optimization approach to Deep learning)

Actually, when the ML algorithm "train", it is equivalent to solving an optimization problem minimizing loss function. (decision variable : parameters of ML)

For example, ridge and lasso regression can be modeled as a SOCP (second-order cone program). Here, their objective function is quadratic + l1, l2 norm => convex optimization. Moreover, Logistic regression can be modeled as convex program with exponential cone..

But most of ML depends on first-order optimization method, because interior-point type algorithm requires us to compute inverse hessian of design matrix, which is costly..