Are the KKT Conditions as Important in Optimization as they were Originally?

Question

It seems like when the KKT conditions were first developed, these were very useful for determining whether the solution to a optimization problem was optimal or not.

However, it seems like nowadays, we hear about the KKT Conditions less - for instance, when reading about all the cool things that Deep Neural Networks are doing (e.g. AlphaGO, Protein Folding, Self Driving Cars) , the KKT Conditions never seem to be mentioned that much.

I was wondering if someone could comment on the following:

With the advent of approximate optimization methods such as Gradient Descent that seem to work quite well for non-convex and high dimensional problems - do the KKT Conditions have as much importance in Optimization now as they did when they were first developed?

Answer 1

Much (most?) of Deep Neural Network training (optimization) has been unconstrained optimization. The KKT condition(s) for unconstrained optimization is that the gradient of the objective function is zero. That's it, that's the entirety of the KKT conditions for unconstrained optimization. That condition was known for centuries prior to formulation and naming of the KKT conditions. In the context of unconstrained optimization, mention might only be made of getting the gradient (close) to zero; and KKT conditions never even mentioned as such.

Many (local) numerical nonlinear optimization algorithms essentially consist of numerically solving the KKT conditions, perhaps while "rolling downhill" (for minimization). So yes, KKT conditions are still quite important in optimization.

I believe there is an increasing trend toward constrained optimization for (Deep) Neural Networks. Hence I expect KKT conditions to be mentioned more frequently in the future in (Deep) Neural Network optimization