3
$\begingroup$

It seems like when the KKT conditions were first developed, these were very useful for determining whether the solution to a optimization problem was optimal or not.

However, it seems like nowadays, we hear about the KKT Conditions less - for instance, when reading about all the cool things that Deep Neural Networks are doing (e.g. AlphaGO, Protein Folding, Self Driving Cars) , the KKT Conditions never seem to be mentioned that much.

I was wondering if someone could comment on the following:

With the advent of approximate optimization methods such as Gradient Descent that seem to work quite well for non-convex and high dimensional problems - do the KKT Conditions have as much importance in Optimization now as they did when they were first developed?

$\endgroup$
1
  • 2
    $\begingroup$ KKt applies to constrained optimisation. $\endgroup$
    – copper.hat
    Feb 6 at 21:28

1 Answer 1

5
$\begingroup$

Much (most?) of Deep Neural Network training (optimization) has been unconstrained optimization. The KKT condition(s) for unconstrained optimization is that the gradient of the objective function is zero. That's it, that's the entirety of the KKT conditions for unconstrained optimization. That condition was known for centuries prior to formulation and naming of the KKT conditions. In the context of unconstrained optimization, mention might only be made of getting the gradient (close) to zero; and KKT conditions never even mentioned as such.

Many (local) numerical nonlinear optimization algorithms essentially consist of numerically solving the KKT conditions, perhaps while "rolling downhill" (for minimization). So yes, KKT conditions are still quite important in optimization.

I believe there is an increasing trend toward constrained optimization for (Deep) Neural Networks. Hence I expect KKT conditions to be mentioned more frequently in the future in (Deep) Neural Network optimization

$\endgroup$
1
  • 2
    $\begingroup$ Generally in deep learning, we don't care at all whether the solution is optimal as long as the resulting model has small enough prediction errors on validation data. Since the problem of minimizing the loss function is highly non-convex, the best that you could hope for would be a local minimum anyway. $\endgroup$ Feb 6 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.