1
$\begingroup$

Can Saddle Points Provide "Better Solutions" to Machine Learning Models Compared to Local Minimums?

The solution to a Machine Learning model (i.e. the final model parameters) are selected by trying to optimize the Loss Function associated with that Machine Learning model. The "best" solution (i.e. "best" choice of model parameters) are those associated with the "global minimum" of this Loss Function (i.e. the smallest value of the Loss Function). Thus, "relatively better" solutions can be considered as solutions that are located closer to the "Global Minimum". Optimization Algorithms (e.g. Gradient Descent) try to search for the "Global Minimum" of the Loss Function by repeatedly searching in the direction of the derivatives corresponding to this Loss Function.

However, there are different obstacles than can occur during this search process. For instance:

  • The Optimization Algorithm can get stuck in a "Local Minimum," or
  • The Optimization Algorithm can get stuck in a "Saddle Point."

I have heard "Saddle Points" as being considered "worse" than "Local Minimums." This is because "Saddle Points" aren't actually a minimum of any sort, whereas "Local Minimums" are at least minimums at the local level. Thus, this would imply that model parameters chosen from a "Saddle Point" should be worse than model parameters chosen from a "Local Minimum". To further illustrate my question, I drew the following graph of a hypothetical Loss Function for some Machine Learning model:

enter image description here

In the above picture, we can see that Loss Function has a smaller loss at the "Saddle Point" compared to the loss at the "Local Minimum". Thus, in this case (assuming that we could not reach "P3") if we had to choose a selection of model parameters from "P2" ("Saddle Point") and "P1" ("Local Minimum"), it would clearly make more sense to pick model parameters from "P2".

My Question: In general, do we know if solutions corresponding to "Local Minimums" points on a Loss Function are considered to be "better" corresponding to "Saddle Points" (e.g. perhaps solutions from "Local Minimums" might be more "stable")? Or is this claim incorrect, and solutions corresponding to regions of the Loss Function with lower Loss values are generally "better" - regardless of whether they come from a "Saddle Point" or a "Local Minimum"?

$\endgroup$

1 Answer 1

2
$\begingroup$

In minimization a lower loss if preferable under all circumstances. Each saddle point has a local minima one could discover from it if one walked in the right direction. Whether an individual saddle point is better than an individual minima depends on their objective. A saddle point can be than some local minima but it can be never be better than "it's" local minima. A saddle point can never be better than all local minima and saddle points but a local minima can be better than all other local minima and all saddle points. In summary saddle points are usually disliked because their existence proves we missed a better solution.

$\endgroup$
4
  • $\begingroup$ Thank you for your answer! Just some questions: $\endgroup$
    – stats_noob
    Commented Feb 7, 2022 at 18:04
  • $\begingroup$ 1) How do we know that all saddle points have a corresponding local minimum? $\endgroup$
    – stats_noob
    Commented Feb 7, 2022 at 18:04
  • $\begingroup$ 2) do we actually have anyway of knowing whether the solution we have came from a saddle point or a local minimum? $\endgroup$
    – stats_noob
    Commented Feb 7, 2022 at 18:05
  • $\begingroup$ Saddle point&not already a local minima&differentiable function implies there is a point nearby which is lower than the saddle point, doing gradient descent from there leads to boundary of set (when there are constraints), another saddle point of lower objective or a local minima. For finding out whether you are in a saddle point (assuming second order derivatives exist) you can check whether the hessian is positive semi-definite. $\endgroup$ Commented Feb 7, 2022 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.