I have the following question that compares how "Sensitivity Analysis" is perceived in the field of Optimization compared to that of Machine Learning.
Sensitivity Analysis in Machine Learning:
At the core of every successful Machine Learning model is an underlying optimization problem that was successfully handled: For an individual problem (e.g. a computer playing chess), the final (data based) Machine Learning model that is actually used to make predictions (e.g. what piece to move next) in the real world, is "optimally fine tuned" using an optimization algorithm (e.g. Gradient Descent).
In Machine Learning, a well-documented concept exists called the "Bias-Variance Tradeoff", which brings up the idea that "a good solution to an optimization problem might not be as good in the future - thus, we might be able to "strategically" find a relatively worse solution that as a result might be perform better in the future". This process of "strategically finding a relatively worse solution to the optimization problem" is called Regularization , and is achieved by adding a "Norm Penalty Term" to the objective function (corresponding to the Machine Learning model) being optimized:
Doing this above process (i.e. Regularization) is said to make the solution (i.e. the Machine Learning model) "less sensitive" to deviations in future conditions compared to the initial conditions corresponding to the initial conditions in which the optimization problem was first solved (in this case, "high sensitivity" is associated with "high variance" : an ideal Machine Learning model has Low Bias and Low Variance).
My Question: In the field of Optimization, are any Statistical/Machine Learning based techniques (similar to Regularization) that are used to preemptively guard solutions from optimization problems against "high sensitivity" - and thus ensure that the optimization solutions are "robust" and successfully pass "sensitivity analysis" tests?
References:
- https://en.wikipedia.org/wiki/Regularization_(mathematics)
- https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff
Note: In Machine Learning, we almost never have a "deterministic objective function" and instead almost always have a "stochastic deterministic function" - that is, we have random measurements (e.g. these measurements might contain errors) corresponding to the function that we want to optimize (not the exact function). This means that there is an inherent stochastic/probabilistic component to Machine Learning optimization problems, and we are generally trying to optimize the "probabilistic expectation" of the objective function. Thus, Machine Learning optimization problems are always considered to be "noisy" and inherently like to display volatile behavior in future conditions - making the role of "sensitivity analysis" even more important in Machine Learning optimization problems.
