4
$\begingroup$

I have the following question that compares how "Sensitivity Analysis" is perceived in the field of Optimization compared to that of Machine Learning.

Sensitivity Analysis in Machine Learning:

At the core of every successful Machine Learning model is an underlying optimization problem that was successfully handled: For an individual problem (e.g. a computer playing chess), the final (data based) Machine Learning model that is actually used to make predictions (e.g. what piece to move next) in the real world, is "optimally fine tuned" using an optimization algorithm (e.g. Gradient Descent).

In Machine Learning, a well-documented concept exists called the "Bias-Variance Tradeoff", which brings up the idea that "a good solution to an optimization problem might not be as good in the future - thus, we might be able to "strategically" find a relatively worse solution that as a result might be perform better in the future". This process of "strategically finding a relatively worse solution to the optimization problem" is called Regularization , and is achieved by adding a "Norm Penalty Term" to the objective function (corresponding to the Machine Learning model) being optimized:

enter image description here

Doing this above process (i.e. Regularization) is said to make the solution (i.e. the Machine Learning model) "less sensitive" to deviations in future conditions compared to the initial conditions corresponding to the initial conditions in which the optimization problem was first solved (in this case, "high sensitivity" is associated with "high variance" : an ideal Machine Learning model has Low Bias and Low Variance).

My Question: In the field of Optimization, are any Statistical/Machine Learning based techniques (similar to Regularization) that are used to preemptively guard solutions from optimization problems against "high sensitivity" - and thus ensure that the optimization solutions are "robust" and successfully pass "sensitivity analysis" tests?

References:

Note: In Machine Learning, we almost never have a "deterministic objective function" and instead almost always have a "stochastic deterministic function" - that is, we have random measurements (e.g. these measurements might contain errors) corresponding to the function that we want to optimize (not the exact function). This means that there is an inherent stochastic/probabilistic component to Machine Learning optimization problems, and we are generally trying to optimize the "probabilistic expectation" of the objective function. Thus, Machine Learning optimization problems are always considered to be "noisy" and inherently like to display volatile behavior in future conditions - making the role of "sensitivity analysis" even more important in Machine Learning optimization problems.

$\endgroup$
2
  • $\begingroup$ It is pretty common to solve (discretized, linear) stochastic programs to guard against uncertainties in input parameters, e.g., with chance-constraints. Also, see (distributionally) robust optimization. The difference is that, usually, you must decide beforehand which uncertainties should be considered. Similar to including only a subset of weights in the regularization term. $\endgroup$
    – ktnr
    Jan 27 at 10:56
  • $\begingroup$ Look into Robust Optimization, including such things as (Wasserstein) (Data-Driven) Distributionally Robust Optimization. $\endgroup$ Jan 27 at 12:58

1 Answer 1

2
$\begingroup$

Machine Learning:

Firstly, sensitivity analysis on Machine Learning models goes way beyond the Bias-Variance tradeoff.

Sensitivity analysis of a Machine Learning model is done mainly to see how the changes in input affect the change in output. To do that, we may change one input while keeping the others constant. Generally, they are kept at the median for this analysis. This in turn gives a good idea about the uncertainty associated with the ML model and makes the models more interpretable which helps us in extracting 'Business values' from the model.

In real life, sensitivity analysis will help convey the model results to the stakeholders so they can focus on the main parameters of the model.

Optimization/Mathematical Modelling:

The theory of sensitivity analysis is well known. I am highlighting the 'robust' part here.

If you're forming a MILP or LP to solve an optimization problem then it will be robust 90% of the time (speaking from my experience). But when they are not, you can add several constraints (or may remove several depending upon the situation) to make them robust.

Most heuristics and meta-heuristics work by defining the 'neighborhood'. If you can define them in an elegant way, depending upon the problem, it will be robust.

$\endgroup$
1
  • $\begingroup$ What does the neighborhood deployed in a heuristic have to do with robustness? And what kind of constraints should be added to make a solution 'robust'? $\endgroup$
    – PeterD
    Jan 28 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.