How to prove that optimizing each component of a system separately gives suboptimal result?

Question

The black-box system shown below has 3 components.

They run sequentially to generate a final output from the input. Each component has its own parameters to optimize for better intermediate results (Output #1, Output #2, etc.). We want to find the parameters that generate best final output. There are two methods to optimize this system

1. Optimize component #1 for best output #1, then component #2 for best output #2, and finally component #3.
2. Optimize all 3 components simultaneously for best final output, regardless of the intermediate results (Output #1, Output #2).

How to prove that method #2 is better than method #1 from the perspective of black-box optimization?

Is there any formal name for optimization methods like method #1? Method #1 looks like coordinate descent but it does not go back to already optimized components. It does not go back and optimize component #1 again after optimizing component #3.

Answer 1

Method #1 is called local optimization, while Method #2 is called global optimization.

In practice, all modelling is local. That is, we have to draw a boundary around the parts of the whole system that we consider in our model. If done well, then the parts of the system that we excluded do not significantly influence the solution. Often the decision about where to put the boundary is based on judgement rather than analysis.

As for your three-step process. Suppose that we have n resources that can be assigned to the components. A couple of potential solutions are:

• Local. Assign all n resources to Component 1. Let's assume that this maximizes throughput of Component 1, though we can't be sure. No resources remain to be assigned to Components 2 or 3, so zero Final Output is produced.
• Global. Assign the n resources across the components to maximize the Final Output. The assignment may or may not be uniform (we don't know what's best, as the components are black boxes).

The Global solution is clearly better than the Local solution.

More generally, there may be cases where the Local and Global solutions are equal. But there cannot be a case where the Local solution is better than the Global solution - otherwise that Local solution would also be the Global solution.

Answer 2

To add the answer by @solver max (I think it is a very good answer), Please be aware that, neither method-$1$ nor method-$2$ has a preference over another. Specifically, for the black-box optimization, as long as, you would not optimize each stage component, obviously, you would not achieve the global solution.

For example, suppose there is a complex system with three stages that produced the non-conformity parts. If in the beginning, you would try optimizing the whole system without considering the probability errors in each stage, it can happen to loos resource efficiency in some stage, as the system will try to optimize the final output.

I think this is one of the best applications of black-box optimization than other methods when one would like to survey a complex system.