# Optimal Methods and LBFGS

I'm working on my thesis in physics. A core part of it is optimizing shapes under electromagnetic (Maxwell's Equations) constraints. To do this, we explore the geometrical space (lets say we have 40-50 parameters) and utilize scipy minimizer to iterate towards the optimal shape. For each iteration, we run the simulation to find the figure of merit. This takes 30 seconds. To find the gradient for each parameter, we unfortunately have to do some annoying stuff that adds around 7 seconds per parameter. Now I am using LBFGS algorithm (second order descent algorithm), but I was thinking to reduce the time.

But I'm not sure about a few things:

• Is it possible to use a different method with LBFGS or any of the scipy minimize options
• And furthermore, what optimization method do you think is most suitable in general here?
• Is there any advice you may have to improve the overall problem solving method
• Can you clarify what you mean by "mini batch"?
– prubin
Apr 5 at 15:51
• Also, are objective function evaluations more or less expensive than derivative evaluations?
– prubin
Apr 5 at 17:50
• Oh I mean mini batch as in the gradient descent method where you only use a few parameters to descend rather than the entire batch. Apr 6 at 7:00
• The objective function evaluation is about 30-40 seconds. The derivatives take about 7 seconds per parameter, so for 50 parameters it is definitely more expensive to find the derivatives Apr 6 at 7:01
• I'm not positive -- use of the terminology does not seem uniformly consistent -- but I think that what "mini batch" gradient descent usually means (in the context of machine learning) calculating a descent step over subsamples (batches) of observations at each iteration rather than over the full data sample. I think what you mean is called "partial gradient descent" (descent using only a portion of the gradient vector), but I won't swear to it.
– prubin
Apr 6 at 16:33

Partial gradient descent (if that is the correct term) seems like a reasonable approach here. I'm not a Python user, so I can't speak to its availability in scipy. Given the comparatively cheap objective function evaluations, another possibility would be to start out using a gradient-free approach, such as Nelder-Mead or one of the variations of it. Once that method gets close to convergence (the size of the simplex gets small), you might switch to a gradient-based method to finish the job (or you might just stick with N-M).

• Hi, that seems like a very interesting approach! Definitely will check this out. And you are correct I was misusing the term minibatch. Apr 7 at 16:07

This kind of optimization problem, where you don't have a mathematical model of the objective function, is often called blackbox optimization. It is very common to optimize physical systems this way.

### Optimization method

Given that the function is expensive to compute, and that you have no guarantee that it is convex, LBFGS is not the best tool for this.

The methods to optimize these are complex and varied, so your best bet is to use an off-the-shelf solver. There are many, both open-source (OpenBox, ...) and proprietary (Hexaly, Knitro, ...).

You are spending a lot of time computing the gradient. In many situations, the gradient comes almost for free using automated differentiation. I can't tell if this is applicable here without knowing more about your problem.

However, many algorithms above don't require a gradient at all! Having the gradient helps, but if it is that costly to evaluate you might just skip it.

### Minibatch

To me, a minibatch is something associated to stochastic gradient descent and machine learning, and not applicable to LBFGS and deterministic optimization.

• Thank you very much! I'll have a look at the OpenBox method. The gradient costs are high because the shape that is generated needs to be put through a lithography model that requires exporting it and reimporting it. That takes about 7 seconds per parameter. Thats why I wanted to reduce the amount of parameters we are dealing with at a time. But I see why it doesn't apply here. The powerful thing about the current method is that the gradient map is actually generated very quickly (with two simulations.) But then we need to go param by param to see how the map related to the params. Apr 5 at 12:56