PSO. Hyperparameter Optimization for Similar Problems

Question

I am doing a project where I am using Particle Swarm Optimization (PSO) for a design optimization problem. Naturally, in there I had to conduct hyperparameter optimization to find the most optimal values of the hyperparameters. I was asked a question by someone that whether the same set of optimal hyperparameters can still work if I increase the system size (i.e. increase search dimensionality) or do I need to do a fresh optimal hyperparameter search.

My response was, ideally yes, we would want to do a separate hyperparameter optimization since the problem changes the objective function landscape. I am looking forwards to your answers. Am I correct in saying this? If not, then what could be a more better response ?

Answer 1

In general you are correct, but the extent to which you are depends heavily on your objective function (and set of constraints, if you have any).

As you say, when the problem changes, the landscape changes, and you will need a different set of hyperparameters. However, small changes, or changes limited to certain features, may alter the landscape in a way that does not require a different algorithm configuration. Starting the new tuning from the original good configuration may help.

What you can do is to find the best configuration for some cases, and observe which hyperparameters change (and how) in relation to the features of the problem you are changing, and which don't. This may at least give you a better understanding of the landscapes generated your problem. For example, you may end up redefining certain hyperparameters as a function of the dimensionality, or of some landscape statistics.

Whether this is worth the effort, it depends on how many times you will have to run your design under different conditions.