2
$\begingroup$

I am using Pyomo + Ipopt as solver to solve a NLP problem. The problem is not extremely complex in terms of dimensionality and number of constraints. However, I am required to solve several thousands of instance of this model with different input parameters. Currently I am using a for loop to loop over. Solving a single instance takes me ~ 1 second on 32 GB RAM Windows PC, Python 3.7.3. That means I have to wait for hours before I am able to retrieve full optimization results.

I profiled into where the time is getting consumed. The model building itself does not takes any time as such but its the model solving via Ipopt that is performance determining here.

I would like to know if someone has previously run into a similar problem and what are the common ways one can try to achieve some sort of performance gains.

EDIT. From a question asked elsewhere I found out that:

Pyomo default behavior is to write an *.nl file, then to call IPOPT to process that file and produce a *.sol file. Pyomo then parses back in the *.sol file.

From this, it seems that achieving speed gains might not be that easy with Pyomo. Still looking forward for the community response.

$\endgroup$
  • $\begingroup$ Can you not run some batches in parallel and merge the results afterward? Simply launching a few Python scripts in parallel would be a simple exercise but would give some feedback about timings.. $\endgroup$ – Erwin Kalvelagen Oct 26 '20 at 15:08
  • $\begingroup$ @ErwinKalvelagen That is what I am doing for now. I will keenly wait for your feedback on timings. $\endgroup$ – your_boy_gorja Oct 26 '20 at 17:32
  • 1
    $\begingroup$ Can you recast your problem using JuMP - I've used this successfully for some optimization problems where, similarly to you, I needed to reduce Python overhead and use parallelism. $\endgroup$ – Richard Nov 2 '20 at 18:59
3
$\begingroup$

Mac probably has a solid state drive, so if your Windows machine doesn't have one (or has a worse one), I/O speed could be your bottleneck.

For small runtimes like these, your version of Python & Pyomo can also be the culprit.

Νow when it comes to making the process faster, there's no general solution I'm afraid other than profiling the Python code and see if you can optimise the code on your end. What you want to see is how many seconds are spent in each Python routine (as opposed to percentages), so that you can derive where the bulk of your overall runtime is spent.

One thing you could try to speed things up would be to run multiple instances of these problems in parallel using the Python multiprocessing toolbox. Since the problems are similar you don't have to worry about writing asynchronous code to get good scaling, which makes things much easier.

$\endgroup$
2
$\begingroup$

Disclaimer: This is not exactly an answer to this question but its a piece of information I found quite useful.

I ran the same piece of code on both Windows PC (32 GB RAM, Windows 10) and Macbook Pro (4GB RAM, Catalina) and found the code to be roughly 5 times faster on Mac. I have little idea why this is happening but it could be due to the pyomo file processing.

$\endgroup$
  • 1
    $\begingroup$ I was hearing from the software engineers that, Unix-based software have better than memory management than Windows platforms. $\endgroup$ – A.Omidi Oct 28 '20 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.