2
$\begingroup$

I am using Pyomo + Ipopt as solver to solve a NLP problem. The problem is not extremely complex in terms of dimensionality and number of constraints. However, I am required to solve several thousands of instance of this model with different input parameters. Currently I am using a for loop to loop over. Solving a single instance takes me ~ 1 second on 32 GB RAM Windows PC, Python 3.7.3. That means I have to wait for hours before I am able to retrieve full optimization results.

I profiled into where the time is getting consumed. The model building itself does not takes any time as such but its the model solving via Ipopt that is performance determining here.

I would like to know if someone has previously run into a similar problem and what are the common ways one can try to achieve some sort of performance gains.

EDIT. From a question asked elsewhere I found out that:

Pyomo default behavior is to write an *.nl file, then to call IPOPT to process that file and produce a *.sol file. Pyomo then parses back in the *.sol file.

From this, it seems that achieving speed gains might not be that easy with Pyomo. Still looking forward for the community response.

Improve this question
$\endgroup$
3
  • $\begingroup$ Can you not run some batches in parallel and merge the results afterward? Simply launching a few Python scripts in parallel would be a simple exercise but would give some feedback about timings.. $\endgroup$
    – Erwin Kalvelagen
    Oct 26, 2020 at 15:08
  • $\begingroup$ @ErwinKalvelagen That is what I am doing for now. I will keenly wait for your feedback on timings. $\endgroup$
    – chupa_kabra
    Oct 26, 2020 at 17:32
  • 1
    $\begingroup$ Can you recast your problem using JuMP - I've used this successfully for some optimization problems where, similarly to you, I needed to reduce Python overhead and use parallelism. $\endgroup$
    – Richard
    Nov 2, 2020 at 18:59

2 Answers 2

Reset to default
3
$\begingroup$

Mac probably has a solid state drive, so if your Windows machine doesn't have one (or has a worse one), I/O speed could be your bottleneck.

For small runtimes like these, your version of Python & Pyomo can also be the culprit.

Νow when it comes to making the process faster, there's no general solution I'm afraid other than profiling the Python code and see if you can optimise the code on your end. What you want to see is how many seconds are spent in each Python routine (as opposed to percentages), so that you can derive where the bulk of your overall runtime is spent.

One thing you could try to speed things up would be to run multiple instances of these problems in parallel using the Python multiprocessing toolbox. Since the problems are similar you don't have to worry about writing asynchronous code to get good scaling, which makes things much easier.

Improve this answer
$\endgroup$
2
$\begingroup$

Disclaimer: This is not exactly an answer to this question but its a piece of information I found quite useful.

I ran the same piece of code on both Windows PC (32 GB RAM, Windows 10) and Macbook Pro (4GB RAM, Catalina) and found the code to be roughly 5 times faster on Mac. I have little idea why this is happening but it could be due to the pyomo file processing.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I was hearing from the software engineers that, Unix-based software have better than memory management than Windows platforms. $\endgroup$
    – A.Omidi
    Oct 28, 2020 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.