2
$\begingroup$

For a study, I need to make runs to evaluate and compare performance of various models. To evaluate them I need to obtain their running times. I wonder if there is a rule to when to start and end timing. For instance, for the problems I also generate data randomly. Should I start prior to that random data generation, should I start timing between the time after generating data and defining the model, or should I only time how fast is the .solve() command? Or any other time?

I am using CPLEX on Python 3.10. For timing, I use time.time().

$\endgroup$
1
  • 4
    $\begingroup$ I would not include the time to generate the data. If the time spent defining the model is significant, I would include it, but seperate it in the results (e.g., specify what % of total run time is spent defining the model). $\endgroup$
    – Kuifje
    Jan 24, 2023 at 11:38

2 Answers 2

3
$\begingroup$

I agree with the comment by @Kuifje: record the solve time as this is what you are concerned with.

However, if you are comparing to some other method (e.g. a smaller MIP with weaker bound or a tailored algorithm) then you might also want to record the “build time” as the total wait time for the user may depend on this as well (new big model solves fast but builds slow, and old small model solves slowly but builds fast, for instance).

Generally, I try to time way too many things when testing, because it might turn out to reveal some insights I hadn’t thought about a priori. Recently we found a bottleneck in a project to be the communication with cplex in a callback: retrieving solution information for variables took up 80% of the time in the separation routine.

$\endgroup$
1
$\begingroup$

In addition to the above answers I'd also look at both solver performance and model performance. Is the model itself taking time to build across other solvers. Timeit() will help. Here's some references on checking model performance. Model build time may depend upon both logic in constraints, model size, presolving if redundant variables present, data structures used etc
Then comes question of solver's performance. For that, solvers output log or runtime queries can provide useful info like optimality gap/MIP gap covered (difference between primal and dual values), nodes created, nodes and branches explored at different checkpoints. This page provides important references for Gurobi (the 3rd link) but you should get the idea.

$\endgroup$

This site is temporarily in read-only mode and not accepting new answers.