Runtimes for benchmarking

Question

For a study, I need to make runs to evaluate and compare performance of various models. To evaluate them I need to obtain their running times. I wonder if there is a rule to when to start and end timing. For instance, for the problems I also generate data randomly. Should I start prior to that random data generation, should I start timing between the time after generating data and defining the model, or should I only time how fast is the .solve() command? Or any other time?

I am using CPLEX on Python 3.10. For timing, I use time.time().

Answer 1

I agree with the comment by @Kuifje: record the solve time as this is what you are concerned with.

However, if you are comparing to some other method (e.g. a smaller MIP with weaker bound or a tailored algorithm) then you might also want to record the “build time” as the total wait time for the user may depend on this as well (new big model solves fast but builds slow, and old small model solves slowly but builds fast, for instance).

Generally, I try to time way too many things when testing, because it might turn out to reveal some insights I hadn’t thought about a priori. Recently we found a bottleneck in a project to be the communication with cplex in a callback: retrieving solution information for variables took up 80% of the time in the separation routine.

Answer 2

In addition to the above answers I'd also look at both solver performance and model performance. Is the model itself taking time to build across other solvers. Timeit() will help. Here's some references on checking model performance. Model build time may depend upon both logic in constraints, model size, presolving if redundant variables present, data structures used etc
Then comes question of solver's performance. For that, solvers output log or runtime queries can provide useful info like optimality gap/MIP gap covered (difference between primal and dual values), nodes created, nodes and branches explored at different checkpoints. This page provides important references for Gurobi (the 3rd link) but you should get the idea.