10

We've written a benchmark toolkit that spits out a html with a number of statistics we find useful. Of these, I'll highlight the ones that I use regularly in bold. Over multiple datasets and multiple algorithms: Best Score Summary (Graph And Table) to see which algorithm works best Best Score Scalability Summary (Graph) Best Score Distribution Summary (...


8

Some further plots that can be helpful are bar charts with the type of solver going along the horizontal axis and the number of problem instances on the vertical axis. However, each bar is colour-coded with certain proportions that can indicate how "successful" a solver is compared to others. Kronqvist et al. (2019) provide several such charts in their ...


7

The problem is, even if you use the same number of threads, a 'rigorous' comparison of different implementations in different programming languages - for example, in terms of running time or of the quality of the solution found in the same time limit - can be quite challenging or tricky. Citing Chapter 18 of Ahuja, Magnanti & Orlin1: "The existing ...


6

There are a couple of situations where I would find box plots enlightening. For performance variability (either time to proven optimum or gap at a fixed time limit), box-and-whiskers plots with the performance measure on the vertical axis and the instance identifier on the horizontal axis (so one box per instance) make comparisons of instances easy. ...


6

Personal opinion: I think it is fair to use as many threads as makes sense in context. For instance, when I use CPLEX I believe it defaults to four threads (one per core on my PC). If I'm solving a problem that is a particular memory hog, I may throttle back to three or even two threads to avoid memory problems. I have no compunction about comparing any of ...


6

Short answer: you can't with any decent level of accuracy. The best you can do is ballpark comparisons. There are so many factors that affect the outcome, that you can maybe get within 20% difference on an identical machine with a decent degree of confidence. I know this for a fact because we benchmark on many identical machines and results always vary. ...


5

Some observations why min sum objectives are computationally more difficult than the makespan objective: For the decision problem, it can be shown that the total lateness objective is at least as general as the makespan objective: Assume we want to know if there exists a schedule with a makespan of at most $M$. When setting the due dates of all jobs to $M$, ...


4

In support of other answers and suggestions that you just run the other algorithm on your hardware, I would argue that failing to match the published results exactly is not necessarily cause for concern. If the authors reran the same examples using their code on their hardware, there is a high likelihood that timings would be at least somewhat different, and ...


3

There are a number of possible explanations (not mutually exclusive). The larger model might have a tighter continuous relaxation. (You can test that by relaxing the integrality restrictions and solving both LPs.) Assuming you are using a solver that has a presolve stage, there may be something in the first model that allows the presolver to tighten things ...


Only top voted, non community-wiki answers of a minimum length are eligible