7
$\begingroup$

I am working with some computational experiments with an Integer Programming (IP) formulation over a well-known set of instances from the literature of my problem. And I would like to compare my formulation with another author formulation from the literature (making a benchmarking).

However, the machine configuration in which the author's formulation was executed is different from mine. So, I thought, if I can find the equivalent time from the author's machine to mine, then I can use this equivalent time on my computational experiments, e.g if one hour of execution on the author's machine is equivalent to two hours of execution on mine machine and the author's experiments were executed with one hour of the time limit, then I can run my experiments with a time limit of two hours and have more precise results to compare.

Hence, I would like to know if there is any way to calculate the equivalent time from two different machines (even if this time be approximated).

$\endgroup$
6
  • 2
    $\begingroup$ If the other formulation is easy to implement, wouldn’t it be safer to compare both formulations after running them on your computer ? (This is not an answer to your question, obviously) $\endgroup$
    – Kuifje
    Jul 5, 2020 at 18:53
  • $\begingroup$ Well, the other formulation is easy to implement (and I already implemented it by myself), however, I am not achieving the (exactly) same author results in my machine running with one hour of the time limit. So, if I could approximate the time limit (as described above) I would be able to know (approximately) the right time limit to run the experiments on my machine. $\endgroup$ Jul 6, 2020 at 0:47
  • 1
    $\begingroup$ I think with some cloud services like google cloud you can generate a virtual machine(VM) with exact specifications of your choice. In addition, Gurobi has support for cloud computing. It comes to my mind that it will be fair enough if you can solve your approach on a machine exactly same as the authors. (Idea has not tried, do research if you decide to use it) $\endgroup$ Jul 6, 2020 at 1:13
  • 2
    $\begingroup$ Can you ask the authors of the other work for their source code, if it is not already available? If they won't provide it, you've made a best effort implementation and should run with that for your timing results. The authors may well have made a mistake in their implementation and no one else will be able to replicate their results either. This seems far less sketchy to me then trying to somehow scale execution time. As always: if the code isn't publicly available, it's not science. $\endgroup$
    – Richard
    Jul 6, 2020 at 5:02
  • 1
    $\begingroup$ If you have already implemented the code yourself, then I think running both formulations on your computer is a more scientific approach. As shown by current events, something that is published is not necessarily true and should not be considered as such. No one will ever criticize you for trying to replicate results and not being able to match them. Even better : if your code is open source, perhaps the community will be able to point out a mistake, or to validate that your implementation is correct, making obsolete (or strengthening) the other publication. $\endgroup$
    – Kuifje
    Jul 6, 2020 at 7:41

2 Answers 2

6
$\begingroup$

Short answer: you can't with any decent level of accuracy. The best you can do is ballpark comparisons.

There are so many factors that affect the outcome, that you can maybe get within 20% difference on an identical machine with a decent degree of confidence. I know this for a fact because we benchmark on many identical machines and results always vary.

Important things under your control

  • The machine itself

  • The operating conditions

  • The version of the kernel

  • The power supply

  • To have identical OS & software on both machines (including running daemons/services)

  • The network cables (assuming that's important)

  • The version of the compiler & libraries. This is crucial if using open source software.

Important things you can't control

  • The machine's hardware. This made both lists to point out that unless you have the factory settings of the reference machine, it's hard to reproduce. What's the motherboard? What's the exact CPU? Does it use single or dual socket? What's the RAM frequency? Does is use 2x16 GB RAM or 4x8 GB?

  • The operating conditions. This also made both lists. Was the reference machine in a cool room, especially if Turbo Boost was activated on multi-hour runs? What's the level of dust inside the box? Was someone else logged in at the same time?

  • The speed of the network, unless it's private and wired.

  • The version of the kernel. Yes this also made both lists (I said this is hard right?). Unless you clone the hard drives, even slight differences in installed software can cause the same Linux kernel to work differently on identical machines, or even not at all on one of them. The best way to resolve this is to use Windows.

  • Different pieces of software. Keep in mind that if you are comparing your method to another author's using your own code, you are comparing algorithms and implementations at the same time, in which case your error bounds can be massive. OR code is extremely implementation sensitive.

$\endgroup$
1
5
$\begingroup$

In support of other answers and suggestions that you just run the other algorithm on your hardware, I would argue that failing to match the published results exactly is not necessarily cause for concern. If the authors reran the same examples using their code on their hardware, there is a high likelihood that timings would be at least somewhat different, and a distinct possibility that (if not run to proven optimality) the incumbent objective value and/or best bound might be different at the same time limit originally used. Unless you get a wildly different result from their algorithm (infeasible when they have a feasible solution, a better solution than their proven optimum, ...), I would vote for side-by-side comparison on your hardware.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.