12
$\begingroup$

In numerical tests of optimization algorithms, one often reads "We used a XY computer with X GB RAM for the experiments".

Usually, when I want to compare my results with theirs, I do not have an XY computer, but a YZ computer. And rerunning the tests of others is often difficult because I do not have the code.

I wonder whether one of the large cloud providers allows you to configure a virtual server in a way that others could easily do the same.

$\endgroup$
8
$\begingroup$

If the purpose of the comparison is to compare solution times (as opposed to validating the answers obtained), my feeling is that the XY v. YZ computer should not be an issue. Getting the source code (or binary) for the other algorithm is definitely a problem. Assuming you have their code (and that it runs on your computer, which may require getting the source and recompiling), and assuming you have enough memory for both methods, I think you can do a plausible comparison. If one algorithm has more stringent hardware requirements than the other (e.g., they need 16GB of RAM to solve problems you can solve in 4GB), that is worth noting. Memory aside, the primary difference between an XY and a YZ computer will be CPU speed and number of simultaneous threads it can handle (which relates to number of cores). Neither should dictate whether an algorithm can or cannot solve a problem (whereas memory might); they just affect solution speed. So running both methods on your computer v. theirs is like running a race on one track v. a different track: as long as both competitors are on the same track, it's a fair race.

For me, a trickier issue in algorithm races lies in comparability of the coding. If my code is in Python and theirs is compiled C code, they may have an inherent speed advantage unrelated to the wonderfulness of their algorithm. Even with the same language, differences in compilers could account for some speed differences. Similarly, if one method is coded by someone who is adept at optimizing for speed and the other is coded by someone who knows the math well but is a dabbler in coding, the "tighter" code might win even if the other algorithm is inherently better.

To get at the cloud part, I'm not aware of a cloud platform that lets you choose between different CPUs. (I think some of them let you configure the amount of RAM and possibly the number of cores, but not Intel XXX chip v. AMD yyy chip, say.) On the other hand, I'm no expert on cloud platforms.

$\endgroup$
  • $\begingroup$ I am not sure about the comment on competitors on the same track. All it takes is to hit the wrong size of a CPU's L1 cache to cause incredible slow-downs (or speed-ups) for one particular code but not for another. $\endgroup$ – Alberto Santini Aug 11 at 15:02
  • $\begingroup$ aws.amazon.com/ec2/instance-types has some info on the used cpus - sometimes it seems that they use just one specific cpu for certain instance types. $\endgroup$ – JakobS Aug 12 at 8:04
3
$\begingroup$

I wonder whether one of the large cloud providers allows you to configure a virtual server in a way that others could easily do the same.

$\endgroup$
  • $\begingroup$ I think what Fabian is aiming at is the possibility to have similar hardware which - to the best of my knowledge - you have no influence over for the computing platforms you mention. I'm not even sure that they all tell you their specific hardware setup. $\endgroup$ – JakobS Aug 12 at 7:47
  • $\begingroup$ @JakobS - One interpretation is that his computer isn't capable and that better capabilities are desired, another is that he desires to use the same computer. In either case I've offered an answer. The first paragraph of my answer addresses your last point and to your penultimate one that is addressed by two of my suggestions. $\endgroup$ – Rob Aug 12 at 7:59
  • $\begingroup$ jep, you are right. I missed that last point in your answer. The service looks interesting and it seems that they have a specified setup of hardware they are using for benchmarking. $\endgroup$ – JakobS Aug 12 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.