17

Let me add a couple of things based on my experience. I created/collated the DIMACS instances for clique and coloring way back in 1992 and they are still used extensively, primarily for graph coloring. I also have kept track of results for a sports scheduling problem, the Traveling Tournament Problem, at http://mat.tepper.cmu.edu/TOURN A couple things I ...


16

There are multiple aspects to this topic. What can be done by testset compilers to prevent finetuning? For one, testset creators are usually not including many very similar problems into the benchmark. For example, MIPLIB2017 was specifically compiled in a way to ensure diversity. The design paper for the CEC2018 benchmarks specifically states: The 15 ...


11

To complement Michael's answer, which I think made the main points already: First, an obvious strategy is to separate the data set used for testing and tuning, from the data set that are used in validation, which is a common practice in other areas (e.g. machine learning). Second, since one can be never sure to avoid a bias, I think it is helpful to state ...


8

If the purpose of the comparison is to compare solution times (as opposed to validating the answers obtained), my feeling is that the XY v. YZ computer should not be an issue. Getting the source code (or binary) for the other algorithm is definitely a problem. Assuming you have their code (and that it runs on your computer, which may require getting the ...


4

I wonder whether one of the large cloud providers allows you to configure a virtual server in a way that others could easily do the same. The NEOS Server is a free internet-based service for solving numerical optimization problems. Hosted by the Wisconsin Institute for Discovery at the University of Wisconsin in Madison, the NEOS Server provides access to ...


Only top voted, non community-wiki answers of a minimum length are eligible