Optimization model testing tooling: likes, dislikes, and wishes

Question

We’re [1] in the thick of building out a testing suite for decision models, starting with a focus on route optimization. There are several angles we care about [2] and are considering: unit testing, scenario testing, historical testing, shadow testing, switchback testing, acceptance testing, and benchmarking.

I’ve seen some discussions here about different aspects of testing [3, 4]. My general impression: cohesive testing still has a ways to go, but it adds value if you can find the time to set it up and maintain it (after all that other work you did you simply build and deploy your model — DoorDash has written about this well [5]).

I’m curious to know: What do you like or dislike about the testing frameworks/tooling available today? What testing capabilities do you wish you could have?

[1] We’re a decision optimization company called Nextmv (https://www.nextmv.io)

[2] https://www.nextmv.io/videos/tools-for-benchmarking-and-testing-in-the-optimization-space

[3] How to generate data for test?

[4] What are technologies or libraries which greatly improve the speed or ease of use for delivering of OR software?

[5] https://medium.com/@DoorDash/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash-f1d938ab7c2a

Answer 1

By testing of decision models I am assuming you are testing if the model has high fidelity to the mathematical formulation, sense of constraints and other properties and attributes associated with the objective and candidate solutions at the nodes. Mathematical formulation can be tested using MPS file, constraint sense and other attributes can be tested by querying different attributes and properties. These should be available on the reference manual just as gurobi's.
You can also perform benchmark tests on solver performance by comparing with other solvers. Algorithm tests may be done with as a unit test by inputting data sets that give known optimal, feasible, sub-optimal, infeasible/unbounded results with set tolerance and optimality gaps.
An important part of testing for solver is if the solver is able to identify and set the appropriate code based on solution status like optimal, infeasible, unbounded etc.
Finally what I'd call stress tests- testing the model with both small scale and large scale datasets.

Finally while not exactly on testing strategies Gurobi did organize a talk on grinch that can ruin your model. This may give you some insights on designing/sampling your test cases.