Implementation gap in logistics

Question

Typically OR-projects steps are: observing a situation, modeling it, solving the model, implementing the solution, and evaluating the situation.

In the area of production and logistics, the models are typically NP-hard combinatorial optimization problems. Typically, the contribution of these papers is to determine the complexity status of the problem and give a solution method with a small optimality gap. Some give some managerial insights based on the model. The focus is on the optimization and the steps that come after modeling and before the implementation of the solution.

But any optimization model also has a predictive element to it. (The prediction is if you take this solution, the objective value will be X and the solution will be implementable.)

Are there any studies on the predictive element of OR models? I would be especially interested in studies of the "implementation gap", i.e. the difference in cost predicted by the model and the real cost (or travel distance or ...).

I could so far not find any such studies, but I think such a study would be highly relevant:

The question of the gap between practice and theory is not new: e.g. Addressing the gap in scheduling research: a review of optimization and heuristic methods in production scheduling ask it in 1993.

There still seems to be a large gap with many models: e.g. for cross-docking, this study Cross-docking operations: Current research versus industry practice finds (in 2015):

But the cross-docking optimization literature evoked earlier seems somewhat disconnected from the actual industrial implementations of cross-docking".

There would probably be a considerable implementation gap if the model is "disconnected from the actual industrial implementations of cross-docking".

Assuming that literature reviews are somewhat representative of literature, we can look at: A ‘‘Meta survey’’ analysis in Operations Research and Management Science: A survey of literature reviews and find that "there is a fair concentration of topical coverage in the broad field of OR/MS, focusing on supply chain and logistics, sustainability, and scheduling". This therefore doesn't seem to be a small niche.

Obviously there is research in the more mathematical sense on strongly abstracted problems like the TSP, CVRP or the RCPSP. I also understand, that studying special cases, can lead to insights that might later be abstracted. However, most extensions of these models typically come with the claim, that it is "relevant for practice", yet I've never seen any inclusion of an "implementation gap".

(I'm aware of the informs journal of applied analytics, which does publish case studies, but even there I couldn't find anything on an "implementation gap").

My question: Is there any research on this "implementation gap". Is there maybe a different name/search term for it?

And why is it not always included anyway; some information on the quality of the model seem almost indispensable to me?

Answer 1

The implementation gap may not be a function of the "quality of the model" (or the algorithm used to solve it, which is a separate dimension). I had an experience with a logistics problem where the model was a simple shortest-path network model and the solution method was Dijkstra's shortest path algorithm. Both are time tested and thoroughly accepted. The implementation never happened, in part because the initial test was a dud. Why? The data used to parameterize the model, taken from a corporate database, was garbage. So there was a large implementation gap, but it was unrelated to the model (or algorithm).

Answer 2

You asked two questions at the end and I only try to answer the second "why is it [implementation gap] not always included anyway":

If I understand your question correctly, I'd say there can be many reasons why such an "implementation gap" that you refer to doesn't exist or reported. For example:

• The academic studies are not necessarily implemented. Why? It can be due to the simplicity of the model's assumptions, the specificity of the assumptions (e.g. your study is on VRP with backhauls with no time windows but I have a case for simultaneous pickups and delivery with time windows), the complexity of the solution or implementation cost of the solution (e.g. Netflix never used its 1 million dollar code!), or that nobody knows about that study (among many other reasons).
• Today's problems can be very different from tomorrow's. You may do research on a specific case study (thus, no such implementation gap) but by introducing even the smallest tweaks or additional constraints, the algorithms you have (and thus your solutions) don't work anymore (For example, you implement a great solution for a VRP where you pick from one central depot and you can deliver to a maximum of 4 warehouses and all of a sudden, I ask you for a small tweak of picking from more than one depot, i.e. multiple picks.) So, how do you measure the implementation gap here?
  • Bear in mind that I don't mean simple models are useless or "just for insight". On the contrary, they can serve as the minimum viable product (MVP) and that may do the job. Maybe you only need a feasible solution, something you don't currently have, and a model with simple assumptions provide that for you.
• You also mentioned that this implementation gap is like the "predictive element" in an optimization model. But for a prediction, you have actual and predicted values. How do you know the actual values? Do you consider the status quo the actual (Whether manual solutions, the solutions from another tool, or solutions that are far from being optimal)? As I mentioned above, we can't expect that even the most relevant studies are implemented, so that to have a sense of the actual vs predicted values. So, when studies perform tests against the standard library of datasets and report their results, that, IMO, is the best that they can do.

And finally, in journals such as IJAA that you mentioned where case studies are published, presumably, those implementations gaps should not exist as the study is already implemented!

Answer 3

In solving optimization problems, the gap is a relative concept. In the academic papers, the authors, have often used small instances to represent the model under study and usually, they could solve them optimality with a small gap.

Indeed, in the real-world problems (specifically, supply chain or scheduling) the problem is large, and the solution might be accepted with a relatively large gap. (separate from which kind of method, exact/heuristic, is used to solve the model.)

On the other hand, however, many practical works have not been published for commercial reasons that are hard to predict and compare the real and academic gap.

If you are interested, some useful references (real-world case study) could be found on the optimization software host like this or this.

Answer 4

It's unlikely that you will find a generalised study on this, as the predictive power of an optimisation model depends on the accuracy of the data we use to build the model, and on the skill of the modeler themselves.

This is also compounded by the fact that the predictive power of a model is also dependent on the solver we use, in particular on whether the solver is able to close the optimality gap and get us close to the global solution which, in many cases, is the only solution of the model which corresponds to reality.