How to reduce the risk of wrong modelling in OR industry projects?

Question

In my experience many OR Industry projects contains at least one person with background in OR (who is not familiar with the technical background) and at least one person from industry with the technical background, let's call them person1 and person2.

In meetings between these persons the initial vague problem definition becomes clearer and clearer. However there are many anecdotes that there are projects where person2 is missing to tell person1 some critical constraints, which have a huge impact on the model/ design of solving algorithms/results etc.

What are strategies/tools/things to do in order to minimize the risk of missing "important" characteristics about the optimization problem?

I thought as person1 about listing in a table all the assumptions that the models relies on and then explicitly discussing every assumption with person2 and also listing all effects which are left out because they "seem" not so important.

I think this is very similar to requirement engineering in software development, but I wonder if there are additional details if there is a mathematical model involved.

Answer 1

One of my favorite things about working in operations research is diving into a new problem to understand all the complexities. Getting a non-OR person to list all the requirements for an optimization problem is almost like asking someone to explain all the steps to ride a bike. They know how to do it, but won't be able to explain all the steps.

Here are some useful strategies to go about it:

Get out in the real world You need to understand the fundamentals of the system you are modeling. If you are working with a factory, you should visit the factory, watch how it works and how the planners do their job and ask all the silly questions. Where does the planning process start? Why did you decide to do A instead of B? Would C be a possibility? This will help you not just understand what the requirements are, but also why certain things are important and maybe even some things that aren't as important as someone told you.

Have a structured approach to uncover the full problem Identify ways you can split the problem and dive deep into the different components separately. For example, when formulating the objective function, you can identify all the stakeholders involved in the system and determine what their objectives are. They shouldn't all be included in the model, but it can help you ensure that you didn't miss anything important.

Gather real-world solutions Create some smaller datasets and discuss with the expert what the solution should be. They could also be real-life datasets you have collected. You can use them ongoingly to pressure test your solutions and see what the differences are between what business did and what your model proposes. Businesses often have an idea of how the ideal world should look like, but when reality hits, rules can be broken, and priorities can often shift. This can also help you discover if you are getting contradictory requirements.

Be agile A model should always be judged by the solution it produces and the value it creates. It is easy to get caught up in writing an elaborate document to describe all the aspects. You will never find the right model in your first attempt, so it is important to get ongoing feedback on the latest version of the model. Build a prototype that includes a way to visualize solutions produced by your model, so it is easy for users to engage and validate your results. A way to do this is to have some fixed interval where you demo the newest version and discuss priorities, e.g., every 2-4 weeks.

---

I always see it as a goal to get as deep into the problem as possible. The deeper you are, the more you will be able to ask the right questions, challenge their assumptions, and build credibility with the business to ultimately create some real-life impact with your model.

Answer 2

1. One way is to always start by modeling a baseline (a model which pictures the current situation). Basically, if you are working with linear programming (for example), you write your model and add constraints that model the existing situation. Then you check your results and analyse some KPIs and make sure they match the company's current KPIs. This will give you (and your interlocutors) some confidence in your model. As @Mark L Stone says in the comment section, this is an iterative process.

   For example, if you are working with flows, constrain your model such that your model outputs the exact flows of the current situation. Then make sure you have the same sites, the same volumes, the same costs, etc. Once you have a valid baseline, it becomes your reference model that you need to optimize. This way you compare comparable objects.

2. Also, a rule of thumb that is (almost) always true : if your model's results do not match roughly a result that you obtain with very basic reasoning and calculations (which the decision maker will do on a corner of a sheet of paper when you show him your results), then it is very likely that there is something wrong with your model.

Answer 3

Simulation experts have dealt with this question for years. If you do a search for "simulation validation" and/or "simulation face validation", you will find lots of hits. As I recall it, most of the suggestions for how to validate simulation models also apply (or generalize) to other types of OR models.

Answer 4

In my experience the best way to manage risk in these kinds of situations is to correctly manage client expectation and avoid 'big design up front'. You should expect the unexpected - i.e. new constraints to appear that were not discussed up-front. You should arrange the project in an iterative manner, so come up with a model, test in a real-world setting (ensuring all stakeholders understand this is a testing or tuning phase for the model, not the final result, and issues are expected), find out what's wrong with it, correct this, and then repeat this cycle as many times as needed. So basically follow something closer to an agile methodology and not a waterfall pattern. Big design up front doesn't work well for OR models, so avoid it.