What are the reasons OR industry projects fail?

Question

You probably have done some OR projects within the industry or - if you are in academia - for the industry. I'm wondering if you've seen some patterns of why such projects fail (or at least do not meet the clients requirements). Surely, there might be causes which are common to any other non-OR project but I'm specifically interested in ones that occur only within an OR context.

Answer 1

In addition to what others have shared, in my experience, the following can cause industry OR projects to fail or can at least cause big issues and/or delays:

1) Quickly changing problem specification. The problem specification often changes very rapidly (e.g. hard constraints become soft), as the client gets a better idea of what optimisation can offer. If you cannot keep up with the changes and develop a decision support system that really solves the client's project, the client might not accept it.

2) End-users' trust and acceptance. If the end-users do not accept or do not trust the decision support system that has been built, then they will not use it, which is one of the worst case scenarios. Two measures can help with this from my experience: involve the end-users as soon as possible, and give them the sense that they are still in charge of making the end-decisions, for instance by providing several solutions from which they can choose from.

3) Many soft constraints. In real-world applications there are often a lot of soft constraints and it can be very challenging to weight them according to the client's preferences (and for the client to find out their preferences!), and to design solving approaches that can solve such complex models.

4) Fear of being replaced. I've heard of cases where domain experts on the clients side would boycott projects, probably because they feared being replaced by the decision support system (often rightly so).

5) Domain language. Probably not project-critical, but it can be a source of misunderstandings if the domain experts and the mathematicians use different domain languages that use the same vocabulary. For instance, if a mathematician talks about a "constraint" then they might imply that it is hard, while for the domain expert it might also be soft.

6) "It's always been like that". With some clients you might come across business rules or process rules that have been installed by someone (who has often already left the company ages ago), and nobody remembers why it's done that way. It can be difficult to identify these cases, and debunk them, but if you do, you get a better idea of the underlying problem and can often provide better solutions.

7) "Unspeakable" constraints. There are some constraints that the client will not be able to give you. For instance, when designing a roster, there might be employees who do not want to work together for personal reasons, so you should not schedule them in the same shift. However, it is not ethical to put this kind of information, say, into a database, from which you then extract these constraints. In such cases, your problem model will not be able to be fully accurate, and will not produce the solution the client would actually prefer.

Answer 2

I can immediately think of two reasons, both of which occurred on a consulting project in which I participated.

1. Every project needs one or more "champions", people in the client organization with a vested interest in seeing the analysis completed and the results implemented. If the champions leave the organization or are transferred to some other part of the organization, the project may easily get shelved.
2. Garbage in, garbage out. If the organization is working with incorrect / corrupted data (and nobody realizes it, or nobody can fix it), the analysis is probably doomed.

One other possible reason (that I've heard mumbled about but not tripped over myself): management wants the projected implemented but the IT guys are opposed (possibly because it will create headaches, real or imagined, down the road when the OR people are gone and the IT people have to maintain it).

Answer 3

Last year I attended a presentation by Maria Antónia Carravilla where she gave several case studies of optimisation projects, examining the factors affecting their success or failure.

Her main point was that project failures often have nothing to do with the technical aspects of the work and everything to do with non-technical factors, in particular stakeholder relationship management. She identified six key dimensions of client involvement in an optimisation project (a couple of which have already been mentioned in other answers):

• Top management
• Middle management
• End users
• IT staff
• Having a team member within the client company
• Deliverables (I think this includes the actual optimisation model/software).

She gave a couple of examples of projects where the optimisation was great in silico but failed as a project because these other aspects weren't adequately handled:

• Consultants called in to improve decision process for a complex manufacturing business. One of the team members said that they could do much better than the company's production director, in his hearing, and he became hostile to the project.
• Consultants working on decision support for reservations in a car rental company. They produced a system that was considerably more efficient than the existing solution, but there was a lot of work involved in providing the necessary data inputs, and the IT department was't willing to support it.

She also pointed out that projects that only need to be run once have very different requirements to those which will become a regular part of the client's business; the latter require much more attention to ease of use, user training, etc. etc.

Answer 4

Adding to the other good answers here...

Failing to employ an iterative process
The idea that a few meetings to communicate and understand the problem, scope the OR solution, desired outputs, etc., will enable the OR magician to disappear only to return with the holy grail is a fantasy.

It requires iteration. Develop initial capability, evaluate, improve or add capability, evaluate, ..., continue.

If during each iterative meeting, the OR team plus stakeholders explicitly ask What are the 10 reasons this won't work (for the stakeholders)?, then fix those issues, the team can often achieve convergence as they employ this strategy. Often the issues the stakeholders raise with a preliminary solution are fixable.

Clearly a good process to start the project affects this, but that's already well-covered in the other answers.

Answer 5

This may or may not be considered as occurring only within an OR context, but not getting senior-enough leadership guidance and buy-in on the client side (whether internal or external) as to the need for and value of the project, resources required (including support from people outside the OR analysts) as well as key assumptions and ground rules.

Here is a specific example (don't blame me, I was not involved nor had any opportunity to be). An OR study was done for a large industrial company, which among other things, involved cost/benefit analysis for a project involving the potential for large capital expenditures. The OR group did their usual wonderful analysis job, dutifully modeling the situation, applying relevant OR techniques, etc, just as they had done for smaller projects. The OR team only interfaced with mid-level people on the client side. The OR team asked those mid-level people what interest rate to use in their study, and were provided an interest rate to use commensurate with the company's borrowing costs.

WHOOPS!! Major Fail!! The recommended capital expenditure was so large that it would have likely lowered the credit rating of the client company, thereby increasing its cost of borrowing, which therefore should have been accounted for in the analysis. When the analysis results, which didn't account for possible credit rating change, was briefed to the CEO of the client company, he gave them the heave-ho.

Answer 6

One obstacle in finding champions within the client company is that with OR solutions, there is a fear of automating decisions, which might make some people's job redundant. Or at least this impression can exist.

In that case, it would be important to emphasize, from the start, that the goal is not to replace people with algorithms, but rather provide them with a tool, to make even better decisions, or answer new questions that were out of scope before.

Answer 7

There are many reasons, but at the end of the day I believe it comes down to people like us trying to make industrial customers understand what optimisation is and how it works. Although this is a natural approach for academically trained people, this is not how products or services are sold, nor how adoption is driven. Adoption of a technology in every single field has historically been driven by automation.

People without the right training are unlikely to be able to really understand what we do, and that's fine. Unfortunately, it is hard to approach the adoption process in a different way due to (i) a lack of out-of-the-box solutions that clients can use and understand, and (ii) any decent industrial software in our field being obscenely expensive.

Think about why people like machine learning so much: in their mind, they just give someone a bunch of data and something good "magically" happens. Even better, all software they will need to use is free!

On the other hand, I have never once come across a situation where we could help a client adopt an OR solution (assuming they're not just buying a new solver or a maths compiler) without some consulting, a customised solution, and some integration work. The lack of our community's ability to do so is, in my opinion, due to a lack of tools designed (at least partially) for non-OR people. Even though some decent software for specialised things does exist, it is so expensive that very few people can afford it (or are willing to take the leap of faith), which stagnates market growth. This specialisation is literally killing the market: no-one wants to fork out half a million or more for something that doesn't do everything they (think they) might need.

One could argue of course that what we do is complicated and requires a certain level of boilerplate, but that's just not true - machine learning used to be equally obscure, but hype along with very good tools have turned that into something that people in the industry want to adopt. Heck, even maths enjoys smaller adoption barriers, largely due the efforts of companies like Wolfram or Mathworks.

In OR, the lack of this plug-and-play functionality makes non-experts distrustful: they don't understand the technology, they don't understand what it is that we'll do for them, they don't understand how they'll interact with it, they don't feel confident they'll be able to keep using the solution without the people who set everything up in the first place, they have to endure long iterations of meetings until we become able to communicate properly, and so on.

Answer 8

Lack of Effective Communication and Challenge Discussions. When you are involved as a technical person leading model development in an optimization project, sometimes lack of communication between you and your clients can affect your model results and ultimately the applicability in real life. You miss one simple constraint and your results become impractical. This is especially true if you are new to the business and you have little experience of how the processes works on your client side.

In such cases, the best way is to ask as many questions as possible, even if they might feel naive to other people. Do not be afraid to use simple terminologies because its often the case that plant engineers or process engineers have little to no experience of operations research. Right effort at right time can save you from a lot of pain later.

Be open and be 100% prepared for the actual outcomes to deviate from what you see in your computer and not to succeed in your first few iterations while you would ideally expect to. Solving optimization problems on computer is one thing and bringing optimization models to life is another.

Answer 9

I haven't had the experience in any project in the industry but in my research which is on a collaborative project, I can see the potential reasons for failure:

• Lake of a common understanding of the problem to solve.

Usually, it is very hard to have the same understanding of details and the same technical terms if you don't have the same background (in terms of educational major).

• Implementation is hard

It is sometimes very hard for the industry or project partner to accept all the details of what you propose and then implement it. As an example, when your solution to a problem in a factory floor needs the replacement of a production cell, it is almost impossible to convince the management to implement the changes unless you speak in their own language (profit, profit, profit). So you need to have an explanation (in terms of money) for all aspects of your solution unless the project will fail.

• Lack of sufficient amount of necessary data

Although most of the time it is hard to gather the data that you need to model the details of the project, sometimes you won't have those data because of either it's impossible to gather the data with the equipment in use, or it needs a long time and hard work to get the data so in this situation you need either to consider some assumptions or estimate them. Both of ways are very prone to failure.