How can I estimate the monetary savings of a operation research application?

Question

Developing operation research applications for industry clients is often very costly since it is in my experience often a custom special development for the client. The cost of developing a running system is often far beyond 100k $/€/... and then there is also maintenance and if you need commercial solvers, then there are also a lot of repetitive costs for licensing, etc. while in production.

• How can I estimate the savings a potential operations research application has? (Maybe one can break this down into the maintenance costs/development costs/savings)
• It would be also interesting to differentiate before/after you developed the solution and it is running in production? (Since also after it is running it is difficult to measure the improvement.)

Some thoughts:

A critical topic is always when people at the company solving this problem (or part of the problem) currently as their jobs. Then, maybe you estimate that instead of two people doing this job, now maybe one person can do the job. Then one could estimate based on their salaries that they might save this money per year.

There the before/after effects are in general good measures after the system is running in production for some time. But it is a highly sensitive topic, because we do not want anybody to lose their jobs. This is a complicated topic that might be in itself a separate question.

The second potential savings is that the solution obtained by the tool is far superior to the current heuristic solution done by the people. However, this is difficult to estimate before and even after you are running your solution. Because most often you are only solving optimally a small frame of large problem and if there are a lot of soft constraints and, in the end, the solution is not taken as it is and analyzed, and different solutions are compared by the people working, then how an I access the benefit of the solution vs. the solution before?

Answer 1

Your application is either replacing some other application (including tasks that are done manually), enhances an existing solution, or it's just a new application altogether.

Although monetary metrics such as the amount of savings, added revenue due to better handling of requests, shorter run time, better results, and so on are more tangible, depending on the type of application, you may need to talk about savings in terms of non-monetary metrics too. So, let me share two cases where non-monetary discussions can help.

Suppose you have an application that helps people plan their days better (call it scheduler). You may just help the planners to be more productive and have larger uninterrupted stretches of time. What are the benefits? Maybe you can find studies that show how much time is wasted on average in a normal day of a planner by just doing very routine and non-value-added tasks, and then discuss how much of those times can be saved by your application and be used better (not an OR application but think of Slack or MS Teams).

Or maybe you have an application that is opening a type of market to the business that was not even possible. To add to Robert's delivery business, consider an application that allows a business to enter a market that they've not sought before. Maybe one is a company doing linehaul but with your application, they can also handle less-than-truckload. Obviously, one needs to do a thorough study to forecast how much of the market one can get but sometimes, you can just talk about the possibilities and let the client's decision-makers' minds do the job.

Now about your last point, regarding applications that can solve a small frame of problem optimally but may fail as more constraints are added. I think that's another problem. Because, when you talk about an application, you should make sure you cover a range of different scenarios and be up-front about its limitations. Think of this similar to writing unit tests for your code! You should know what each function does and what it can't do and you should have proper testing and messages in place to let the user know what they can and can't expect from your application.

Answer 2

I think this is easiest for applications where cost reduction is an explicit goal of the model.

Let's say a company is in the delivery business and uses some specific method / heuristic to assess routes.

If your solution finds better routes and you can compare them directly, you could compute something like 10% reduction in delivery cost and translate that in monetary savings.

Other kinds of improvements (e.g. reduced risks) always have to be translated into a common (monetary) unit, I guess.

Answer 3

Complementing the other answers, I'd like to add you can design experiments trying to isolate the application's effect in order to measure it easier. One could start by using the new solution on a subset¹ of instances. For example:

• if it's related to geographically isolated problems/models, choose some of them to apply change while the other ones keep working business as usual
• if it's a new personalized model (e.g. for pricing), choose the treatment and control group to see how different users react.
• and so on.

There are several resources to learn about experiment design, causality analysis and A/B testing, among them:

• Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, a recent book by Kohavi, Tang and Xu. The authors have a ton of experience running experiments at large companies.

• If you prefer online courses, there's a free one by Google at Udacity on the topic, and another one offered by UC San Diego at Coursera.

I've been reading the book and it shares several insights to take into account. Never attended the online courses so can't speak about their quality.

Also, I think the earlier you (and the relevant stakeholders) define the project's goal and metrics for measuring success, the better for experimentation, development, implementation and final impact. In my opinion, defining impact goals clearly² and correctly designing experiments are both relevant steps to ensure success in this kind of application.

---

^{1 Ideally a representative subset, whenever real-world conditions allow it.
2 Including how the experiment will be measured, and how this goal relates to the project's "big picture" main objective as a whole.}