Best ways to use machine learning / AI as an OR scientist

Question

I have come across GUROBI's webinar "Mathematical optimization and machine learning".

In essence, Mathematical Optimization (MO) and Machine Learning (ML) are different but complementary technologies. Simply put – Mixed Integer Programming (MIP) answers questions that ML cannot. Machine learning makes predictions while MIP makes decisions. They call Artificial Intelligence (AI) the combination of both technologies.

More specifically :

1. ML predictions can determine the need to make a MO decision.
2. ML predictions can be used as MO decision constraints (when the data is missing).

Other interesting examples are given here by FunArtech, a startup in Montreal who claim to be the future of AI. For example, they claim that "some research is conducted in order to predict on what variable and how to branch in search trees."

This is interesting and leads me to asking the following (broad, I agree) questions :

1. In your personal experience, as an OR scientist, to what extent and how have you used ML for MO ?
2. What would you recommend MO practitioners to learn in priority ? General ML techniques ? Neural networks ? Data statistics ?

And I am aware the question is broad and this may be the wrong place to ask.

---

Related questions :

• Examples of ML applied to OR
• Use of ML to determine parameter values
• What are the tradeoffs between “exact” and Reinforcement Learning methods for solving optimization problems

Answer 1

First, I would argue that the technologies are not necessarily complementary, but can be supplementary. A paper uploaded to Optimization Online just last month discusses the use of machine learning to decide how to linearize MIQP models.

My answer to your first question is no doubt shaped by my being an academic, which means I rarely solve the same problem twice (other than perhaps running multiple instances before submitting the paper and moving on to something else). I never use ML for optimization, because ML requires a substantial library of solved instances for training purposes, which I never have.

I don't have an answer to your second question as a result of never having tried to use ML.

Answer 2

OK, let's try to answer this better. I'm not going to illustrate or point to the right references. I just want to throw some pointers. Beside, I write about my vision which is not mainstream (and I believe it is not possible not to be personal about this: experts do not agree for the moment).

"In essence, Mathematical Optimization (MO) and Machine Learning (ML) are different but complementary technologies."

Yes and no. A few years back, ML (or parts of ML) was considered as a sub field of MO... ML uses MO to optimize its predictions (for instance, to minimize the discrepancy between its predictions and the reality (supervised learning)). But yes, they are complementary to say the least as one (ML) predicts, the other (MO) optimizes. The frontier between both is blurry.

"Simply put – Mixed Integer Programming (MIP) answers questions that ML cannot. Machine learning makes predictions while MIP makes decisions. They call Artificial Intelligence (AI) the combination of both technologies."

Exactly, except that some AI experts kicked MO out of AI. And that AI is much broader than that.

"More specifically :

ML predictions can determine the need to make a MO decision. ML predictions can be used as MO decision constraints (when the data is missing)."

Sorry, but I don't understand this. ML is able to find patterns in your data and reproduces up to a point the "past" (it can be used to innovate and optimize but it is not its strongest point: my view). If you want to be pro-active and make decisions based on optimization (and not observations) then you use MO. If you want to make decisions based on observations (and not optimization) you use ML. At Funartech, we combine both to take decisions. ML methods are data resources hungry, i.e. it is almost impossible to use ML if you do not have (clean) data. And most of the time, you need huge amounts of data.

"Other interesting examples are given here by FunArtech, a startup in Montreal who claim to be the future of AI. For example, they claim that "some research is conducted in order to predict on what variable and how to branch in search trees.""

We do claim so because we have better practical and theoretical results. Yes we do.

In a very basic and scholar approach, you could divide the use of ML and MO in four categories:

1. the combination of ML and MO as two black boxes used separately. This has become mainstream in Québec (but not in Silicon Valley for instance) and other parts of the world. There are several examples on our website (https://www.funartech.com/approach/use-cases).

2. and 3. You can use one to improve the other. MO can be used to optimize and improve ML (this IS ML...) and it goes the other way too: ML can be used to improve MO algorithms. The use of ML to branch in a search tree is one such example.

3. we have found a new hybridization of both fields with new algorithms. This is an industrial secret for the moment.

"This is interesting and leads me to asking the following (broad, I agree) questions :

In your personal experience, as an OR scientist, to what extent and how have you used ML for MO ?"

I my personal experience, I mix both all the time.

"What would you recommend MO practitioners to learn in priority ? General ML techniques ? Neural networks ? Data statistics ?"

I would recommend to first figure out what type of problems you want to solve and to dig the corresponding ML/MO parts needed to solve such problems. ML is not easy but as a MO practitioner you have a base knowledge that will help you understand the mathematics of ML better than most ML experts that don't have such a mathematical background. Even if the ML field is broad (huge!), the basic principles and ideas are "small". The level of mathematics required is the one of a first year student at university, so totally accessible.

Answer 3

I would refer interested readers to the answer here & here (by @Marco Lübbecke). My recent applied research work (at the intersection of ML and OR) [preprint] is based on a similar thought.

I have been doing research in the development of an airline crew pairing optimization framework for large-scale and complex flight networks (named as AirCROP). Its kernel is a Column Generation (CG)-based heuristic [preprint].

One day, the industry's contact manager asked me if there is a way to build intelligence in the framework, i.e., is it possible to learn something from previous runs to enhance the performance of a future run, or is it possible to do the same among the iterations of a single run itself? Though I have some ideas regarding the former (for future), I have worked on the latter in the above-mentioned preprint.

In the above work, I have attempted to learn some important flight-connections that got rejected in the previous CG-iterations but could be useful in the future iterations.

For this, I have developed an online-learning framework to learn such flight-connections in an unsupervised manner using a Variational Graph Autoencoder (VGAE, a Graph Neural Network) and use them on-the-fly for creating new columns so as to enhance the CG-convergence and/or decrease the required CG-iterations. It has shown significant improvements but the solution-ing time of the overall framework has increased a lot, almost doubled, due to the addition of the learning time of VGAE.

Now, I have an intuition and is developing a strategy to replace the ML part so that I am still able to do the same but in lesser time.