AI gets a lot of attention these days. Does constraint optimization get more attention, too? Why (not)?

Question

Looking at the news as well as at content of tech conferences, I think it is fair to say that AI is getting a lot of attention -- one might even call it an AI hype (like in the 80's). Plenty of articles describe how CEO's and CIO's are either exploring or are being advised they should be exploring AI within their company. This reflects in big budgets and the creation of dedicated AI and "data scientist" teams.

Yet, in practice, these teams seem to treat AI as a substitute by Machine Learning -- and specifically Deep Learning Neural Nets (or Random Forests at best). In any case, not constraint solvers (regardless of the underlying tech being MH, LP or CP).

Do constraint solvers have trouble riding the AI wave? Why?

Opinionated disclaimer: this question is provoking opinionated, rather than fact-based answers. Take all answers with a grain of salt. See comments discussion for more info.

Answer 1

I go out on a limb and have a controversial answer:

All the problems in machine learning are not optimization problems.

Before you object, think twice. What about training a deep net? Isn't that an optimization problem. It sure is! But the underlying problem, i.e. the one which is the machine learning problem, is to build a deep net that is able to predict the outcome for unseen data. (If you know the terms: Training the net is minimizing the empirical loss, but the goal is to minimize the expected loss over the unknown distribution of the data - why minimization of the empirical loss works as good as is does is not full explained to this day.)

Isn't clustering an optimization problem? Well, yes and no. You can cluster according to some objective and then you need to solve an optimization problem. But there are different objectives which could make sense and again, the underlying problem in unsupervised learning does not have an objective, but the goal is "to discover hidden clusters in the data".

Isn't regression an optimization problem? The answer is the same: You can use optimization to compute the regressors, but the underlying problem is prediction of unseen data.

Don't get me wrong: Machine learning uses and needs a lot of optimization (and constraint programming is no exception) but the real problems in machine learning (mostly prediction) are of a different type.

I may have another point as to why AI/ML gets so much attention: Recent results in machine learning are surprising (to both scientists and non-scientists), since ML can make the computer do things which we as humans can not explain, because we have no idea how they work. I (and I guess anybody else) do not have any idea on how to write an algorithm that makes the computer tell what is on an image. It's not that we did not try. Computer vision developed tools for about 70 years, but nobody could write a programm that makes a computer tell "This image shows a car." Now machine learning can produce algorithms which do exactly that. But the difference is: The programmer did not directly encode any mechanism to detect cars. That's a big difference to all the things that have been done in optimization. We have tons of great code that can make the computer do incredible things. But we worked hard to figure out all that lines of code and basically we can tell exactly what we told the computer to do. If you don't like images: It's the same with the question "How can we make a computer understand spoken language?"

Answer 2

This is an interesting question. One way that I'm thinking about this question is to look at how companies are actually using the term AI. I wrote blog post arguing that AI is being used by industry as an umbrella term. And as an umbrella term, it includes deep learning, machine learning, and optimization. I didn't call it out in the blog, but Constraint Programming would definitely fall under the umbrella. (Of course, reasonable people could argue that industry is using the term incorrectly.)

I think that when the latest version of the term AI surfaced around 2012 with breakthroughs in Deep Learning, the scientists were not using it as an umbrella term. Just like you mentioned in your post that in 80's, researchers thought they were on to something different.

But, the business community seems to have taken over the term AI over the last few years. It has largely replaced the term "analytics." I think this is a good thing-- the term "analytics" allowed people to get away with just doing reporting. AI implies that you need some algorithms to help make decisions.

We've also seen a recent rise in the term "Artificial General Intelligence" (AGI) to refer to algorithms that think more like a person. AGI seems to be what people were (or are) thinking Deep Learning may do and, maybe, what people thought CP would do in the 80's.

Answer 3

AI in the 80s has much of the theory, a fraction of the compute capacity and none of the data.

Machine Learning covers a very broad field of statistical modelling. A "data scientist" faces operational and practical challenges in handling the huge volume of data. Of course there is hype but there is also a step-change in effectiveness.

The Vanishing Gradient Problem had blocked efforts to learn complex features automatically, Hinton's paper in 2006 allowed unsupervised self-training of much more complex networks given enough data.

Continuous dense data is often susceptible to Linear analysis and the kernel trick is very powerful for non-linear fitting. Robustness and sparsity can be incorporated by the LASSO penalty which is well understood. Constrained optimisation is tractable for convex problems, but scalability becomes an issue and even proving you have a convex model can be tough for general types of constraint. The big-data approach often involves imposing sparsity, a lot of progress has been made with much stronger non-convex models : compressive sensing. There are no guarantees and some data is better suited to certain techniques.

Decision trees and forests are not necessarily cutting edge, random forest is one example of a general technique "Bagging". I think the question is better posed if you split the optimisation methods (convex/non-convex) and the numerical methods (algebraic/scaling).