18
$\begingroup$

Looking at the news as well as at content of tech conferences, I think it is fair to say that AI is getting a lot of attention -- one might even call it an AI hype (like in the 80's). Plenty of articles describe how CEO's and CIO's are either exploring or are being advised they should be exploring AI within their company. This reflects in big budgets and the creation of dedicated AI and "data scientist" teams.

Yet, in practice, these teams seem to treat AI as a substitute by Machine Learning -- and specifically Deep Learning Neural Nets (or Random Forests at best). In any case, not constraint solvers (regardless of the underlying tech being MH, LP or CP).

Do constraint solvers have trouble riding the AI wave? Why?

Opinionated disclaimer: this question is provoking opinionated, rather than fact-based answers. Take all answers with a grain of salt. See comments discussion for more info.

$\endgroup$
  • 4
    $\begingroup$ I very much like this discussion; but is this primarily opinion based and may violate the rules? $\endgroup$ – Marco Lübbecke Jul 17 '19 at 12:57
  • 4
    $\begingroup$ Let's call optimization "Deep Decision Making" (credit to Marco) and not only we will ride the wave, but we are creating the new generation of AI models! $\endgroup$ – EhsanK Jul 17 '19 at 13:32
  • 1
    $\begingroup$ @MarcoLübbecke Good point. Do we have the same rules here as StackOverflow? There are other StackExchange websites that probably do allow opinions, such as Software Recommendations. Shall I delete this question? $\endgroup$ – Geoffrey De Smet Jul 18 '19 at 7:03
  • 3
    $\begingroup$ We could hold opinion based questions on Quora, banning them here, to avoid mixing science with opinion. Alternatively, we could allow them here - to pull the OR community together - but annotate the questions/answers with an "opinionated disclaimer" line. $\endgroup$ – Geoffrey De Smet Jul 18 '19 at 7:05
  • 3
    $\begingroup$ Personally I like this question and I up-voted it. I do not think it is too subjective, or anyway I think it is "good subjective" when evaluated as suggested in Good Subjective, Bad Subjective. But that's just my opinion, others might differ. Meta is probably a better place to have a discussion about it than here in the comments. @MarcoLübbecke since you raised the issue, maybe you want to post on Meta? Link to this question and ask whether the community feels it's too subjective? $\endgroup$ – LarrySnyder610 Jul 18 '19 at 15:21
6
$\begingroup$

I go out on a limb and have a controversial answer:

All the problems in machine learning are not optimization problems.

Before you object, think twice. What about training a deep net? Isn't that an optimization problem. It sure is! But the underlying problem, i.e. the one which is the machine learning problem, is to build a deep net that is able to predict the outcome for unseen data. (If you know the terms: Training the net is minimizing the empirical loss, but the goal is to minimize the expected loss over the unknown distribution of the data - why minimization of the empirical loss works as good as is does is not full explained to this day.)

Isn't clustering an optimization problem? Well, yes and no. You can cluster according to some objective and then you need to solve an optimization problem. But there are different objectives which could make sense and again, the underlying problem in unsupervised learning does not have an objective, but the goal is "to discover hidden clusters in the data".

Isn't regression an optimization problem? The answer is the same: You can use optimization to compute the regressors, but the underlying problem is prediction of unseen data.

Don't get me wrong: Machine learning uses and needs a lot of optimization (and constraint programming is no exception) but the real problems in machine learning (mostly prediction) are of a different type.

I may have another point as to why AI/ML gets so much attention: Recent results in machine learning are surprising (to both scientists and non-scientists), since ML can make the computer do things which we as humans can not explain, because we have no idea how they work. I (and I guess anybody else) do not have any idea on how to write an algorithm that makes the computer tell what is on an image. It's not that we did not try. Computer vision developed tools for about 70 years, but nobody could write a programm that makes a computer tell "This image shows a car." Now machine learning can produce algorithms which do exactly that. But the difference is: The programmer did not directly encode any mechanism to detect cars. That's a big difference to all the things that have been done in optimization. We have tons of great code that can make the computer do incredible things. But we worked hard to figure out all that lines of code and basically we can tell exactly what we told the computer to do. If you don't like images: It's the same with the question "How can we make a computer understand spoken language?"

| improve this answer | |
$\endgroup$
6
$\begingroup$

This is an interesting question. One way that I'm thinking about this question is to look at how companies are actually using the term AI. I wrote blog post arguing that AI is being used by industry as an umbrella term. And as an umbrella term, it includes deep learning, machine learning, and optimization. I didn't call it out in the blog, but Constraint Programming would definitely fall under the umbrella. (Of course, reasonable people could argue that industry is using the term incorrectly.)

I think that when the latest version of the term AI surfaced around 2012 with breakthroughs in Deep Learning, the scientists were not using it as an umbrella term. Just like you mentioned in your post that in 80's, researchers thought they were on to something different.

But, the business community seems to have taken over the term AI over the last few years. It has largely replaced the term "analytics." I think this is a good thing-- the term "analytics" allowed people to get away with just doing reporting. AI implies that you need some algorithms to help make decisions.

We've also seen a recent rise in the term "Artificial General Intelligence" (AGI) to refer to algorithms that think more like a person. AGI seems to be what people were (or are) thinking Deep Learning may do and, maybe, what people thought CP would do in the 80's.

| improve this answer | |
$\endgroup$
2
$\begingroup$

AI in the 80s has much of the theory, a fraction of the compute capacity and none of the data.

Machine Learning covers a very broad field of statistical modelling. A "data scientist" faces operational and practical challenges in handling the huge volume of data. Of course there is hype but there is also a step-change in effectiveness.

The Vanishing Gradient Problem had blocked efforts to learn complex features automatically, Hinton's paper in 2006 allowed unsupervised self-training of much more complex networks given enough data.

Continuous dense data is often susceptible to Linear analysis and the kernel trick is very powerful for non-linear fitting. Robustness and sparsity can be incorporated by the LASSO penalty which is well understood. Constrained optimisation is tractable for convex problems, but scalability becomes an issue and even proving you have a convex model can be tough for general types of constraint. The big-data approach often involves imposing sparsity, a lot of progress has been made with much stronger non-convex models : compressive sensing. There are no guarantees and some data is better suited to certain techniques.

Decision trees and forests are not necessarily cutting edge, random forest is one example of a general technique "Bagging". I think the question is better posed if you split the optimisation methods (convex/non-convex) and the numerical methods (algebraic/scaling).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.