11
$\begingroup$

This post is not really about a specific question but rather a topic I am curious about to know more.

We know that when it comes to integrate machine/statistical learning with optimization for the purpose of replacing ideal equations with data-driven models, the choices are quite limited - mostly regression (linear, polynomial) and spline models or at best a shallow neural network that can be decoded back by hand as an analytical expression. Such limited choices naturally mean that generally speaking highly non-linear functions would be difficult to approximate well. Deep neural networks are known to do exceptionally well when it comes to model complex functions. However, the problem is that they are hard to represent then as an expression/formula (without needing a mile long paper), which however if becomes possible would indeed be great from optimization perspective.

I would like to know if there been some work done on this topic previously or is any active research going on that you guys may be aware of. Keen to hear about it. Thanks.

EDIT - My question is more from the side of using AML's (Pyomo, JuMP etc), where one has the luxury of directly writing the algebraic expressions.

Improve this question
$\endgroup$
4

2 Answers 2

Reset to default
5
$\begingroup$

It is possible. Although I haven't done this for a deep neural network (my experience was on a multi-layer perceptron), it shouldn't be that different procedure-wise.

Considering that each neuron represents an analytical expression, it very possible to unwrap your network into one massive function. You can in principle do this using SymPy. You can walk your NN and recursively print the neurons to functions into a massive string and then parse it with SymPy. This will give you all the information you need to build a Pyomo model in code, however there are quite a few pitfalls.

  1. SymPy will likely not be able to handle this for any NN large enough to be useful.
  2. Pyomo (at least up till a couple of years ago) doesn't scale very well for big expressions.
  3. The resulting optimisation problem is highly non-linear and will probably destroy most solvers.
  4. A lot of code is required to glue all these pieces together properly.

Generally speaking, neural networks are not very amenable to classical optimisation technology, because (i) there are no constraints to exploit in order to reduce the solution space, (ii) the problems can be so large that memory/differentiation efficiency issues overpower most implementations. One way to handle this in practice would be to plug the model into a stochastic solver, but at that point we are not really benefiting from classical optimisation theory anymore.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Wow! This is so much interesting information in just one single answer. I understand, what's the major disadvantage of using extremely non linear functions in AML's like Pyomo. Thanks a lot. $\endgroup$
    – chupa_kabra
    Commented Aug 28, 2019 at 4:50
  • 1
    $\begingroup$ There's no inherent disadvantage with respect to pyomo, it's just that solvers will have a lot of trouble solving the resulting nonlinear problem. $\endgroup$
    – Nikos Kazazakis
    Commented Aug 28, 2019 at 9:58
4
$\begingroup$

Instead of deep learning, you can try to use so-called "explainable machine learning", where you learn directly Boolean rules which are generally restricted to be short (of course, accuracy may suffer in this case). See, for example, this paper: https://papers.nips.cc/paper/7716-boolean-decision-rules-via-column-generation.pdf

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.