# How to model nonlinear regression?

As part of my research in statistics, I recently stumbled upon this paper1 which provides an operational perspective into linear models.

In simple linear regression, quadratic programming can be used to solve the problem where for least squares, the objective is $$\begin{array}{ll} \min & Q(\beta) = (Y-X\beta)'(Y-X\beta)\\ \text{s.t.} & A\beta\ge C \\ & \beta \ge 0 \end{array}$$ The notation should be fairly self-explanatory.

However, for nonlinear regression, things are more complicated. For example, the Michaelis-Menten model is multivariate, given by $$f(x,\beta)=\beta_1 x/(\beta_2+x)$$. It is possible to transform any nonlinear model to a linear one, but there is an element of risk as the errors are altered.

Is there any literature that provides a procedure on how to tackle this type of regression?

Reference

 Wang, D. Q., Wu, B. (2006). Teaching Linear Models Based on Operations Research in Statistical Education. ICOTS-7.

• Fully on topic, thanks! May 31, 2019 at 11:52
• "It is possible to transform any nonlinear model to a linear one, but there is an element of risk as the errors are altered." I don't believe this to be the case. the nonlinear model (x is data, a is parameter to be estimated), y ~ exp(a*x) can be transformed to the linear model log(y) ~ a + log(x), in which case the (residual) error distributions being optimized are different between the two different models, as you say. But even this can not be done for general nonlinear models. May 31, 2019 at 12:07
• @MarkL.Stone Evidently the errors in $\ln y=\ln\mu +\alpha x+\epsilon$ are additive but are multiplicative after exponentiation. This can be a problem if the logged errors are already considerable. May 31, 2019 at 12:09
• @TheSimpliFire See my comment now, which I was expanding when you posted your comment. May 31, 2019 at 12:12

NEOS has a nice web page on nonlinear least squares. It contains several classic (i.e., not so new, but still good) references for nonlinear least squares.

There is a very nice introduction to the mathematics of nonlinear least squares, and the algorithms to solve them, "Non-linear least squares problems: The Gauss-Newton method" by Niclas Börlin.

There are indeed specialized algorithms for nonlinear least squares, such as Levenberg-Marquardt, and Gauss-Newton (a.k.a. Iterative Linear Least Squares), which exploit the special structure in least squares problems Generally, they only provide an advantage vs. more general nonlinear optimization algorithms when the residuals at the optimum are "small". If the residuals at the optimum are not small, the nonlinear least squares algorithms lose their advantage, and if not implemented well, may even fail.

Given the advances in general nonlinear optimization algorithms and software over the last several decades, I think that in most cases, specialized nonlinear least squares algorithms usually don't have much merit, and high quality more general software, which also is more flexible in terms of allowing constraints, can, and should be used.

Finally, note that (semi-) rigorous global optimizers, such as BARON, can find the globally optimal solution to non-convex (nonlinear regression) problems, at least if the problem isn't too large (in terms of number of parameters to be estimated).

From a pure optimization point of view you can say that it is possible to transform a linear fractional problem into a linear one (see for example here). In your case, this seems not feasible, as you have a product of variables in the numerator.

Nonlinear regression problems are usually solved by employing continuous optimization algorithms such as gradient descent, Newton's method and their friends.

• For highly non-convex nonlinear regression problems, you might consider probabilistic approaches too, like simulated annealing, or genetic algorithms. May 31, 2019 at 11:17
• For that I'd rather recommend multiple starts while using an continuous solver as the problem seems to be differentiable and for these problems using gradient (or even (approximated) hessian) information will be highly beneficial. simulated annealing or genetic algorithms usually perform better when there are integral variables involved. May 31, 2019 at 12:08
• There are (semi-) rigorous global optimizers, such as BARON, which can find the globally optimal solution to non-convex (nonlinear regression) problems, at least if the problem isn't too large (in terms of number of parameters to be estimated). May 31, 2019 at 12:15
• Good point @mark-l-stone. You will have to check whether your problem size permits you to use these kind of solvers. Note however that even for small size problems (<50 variables) some problems might take a long time to be solved to optimality. By using these solvers you can on the other hand then be sure that the solution is indeed the best possible. From my experience this is usually not asked for regression problems. May 31, 2019 at 12:19

Here is a book often used by statisticians: Nonlinear Regression Analysis and Its Applications by Douglas M. Bates and Donald G. Watts. Douglas Bates is a member of the R core team and not without responsibility for the nonlinear regression routines in R ...

This other book by Seber & Wild can also be useful.