16
$\begingroup$

As part of my research in statistics, I recently stumbled upon this paper1 which provides an operational perspective into linear models.

In simple linear regression, quadratic programming can be used to solve the problem where for least squares, the objective is $$\begin{array}{ll} \min & Q(\beta) = (Y-X\beta)'(Y-X\beta)\\ \text{s.t.} & A\beta\ge C \\ & \beta \ge 0 \end{array}$$ The notation should be fairly self-explanatory.

However, for nonlinear regression, things are more complicated. For example, the Michaelis-Menten model is multivariate, given by $f(x,\beta)=\beta_1 x/(\beta_2+x)$. It is possible to transform any nonlinear model to a linear one, but there is an element of risk as the errors are altered.

Is there any literature that provides a procedure on how to tackle this type of regression?

Reference

[1] Wang, D. Q., Wu, B. (2006). Teaching Linear Models Based on Operations Research in Statistical Education. ICOTS-7.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Fully on topic, thanks! $\endgroup$
    – LarrySnyder610
    May 31, 2019 at 11:52
  • 2
    $\begingroup$ "It is possible to transform any nonlinear model to a linear one, but there is an element of risk as the errors are altered." I don't believe this to be the case. the nonlinear model (x is data, a is parameter to be estimated), y ~ exp(a*x) can be transformed to the linear model log(y) ~ a + log(x), in which case the (residual) error distributions being optimized are different between the two different models, as you say. But even this can not be done for general nonlinear models. $\endgroup$
    – Mark L. Stone
    May 31, 2019 at 12:07
  • $\begingroup$ @MarkL.Stone Evidently the errors in $\ln y=\ln\mu +\alpha x+\epsilon$ are additive but are multiplicative after exponentiation. This can be a problem if the logged errors are already considerable. $\endgroup$
    – TheSimpliFire ♦
    May 31, 2019 at 12:09
  • $\begingroup$ @TheSimpliFire See my comment now, which I was expanding when you posted your comment. $\endgroup$
    – Mark L. Stone
    May 31, 2019 at 12:12

3 Answers 3

Reset to default
10
$\begingroup$

NEOS has a nice web page on nonlinear least squares. It contains several classic (i.e., not so new, but still good) references for nonlinear least squares.

There is a very nice introduction to the mathematics of nonlinear least squares, and the algorithms to solve them, "Non-linear least squares problems: The Gauss-Newton method" by Niclas Börlin.

There are indeed specialized algorithms for nonlinear least squares, such as Levenberg-Marquardt, and Gauss-Newton (a.k.a. Iterative Linear Least Squares), which exploit the special structure in least squares problems Generally, they only provide an advantage vs. more general nonlinear optimization algorithms when the residuals at the optimum are "small". If the residuals at the optimum are not small, the nonlinear least squares algorithms lose their advantage, and if not implemented well, may even fail.

Given the advances in general nonlinear optimization algorithms and software over the last several decades, I think that in most cases, specialized nonlinear least squares algorithms usually don't have much merit, and high quality more general software, which also is more flexible in terms of allowing constraints, can, and should be used.

Finally, note that (semi-) rigorous global optimizers, such as BARON, can find the globally optimal solution to non-convex (nonlinear regression) problems, at least if the problem isn't too large (in terms of number of parameters to be estimated).

Improve this answer
$\endgroup$
8
$\begingroup$

From a pure optimization point of view you can say that it is possible to transform a linear fractional problem into a linear one (see for example here). In your case, this seems not feasible, as you have a product of variables in the numerator.

Nonlinear regression problems are usually solved by employing continuous optimization algorithms such as gradient descent, Newton's method and their friends.

Improve this answer
$\endgroup$
4
  • $\begingroup$ For highly non-convex nonlinear regression problems, you might consider probabilistic approaches too, like simulated annealing, or genetic algorithms. $\endgroup$
    – c2rosa
    May 31, 2019 at 11:17
  • $\begingroup$ For that I'd rather recommend multiple starts while using an continuous solver as the problem seems to be differentiable and for these problems using gradient (or even (approximated) hessian) information will be highly beneficial. simulated annealing or genetic algorithms usually perform better when there are integral variables involved. $\endgroup$
    – JakobS
    May 31, 2019 at 12:08
  • 3
    $\begingroup$ There are (semi-) rigorous global optimizers, such as BARON, which can find the globally optimal solution to non-convex (nonlinear regression) problems, at least if the problem isn't too large (in terms of number of parameters to be estimated). $\endgroup$
    – Mark L. Stone
    May 31, 2019 at 12:15
  • $\begingroup$ Good point @mark-l-stone. You will have to check whether your problem size permits you to use these kind of solvers. Note however that even for small size problems (<50 variables) some problems might take a long time to be solved to optimality. By using these solvers you can on the other hand then be sure that the solution is indeed the best possible. From my experience this is usually not asked for regression problems. $\endgroup$
    – JakobS
    May 31, 2019 at 12:19
2
$\begingroup$

Here is a book often used by statisticians: Nonlinear Regression Analysis and Its Applications by Douglas M. Bates and Donald G. Watts. Douglas Bates is a member of the R core team and not without responsibility for the nonlinear regression routines in R ...

This other book by Seber & Wild can also be useful.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.