Probability & Statistics Codexery

Non-linear least squares

Last updated

A stationary point, such as a maximum or saddle point, will appear in the objective function between any two minima when multiple minima are present.[1] At a stationary point of the objective function, the normal equations matrix lacks positive definiteness.[1]

Key facts and history

If divergence occurs, a simple expedient is to reduce the length of the shift vector by a fraction.[2] The shift vector length may be successively halved until the objective function value is less than at the last iteration.[2] The Marquardt parameter modifies the normal equations by adding λ times the identity matrix.[3] Alternating variable search varies each parameter in turn to reduce the sum of squares.[2]

It is sometimes possible to convert a non-linear model into a linear one through a transformation.[4] Most iterative minimization algorithms rely on the basic assumption that a linear approximation is valid, particularly near the best estimator.[4] Determining the Marquardt parameter has been the subject of multiple proposed strategies.[5]

What the research sources say

The normal equations are more generally written with a weight matrix W when observations are correlated.[6] Generalized least squares can be applied for inference when a linear approximation holds, using the Linear Template Fit equations.[4] Although a line search could optimize the fraction f for shift-cutting, the need to recalculate the objective function for each trial makes stringent optimization not worthwhile.[2] Overly stringent optimization of the Marquardt parameter is wasteful, similar to the case with shift-cutting.[5]

The smallest singular value of the Jacobian can serve as the cut-off value for the Marquardt parameter.[5]

References

  1. Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: In the absence of round-off error and of experimental error in the independent variable the normal equations matrix would be singular (original not read by us) en.wikipedia.org
  2. Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: M.J. Box, D. Davies and W.H. Swann, Non-Linear optimisation Techniques, Oliver & Boyd, 1969 (original not read by us) en.wikipedia.org
  3. Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: This technique was proposed independently by Levenberg (1944), Girard (1958), Wynne (1959), Morrison (1960) and Marquardt (1963). Marquardt's name alone is used (original not read by us) en.wikipedia.org
  4. Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: Britzger, "The Linear Template Fit", Eur. Phys. J. C, 2022, p. 731 (original not read by us) en.wikipedia.org
  5. Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: C.L. Lawson and R.J. Hanson, Solving Least Squares Problems, Prentice–Hall, 1974; R. Fletcher, UKAEA Report AERE-R 6799, H.M. Stationery Office, 1971 (original not read by us) en.wikipedia.org
  6. Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: This implies that the observations are uncorrelated. If the observations are correlated, the expression <math display="block">S = \sum_k \sum_j r_k W_{kj} r_j</ (original not read by us) en.wikipedia.org

Each numbered sentence above was checked against the source it cites before publication. Sentences citing Wikipedia adapt text from the English Wikipedia article Non-linear least squares (contributors), used under CC BY-SA 4.0; this entry is available under the same licence. Other sources were used for facts only; no text or images are copied from them.

More in Probability & Statistics

Sources

Compiled from Wikipedia and the sources listed below. Text from Wikipedia is available under CC BY-SA 4.0; this entry is adapted from it.

Spotted an error? Know more?

Reader corrections go straight into our review queue. Suggest an edit · How this site is sourced

Comments

Loading…
Open in the interactive codex →