Non-linear least squares
Last updated
A stationary point, such as a maximum or saddle point, will appear in the objective function between any two minima when multiple minima are present.[1] At a stationary point of the objective function, the normal equations matrix lacks positive definiteness.[1]
Key facts and history
If divergence occurs, a simple expedient is to reduce the length of the shift vector by a fraction.[2] The shift vector length may be successively halved until the objective function value is less than at the last iteration.[2] The Marquardt parameter modifies the normal equations by adding λ times the identity matrix.[3] Alternating variable search varies each parameter in turn to reduce the sum of squares.[2]
It is sometimes possible to convert a non-linear model into a linear one through a transformation.[4] Most iterative minimization algorithms rely on the basic assumption that a linear approximation is valid, particularly near the best estimator.[4] Determining the Marquardt parameter has been the subject of multiple proposed strategies.[5]
What the research sources say
The normal equations are more generally written with a weight matrix W when observations are correlated.[6] Generalized least squares can be applied for inference when a linear approximation holds, using the Linear Template Fit equations.[4] Although a line search could optimize the fraction f for shift-cutting, the need to recalculate the objective function for each trial makes stringent optimization not worthwhile.[2] Overly stringent optimization of the Marquardt parameter is wasteful, similar to the case with shift-cutting.[5]
The smallest singular value of the Jacobian can serve as the cut-off value for the Marquardt parameter.[5]
References
- Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: In the absence of round-off error and of experimental error in the independent variable the normal equations matrix would be singular (original not read by us) en.wikipedia.org
- Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: M.J. Box, D. Davies and W.H. Swann, Non-Linear optimisation Techniques, Oliver & Boyd, 1969 (original not read by us) en.wikipedia.org
- Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: This technique was proposed independently by Levenberg (1944), Girard (1958), Wynne (1959), Morrison (1960) and Marquardt (1963). Marquardt's name alone is used (original not read by us) en.wikipedia.org
- Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: Britzger, "The Linear Template Fit", Eur. Phys. J. C, 2022, p. 731 (original not read by us) en.wikipedia.org
- Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: C.L. Lawson and R.J. Hanson, Solving Least Squares Problems, Prentice–Hall, 1974; R. Fletcher, UKAEA Report AERE-R 6799, H.M. Stationery Office, 1971 (original not read by us) en.wikipedia.org
- Wikipedia, "Non-linear least squares" (CC BY-SA 4.0), which cites: This implies that the observations are uncorrelated. If the observations are correlated, the expression <math display="block">S = \sum_k \sum_j r_k W_{kj} r_j</ (original not read by us) en.wikipedia.org
Each numbered sentence above was checked against the source it cites before publication. Sentences citing Wikipedia adapt text from the English Wikipedia article Non-linear least squares (contributors), used under CC BY-SA 4.0; this entry is available under the same licence. Other sources were used for facts only; no text or images are copied from them.
More in Probability & Statistics
Sources
Compiled from Wikipedia and the sources listed below. Text from Wikipedia is available under CC BY-SA 4.0; this entry is adapted from it.
- Wikipedia: Non-linear least squares (CC BY-SA 4.0).
- Word definitions: the Codexery glossary, each quoted from its Wikipedia article.
Spotted an error? Know more?
Reader corrections go straight into our review queue. Suggest an edit · How this site is sourced