Maximum likelihood estimation

Lore & Background

Maximum likelihood estimation (MLE) models a set of observations as a random sample from an unknown joint probability distribution that is expressed in terms of a vector of parameters. The goal is to determine the parameter values for which the observed data have the highest joint probability. The likelihood function is the real-valued result of evaluating the joint density at the observed data sample; for independent random variables, this function becomes the product of univariate density functions. The maximum likelihood estimate is the specific point in the parameter space that maximizes this likelihood function. If the likelihood function is differentiable, the derivative test for finding maxima can be applied, and in some cases the first-order conditions can be solved analytically—for example, the ordinary least squares estimator for linear regression maximizes the likelihood when errors are assumed to be normally distributed with constant variance. In practice, it is often convenient to work with the natural logarithm of the likelihood function, called the log-likelihood, because the logarithm is monotonic and the maximum occurs at the same parameter value. For many models, no closed-form solution exists, and the MLE must be found via numerical optimization. A sufficient condition for the existence of an MLE is that the likelihood function be continuous over a compact parameter space; for an open space, the function may increase without reaching a supremum. The Hessian matrix of second-order partial derivatives is used to confirm whether a root of the likelihood equations is a local maximum, indicated by negative semi-definiteness. Most common probability distributions, particularly those in the exponential family, are logarithmically concave. From a Bayesian perspective, MLE is generally equivalent to maximum a posteriori estimation with a uniform prior. In frequentist inference, MLE is a special case of an extremum estimator. When the parameter space is restricted, constraints can be imposed using the method of substitution or the method of Lagrange multipliers, the latter allowing for a statistical test of the constraint’s validity. Nonparametric maximum likelihood estimation can be performed using the empirical likelihood.

Reader's Guide

Maximum likelihood estimation is a foundational technique in statistical inference. Its significance lies in its ability to provide a principled and flexible way to estimate parameters from data. The method is intuitive: it selects the parameter values that make the observed data most probable. In practice, the natural logarithm of the likelihood function is often used because it is monotonic and easier to work with. The method has connections to other inferential frameworks: from a Bayesian perspective, MLE is generally equivalent to maximum a posteriori estimation with a uniform prior; in frequentist inference, it is a special case of an extremum estimator. The ordinary least squares estimator for linear regression, under normal errors with equal variance, is an example of an MLE that can be solved analytically. The method's flexibility and intuitive appeal have made it a dominant tool across many fields of science and engineering.

Did You Know?

• Maximum likelihood estimation maximizes a likelihood function so that, under the assumed statistical model, the observed data is most probable.
• If the likelihood function is differentiable, the derivative test for finding maxima can be applied.
• From the perspective of Bayesian inference, MLE is generally equivalent to maximum a posteriori estimation with a prior distribution that is uniform in the region of interest.
• In frequentist inference, MLE is a special case of an extremum estimator, with the objective function being the likelihood.