Likelihood function
Measures relative merit of models for given data.
The likelihood function, often simply called the likelihood, measures the relative merit of different statistical models for explaining a given set of observed data. These models are typically defined by one or more unknown parameters, often denoted as a single parameter or a vector of parameters. For a probability function or probability density function that describes the probability or density of data given a specific parameter value, the likelihood is any function of that parameter that is proportional to that probability or density. In maximum likelihood estimation, the parameter value that maximizes the likelihood function serves as a point estimate for the unknown parameter. The precision of this estimate is often indicated by the Fisher information, which can be approximated by the Hessian matrix of the likelihood at its maximum. In Bayesian statistics, the focus shifts to the posterior probability of the parameter given the data, which is derived from the likelihood using Bayes' rule. The likelihood function is defined differently for discrete and continuous distributions. For a discrete random variable, the likelihood of a parameter given an observed outcome is the probability mass function evaluated at that outcome, considered as a function of the parameter. For a continuous random variable, it is the probability density function evaluated at the observation, again as a function of the parameter. A common error is to interpret the likelihood as the probability that the parameter is true given the data; this is not correct and can lead to the prosecutor's fallacy. The likelihood does not integrate or sum to one over the parameter space. In measure-theoretic probability, the likelihood function is the Radon–Nikodym derivative of the probability distribution with respect to a dominating measure, interpreted as a function of the parameter. For likelihoods to be comparable, they must be derivatives with respect to the same dominating measure.
- field
- Statistics
- known_for
- Likelihood function, maximum likelihood estimation, Fisher information
Lore & Background
The likelihood function is defined as any function of a parameter θ equal to a positive constant times the probability (or probability density) of the observed data x given θ. For discrete probability distributions, the likelihood is the probability mass function evaluated at the observed outcome, considered as a function of the parameter rather than of the data. For continuous distributions, it is the probability density function similarly reoriented. The notation L(θ | x) is standard, though in the frequentist paradigm the notation L(θ ; x) or L(θ | x) is often used to emphasize that θ is regarded as a fixed unknown quantity, not a random variable being conditioned upon. A defining characteristic is that the likelihood function does not integrate or sum to one over the parameter space; for example, the integral of the likelihood for a coin’s probability of heads, given two heads in two flips, is 1/3 over the parameter’s range [0,1]. This distinguishes it from a probability density over θ. A common and potentially serious error is interpreting the likelihood as the probability that the parameter value is true given the observed data, which is a fallacy with consequences such as the prosecutor’s fallacy. In maximum likelihood estimation, the parameter value that maximizes the likelihood serves as a point estimate for the unknown parameter, and the Fisher information, often approximated by the Hessian matrix of the log-likelihood at that maximum, indicates the estimate’s precision. In Bayesian statistics, the posterior probability of the parameter given the data is computed via Bayes’ rule, which uses the likelihood as a key component. The likelihood function can be constructed for any distribution—discrete, continuous, or mixed—by using the Radon–Nikodym derivative of the probability distribution with respect to a common dominating measure, ensuring comparability across models.
Reader's Guide
The likelihood function serves as the basis for maximum likelihood estimation, where the parameter value that maximizes the likelihood is used as a point estimate for the unknown parameter. The Fisher information, often approximated by the Hessian matrix of the likelihood at its maximum, indicates the precision of that estimate. In Bayesian statistics, the likelihood is combined with a prior distribution via Bayes' rule to obtain the posterior probability of the parameter given the data. The likelihood itself does not give the probability that a parameter value is true; it only reflects the probability of observing the data under that parameter. This distinction is critical for correct statistical reasoning.
Did You Know?
- The likelihood function is defined as c times the probability of data given a parameter, for any positive constant c.
- In maximum likelihood estimation, the parameter that maximizes the likelihood serves as a point estimate.
- The Fisher information, often approximated by the likelihood's Hessian matrix at the maximum, indicates estimate precision.
- The likelihood function should not be confused with the posterior probability of the parameter given the data.
More in Probability And Stochastic Processes 1-21
Spotted an error? Know more?
This is a living reference — every entry is fact-audited, and reader corrections feed straight into our audit queue. Suggest an edit · See this site's audit record