Bayesian inference

Lore & Background

Bayesian inference is a method of statistical inference that applies Bayes' theorem to calculate the probability of a hypothesis, incorporating prior evidence and updating this probability as new data become available. The defining characteristic of this approach is the use of a prior distribution to estimate posterior probabilities, which are derived from two antecedents: a prior probability and a likelihood function. The likelihood function, obtained from a statistical model for observed data, indicates the compatibility of the evidence with a given hypothesis. The posterior probability is proportional to the product of the prior probability and the likelihood, divided by the marginal likelihood, which remains constant across competing hypotheses. This framework allows for the dynamic analysis of sequential data through Bayesian updating. The method has a wide range of applications, spanning science, engineering, philosophy, medicine, sport, psychology, and law. In the philosophy of decision theory, it is closely linked to subjective probability, often termed "Bayesian probability." Notably, while Bayesian updating is computationally convenient and widely used, it is not the only rational updating rule; non-Bayesian rules, such as those discussed in probability kinematics, can also avoid logical inconsistencies like Dutch books. The additional assumptions required to uniquely mandate Bayesian updating have been considered substantial and unsatisfactory by some scholars.

Reader's Guide

Bayesian inference is significant as a foundational technique in statistics, particularly in mathematical statistics, enabling dynamic analysis of sequential data through Bayesian updating. Its application spans diverse fields including science, engineering, philosophy, medicine, sport, psychology, and law. In the philosophy of decision theory, it is closely related to subjective probability, often called Bayesian probability. The method formalizes how prior beliefs are updated with new evidence, with the posterior probability proportional to the prior probability multiplied by the likelihood. The marginal likelihood, or model evidence, is constant across hypotheses and does not affect relative probabilities.

Did You Know?

• Bayesian inference uses a prior distribution to estimate posterior probabilities.
• Bayesian updating is particularly important in the dynamic analysis of a sequence of data.
• The likelihood function indicates the compatibility of evidence with a given hypothesis.
• The marginal likelihood is the same for all possible hypotheses being considered.

The Philosophical Foundation

Bayesian probability reinterprets what it means to assign a number to an uncertain event. Rather than treating probability as a long-run frequency or an inherent physical propensity, the Bayesian view frames it as a measure of reasonable expectation—a quantification of what one knows or believes given available information. This makes it, in essence, an extension of propositional logic into the realm of hypotheses: propositions whose truth or falsity remains unknown. Where a frequentist would test a hypothesis without attaching a probability to it (since the hypothesis is either true or false, yielding a probability of 0 or 1), a Bayesian assigns a continuous value between 0 and 1 reflecting the degree of uncertainty. The framework belongs to the family of evidential probabilities, meaning that evaluating any hypothesis requires first specifying a prior probability, which is then updated into a posterior probability once new relevant evidence arrives. A standard set of procedures and formulae governs this calculation, giving the approach a rigorous mathematical backbone.

The Methodological Engine

Bayesian methodology rests on a few interlocking principles. First, all sources of uncertainty in a statistical model—whether arising from inherent randomness or from a simple lack of information—are represented through random variables or unknown quantities. Second, the analyst must determine a prior probability distribution that reflects whatever information is available before observing new data. Third, and most distinctively, Bayes' theorem is applied sequentially: as additional data arrive, the current posterior distribution is computed, and that posterior immediately becomes the prior for the next round of updating. This iterative cycle means the analysis naturally incorporates evidence as it accumulates, rather than requiring a single fixed procedure. The framework also distinguishes itself sharply from frequentist practice. For a frequentist, a hypothesis is a binary proposition—true or false—so its probability is strictly 0 or 1. In Bayesian statistics, because the truth value may be uncertain, the assigned probability can occupy any point on the continuum from 0 to 1, capturing the analyst's graded confidence.

A Long and Winding History

Early practitioners, following Laplace's principle of insufficient reason and using uniform priors, called their work inverse probability because it reasoned backward from observed effects to underlying causes. After the 1920s, this tradition was largely displaced by what became known as frequentist statistics. The adjective Bayesian itself only entered common usage in the 1950s, with Bayesianism and neo-Bayesianism coined in the 1960s.

Two Camps and a Modern Renaissance

Bayesian probability splits broadly into two interpretive camps. Objectivists treat probability as a logical extension: given the same knowledge, every rational agent—even a hypothetical robot—should arrive at the same probability, a position justified by Cox's theorem. Subjectivists, by contrast, see probability as encoding personal belief, where rationality and coherence permit a wide range of legitimate assignments, constrained by the Dutch book argument or by decision theory and de Finetti's theorem. The practical divergence between the two camps centers largely on how the prior probability is constructed. In the objectivist stream, analysis depends solely on the assumed model and the data, with no subjective decisions required; subjectivists deny that fully objective analysis is possible in the general case. The 1980s brought a dramatic surge in Bayesian research and applications, driven largely by the discovery of Markov chain Monte Carlo methods, which removed many computational obstacles, and by growing interest in complex, nonstandard problems. Today, while frequentist statistics still anchors much undergraduate teaching, Bayesian methods are widely accepted and actively used, particularly in machine learning.