Akaike information criterion

Lore & Background

The Akaike information criterion (AIC) is an estimator of prediction error and thereby relative quality of statistical models for a given set of data. Given a collection of models for the data, AIC estimates the quality of each model, relative to each of the other models. Thus, AIC provides a means for model selection. AIC is founded on information theory. When a statistical model is used to represent the process that generated the data, the representation will almost never be exact; so some information will be lost by using the model to represent the process. AIC estimates the relative amount of information lost by a given model: the less information a model loses, the higher the quality of that model.

Reader's Guide

The Akaike information criterion (AIC) is a fundamental tool in statistical model selection, balancing goodness of fit with model simplicity to avoid overfitting. It is defined as AIC = 2k - 2ln(Ĵ), where k is the number of estimated parameters and Ĵ is the maximized likelihood. The preferred model among candidates is the one with the minimum AIC value. AIC is founded on information theory and estimates the relative information lost when a model represents the data-generating process. It is widely used for statistical inference and forms the basis of a paradigm for the foundations of statistics. AIC does not indicate absolute model quality, so validation of the selected model is recommended. It can replicate every statistical hypothesis test, as every test can be formulated as a comparison of models. AIC is valid for nested and non-nested models alike, unlike the likelihood-ratio test which is restricted to nested models.

Did You Know?

• AIC estimates the relative amount of information lost by a model; the less information lost, the higher the model quality.
• The quantity exp((AICmin − AICi)/2) can be interpreted as being proportional to the probability that the ith model minimizes the estimated information loss.
• AIC tells nothing about the absolute quality of a model, only the quality relative to other models.

Information-Theoretic Roots and the Problem of Lost Information

At its core, AIC rests on a deceptively simple observation from information theory: no statistical model perfectly captures the process that produced a dataset. When a model stands in for that unknown generative process, some information is inevitably lost. The goal, then, is not to find a perfect representation but to identify which candidate model discards the least information. In principle, if the true process were known, one could quantify this loss precisely using the Kullback–Leibler divergence between the true distribution and each candidate. In practice, the true process is never observed. Hirotugu Akaike demonstrated in 1974 that, despite this fundamental uncertainty, it remains possible to estimate how much more or less information one model loses relative to another. That relative estimate is what AIC delivers. The framework has since grown into a foundational paradigm in statistics, underpinning not just model comparison but broader questions of statistical inference. The criterion is named for Akaike, the Japanese statistician who formulated it, and its influence extends well beyond any single application domain.

The Formula and Its Balancing Act

The AIC value for any given model is computed as two times the number of estimated parameters minus two times the natural logarithm of the maximized likelihood. This compact expression encodes a deliberate tension. The likelihood term rewards a model for fitting the observed data well, while the parameter-count term imposes a penalty that grows as the model becomes more complex. This matters because adding parameters to a model almost invariably improves its apparent fit to the data, which means that without a complexity penalty the most elaborate model would always win. The penalty term exists precisely to counteract that tendency, steering the analyst away from overfitting. At the same time, a model that is too simple will underfit and lose substantial information. AIC sits between these two risks. When comparing a collection of candidate models, the one with the smallest AIC is preferred, since it represents the best compromise between capturing the data's structure and remaining parsimonious. The criterion thus operationalizes the classic trade-off between goodness of fit and model simplicity in a single, comparable number.

From Scores to Decisions: Practical Model Selection

In applied work, an analyst begins with a set of candidate models, computes each one's AIC, and identifies the minimum value. The differences between AIC values then carry probabilistic meaning: the quantity exp((AICmin − AICi)/2) is proportional to the probability that model i minimizes the estimated information loss. Consider three models with AIC values of 100, 102, and 110. The second model is roughly 0.368 times as probable as the best to minimize information loss, while the third drops to about 0.007, effectively eliminating it from serious consideration. At that point the analyst faces a genuine choice: collect additional data to sharpen the distinction between the top two, accept that the evidence is insufficient to pick one, or construct a weighted average of the two leading models with weights proportional to 1 and 0.368, then perform inference on that combined model. This relative-likelihood interpretation is closely related to the likelihood-ratio test, yet AIC carries a crucial advantage: it places no requirement that the compared models be nested, a restriction that limits the likelihood-ratio test.

What AIC Cannot Tell You and Where It Reaches

A critical limitation must be stated plainly: AIC is purely a relative measure. It says nothing about the absolute adequacy of any model in the candidate set. If every model under consideration fits the data poorly, AIC will still select the best among them without issuing any warning. For this reason, standard practice after an AIC-based selection calls for independent validation—examining whether residuals appear random, testing predictive accuracy, and otherwise checking that the chosen model genuinely captures the data-generating process. Additionally, the asymptotic validity of AIC means that with small sample sizes the estimate can be biased, and a corrected version, AICc, is often preferred. Beyond model selection, AIC's reach extends to hypothesis testing itself. Because any statistical hypothesis test can be recast as a comparison between two models, AIC can in principle replicate tests such as Student's t-test for comparing two population means. This universality, combined with its information-theoretic grounding, is what elevates AIC from a convenient scoring rule to a foundational tool in statistical inference.

Gallery