Probability &
Statistics

A set is a collection of distinct objects, called its elements. We write $x \in A$ if x is an element of A, and $x \notin A$ otherwise. The basic operations on subsets A, B of a fixed universe S are:

Two sets are disjoint if $A \cap B = \varnothing$ (the empty set). A collection of sets $\{A_i\}$ is pairwise disjoint if $A_i \cap A_j = \varnothing$ whenever $i \ne j$, and forms a partition of S if, additionally, $\bigcup_i A_i = S$. De Morgan's laws relate union, intersection, and complement:

These set operations are the language in which every probabilistic statement about events (A3.4) is ultimately expressed.

• $\varnothing$the empty set, containing no elements.
• $A^c$the complement of A within the universe S.

A function $f : S \to T$ assigns to each $x \in S$ exactly one $f(x) \in T$. It is injective if distinct inputs give distinct outputs, surjective if every $y \in T$ is $f(x)$ for some x, and bijective if both — in which case it has a well-defined inverse $f^{-1}$.

A set is countable if it is finite or admits a bijection with $\mathbb{N} = \{1,2,3,\dots\}$; otherwise it is uncountable. The integers $\mathbb{Z}$ and rationals $\mathbb{Q}$ are countable, while the reals $\mathbb{R}$ are uncountable. This distinction is the reason probability theory must treat discrete random variables (countably many possible values, A3.13) and continuous random variables (uncountably many, typically all of $\mathbb{R}^d$) with genuinely different tools — sums versus integrals.

• $f^{-1}$inverse function, defined when f is bijective.
• countablefinite or in bijection with the natural numbers.

Elementary probability on finite sets reduces to counting. The number of ways to arrange n distinct objects in order (a permutation) is $n! = n(n-1)\cdots 1$, with $0! = 1$. The number of ordered selections of k objects from n (without replacement) is

The number of unordered selections of k from n, the binomial coefficient, is

These coefficients satisfy Pascal's rule $\binom{n}{k} = \binom{n-1}{k-1} + \binom{n-1}{k}$ and appear directly as the p.m.f. normalization of the Binomial distribution (A3.42).

• $n!$factorial; number of orderings of n distinct items.
• $\binom{n}{k}$"n choose k"; size of a k-element subset count.

An experiment with uncertain outcome is modeled by a sample space $\Omega$, the set of all possible outcomes. An event is a subset $A \subset \Omega$: the statement "event A occurs" means the realized outcome $\omega \in \Omega$ satisfies $\omega \in A$. The set operations of A3.1 translate directly into statements about events: $A \cup B$ is "A or B", $A \cap B$ is "A and B", $A^c$ is "not A", and disjoint events are mutually exclusive (cannot both occur).

Example: rolling a six-sided die has $\Omega = \{1,2,3,4,5,6\}$; the event "the roll is even" is the subset $A = \{2,4,6\}$.

• $\Omega$sample space; the set of all possible outcomes.
• eventa subset of $\Omega$ whose occurrence can be assigned a probability.

A probability measure on $\Omega$ assigns to each event A a number $\mathbb{P}(A) \in [0,1]$ satisfying the three Kolmogorov axioms:

• (P1) $\mathbb{P}(A) \ge 0$ for every event A;
• (P2) $\mathbb{P}(\Omega) = 1$;
• (P3) for pairwise disjoint events $A_1, A_2, \dots$ (finitely or countably many), $\mathbb{P}\bigl(\bigcup_i A_i\bigr) = \sum_i \mathbb{P}(A_i)$ (countable additivity).

From these three axioms alone, every other rule of probability can be derived. The pair $(\Omega, \mathbb{P})$ — or more precisely the triple including a σ-algebra of events, formalized in A3.61 — is called a probability space.

• $\mathbb{P}$a probability measure, satisfying (P1)–(P3).

Several familiar facts follow immediately from (P1)–(P3):

For events that are not disjoint, the inclusion–exclusion formula corrects for double-counting:

More generally, Boole's inequality (the union bound) gives, for any events $A_1,\dots,A_n$ regardless of overlap,

a simple but extremely useful bound that reappears in the union-bound arguments behind PAC generalization bounds (A3.119).

• union boundBoole's inequality; probability of a union is at most the sum of probabilities.

When $\Omega$ is finite and every outcome is equally likely (the classical or uniform model), the axioms force $\mathbb{P}(\{\omega\}) = 1/|\Omega|$ for each outcome, and hence for any event A,

the familiar "favorable outcomes over total outcomes" rule. Combined with the counting tools of A3.3, this is enough to compute probabilities for card games, dice, and sampling problems without any further machinery — but it only applies under the explicit assumption of a uniform model, which most real-world random phenomena do not satisfy.

• $|A|$cardinality (number of elements) of the finite set A.

The conditional probability of event A given that event B has occurred is

defined whenever $\mathbb{P}(B) > 0$. Intuitively, conditioning on B shrinks the sample space to B itself and renormalizes probabilities so that $\mathbb{P}(B\mid B) = 1$. For fixed B, the map $A \mapsto \mathbb{P}(A\mid B)$ is itself a valid probability measure, satisfying all three axioms of A3.5.

Rearranging the definition gives the multiplication rule $\mathbb{P}(A\cap B) = \mathbb{P}(A\mid B)\,\mathbb{P}(B) = \mathbb{P}(B\mid A)\,\mathbb{P}(A)$, which extends to any finite chain of events:

• $\mathbb{P}(A\mid B)$probability of A, recomputed on the reduced sample space B.

If $\{B_1,\dots,B_n\}$ partitions $\Omega$ (A3.1) with each $\mathbb{P}(B_i)>0$, then any event A can be decomposed along the partition:

This law of total probability is the discrete ancestor of the tower property of conditional expectation (A3.30) and is frequently the easiest route to computing a marginal probability when the problem naturally splits into cases.

• $\{B_i\}$a partition of $\Omega$ into mutually exclusive, exhaustive cases.

Combining the definition of conditional probability (A3.8) with the law of total probability (A3.9) gives Bayes' rule: for events A and a partition $\{B_i\}$,

This elementary form already contains the essential idea exploited throughout Bayesian statistics (A3.99): it inverts the direction of conditioning, turning "probability of evidence given hypothesis" into "probability of hypothesis given evidence." The general, density-based version for random variables is given in A3.32, with the Bayesian-inference terminology (prior, likelihood, posterior) introduced there.

• $\mathbb{P}(B_i\mid A)$the "inverted" conditional probability that Bayes' rule computes.

Events A and B are independent if knowing one occurred gives no information about the other:

equivalently $\mathbb{P}(A\mid B) = \mathbb{P}(A)$ when $\mathbb{P}(B)>0$. A collection $A_1,\dots,A_n$ is mutually independent if $\mathbb{P}\bigl(\bigcap_{i\in S} A_i\bigr) = \prod_{i\in S}\mathbb{P}(A_i)$ for every subset S — a strictly stronger condition than pairwise independence, which can hold even when mutual independence fails. Independence should not be confused with disjointness: independent events with positive probability are never disjoint.

• independentjoint probability factorizes into the product of individual probabilities.

Events A and B are conditionally independent given C (with $\mathbb{P}(C)>0$) if

Conditional independence neither implies nor is implied by unconditional independence: two events can become independent only after conditioning on a third (as in many causal structures and graphical models), or can be unconditionally independent yet dependent once a common cause is revealed. This notion, applied to random variables rather than events, underlies the factorization structure of Markov chains (A3.107) and graphical models more broadly.

• $A \perp B \mid C$common notation for "A and B are conditionally independent given C."

A random variable is a function $X : \Omega \to R_X$ assigning a numerical value (or, more generally, a value in some range space $R_X$) to every outcome of the underlying experiment. Capital letters (X, Y, Z) denote random variables; lowercase letters (x, y, z) denote specific realized values. The bracket notation $[X=x]$ or $[X\in A]$ refers to the underlying event in $\Omega$: $[X=x] = \{\omega\in\Omega : X(\omega)=x\}$, so that $\mathbb{P}(X=x) := \mathbb{P}([X=x])$.

A random variable is discrete if its range $R_X$ is finite or countable (A3.2); it is continuous if it takes values in $\mathbb{R}^d$ and assigns probability zero to every single point. This section and the next develop the discrete case fully before turning to continuous random variables in Tier 3.

• $R_X$the range (set of possible values) of X.
• $[X=x]$the event in $\Omega$ on which X takes the value x.

For a discrete random variable X, the probability mass function (p.m.f.) is $p_X(x) = \mathbb{P}(X=x)$ for $x \in R_X$. It satisfies $p_X(x)\ge 0$ and $\sum_{x\in R_X} p_X(x) = 1$, and the probability of any event $[X\in A]$ is obtained by summing:

The p.m.f. completely determines the distribution of X: any question about probabilities involving X alone can be answered from $p_X$. The cumulative distribution function (c.d.f.) $F_X(x) = \mathbb{P}(X\le x) = \sum_{x'\le x} p_X(x')$ is an equivalent, non-decreasing, right-continuous encoding of the same information, and generalizes directly to the continuous case (A3.43).

• $p_X$p.m.f.; $p_X(x)=\mathbb{P}(X=x)$, sums to 1 over $R_X$.
• $F_X$c.d.f.; $F_X(x) = \mathbb{P}(X\le x)$.

The expectation (or mean) of a discrete real-valued random variable X is the probability-weighted average of its possible values:

defined whenever the sum converges absolutely (i.e. $\sum_x |x|\,p_X(x) < \infty$). More generally, for any function $\phi: R_X \to \mathbb{R}$, the expectation of $\phi(X)$ is $\mathbb{E}(\phi(X)) = \sum_{x} \phi(x)\,p_X(x)$ — there is no need to first compute the distribution of $\phi(X)$ separately, a fact sometimes called the law of the unconscious statistician. Expectation is the single most important summary of a random variable's distribution and is linear: $\mathbb{E}(aX+bY) = a\,\mathbb{E}(X) + b\,\mathbb{E}(Y)$ for constants a, b.

• $\mathbb{E}(X)$expectation; the probability-weighted average value of X.

The simplest non-trivial discrete distribution is the Bernoulli distribution $\operatorname{Ber}(p)$: a random variable on $\{0,1\}$ with $\mathbb{P}(X{=}1)=p$ and $\mathbb{P}(X{=}0)=1-p$, modeling a single coin flip or yes/no trial. Its mean and variance are $\mathbb{E}(X)=p$ and $\operatorname{Var}(X) = p(1-p)$ (variance is formally defined in A3.27, but is computed here directly from the definition).

Summing N independent $\operatorname{Ber}(p)$ trials gives the Binomial distribution $\operatorname{Bin}(N,p)$, counting the number of successes, with p.m.f.

using the binomial coefficients of A3.3; $\mathbb{E}(X) = Np$ and $\operatorname{Var}(X) = Np(1-p)$.

• $\operatorname{Ber}(p)$single trial with success probability p.
• $\operatorname{Bin}(N,p)$number of successes in N independent Bernoulli trials.

The categorical distribution generalizes Bernoulli to K outcomes with probabilities $p_1,\dots,p_K$ summing to 1; it is the basic model for multi-class classification labels. The geometric distribution on $\{1,2,3,\dots\}$ models the number of trials until the first success in repeated independent $\operatorname{Ber}(p)$ trials:

The Poisson distribution $\operatorname{Pois}(\lambda)$ on $\{0,1,2,\dots\}$ models counts of rare independent events in a fixed interval:

and arises as the limit of $\operatorname{Bin}(N,\lambda/N)$ as $N\to\infty$ with $\lambda$ fixed — the rigorous statement of the "law of rare events."

• $\operatorname{Pois}(\lambda)$Poisson distribution; mean equals variance equals rate $\lambda$.

For two discrete random variables X, Y on the same probability space, the joint p.m.f. is $p_{X,Y}(x,y) = \mathbb{P}(X{=}x, Y{=}y)$, satisfying $\sum_{x,y} p_{X,Y}(x,y)=1$. The individual ("marginal") p.m.f.s are recovered by summing out the other variable:

This generalizes to any finite collection of discrete random variables, and the joint p.m.f. always contains strictly more information than the two marginals separately — the marginals alone cannot recover the joint without an independence assumption (A3.20).

• $p_{X,Y}$joint p.m.f. of the pair (X,Y).

If $X : \Omega \to R_X$ and $Y : \Omega \to R_Y$ are discrete random variables, the conditional probability of Y given X is

defined whenever $\mathbb{P}(X=x) > 0$, and left undefined otherwise. For fixed x, the map $y \mapsto \mathbb{P}(Y=y\mid X=x)$ is itself a valid p.m.f. on $R_Y$, called the conditional p.m.f. of Y given $X=x$. This is the elementary building block from which conditional expectation (A3.20), conditional densities for continuous variables (A3.47), and ultimately the general measure-theoretic definition (A3.86) are all derived.

• $R_X, R_Y$the (discrete) range spaces of X and Y.

X and Y are independent if $p_{X,Y}(x,y) = p_X(x)\,p_Y(y)$ for all x, y — the discrete analogue of A3.11 applied to random variables rather than events. Under independence, conditioning has no effect: $\mathbb{P}(Y=y\mid X=x) = p_Y(y)$.

For scalar- or vector-valued discrete Y, the conditional expectation of Y given X, denoted $\mathbb{E}(Y \mid X)$, is the random variable

defined for all $\omega$ with $\mathbb{P}(X = X(\omega)) > 0$. Although defined on $\Omega$, it depends on $\omega$ only through $X(\omega)$, justifying the function notation $\mathbb{E}(Y\mid X{=}x) = \sum_y y\,\mathbb{P}(Y{=}y\mid X{=}x)$, a genuine function on $R_X$ (the "regression function").

• $\mathbb{E}(Y\mid X)$a random variable on $\Omega$, constant on the fibers of X.
• $\mathbb{E}(Y\mid X{=}x)$a deterministic function of x; the "regression function."

The discrete conditional expectation of A3.20 satisfies two properties that will serve as the defining axioms in the general measure-theoretic case (A3.87):

• (CE1) $\mathbb{E}(Y\mid X)(\omega)$ depends on $\omega$ only through $X(\omega)$.
• (CE2) For every bounded $f : R_X \to [0,+\infty)$: $\mathbb{E}\bigl(\mathbb{E}(Y\mid X)\, f(X)\bigr) = \mathbb{E}(Y f(X))$.

Taking $f\equiv 1$ in (CE2) gives the tower property $\mathbb{E}(\mathbb{E}(Y\mid X)) = \mathbb{E}(Y)$: the unconditional mean equals the mean of the conditional mean. A second consequence, "taking out what is known," states that any function of X already known can be pulled outside a conditional expectation given X:

• tower propertythe unconditional mean equals the mean of the conditional mean.

If X and Y are independent discrete random variables on $\mathbb{Z}$, the p.m.f. of their sum $Z = X+Y$ is given by the convolution of the individual p.m.f.s:

Convolution is the reason the Binomial distribution (A3.16) arises as the sum of i.i.d. Bernoullis, and the same identity, with sums replaced by integrals, governs sums of independent continuous random variables (A3.49). It is also the combinatorial seed of the Central Limit Theorem (A3.74): repeated convolution of a fixed distribution with itself smooths toward a Gaussian shape.

• convolutionthe operation combining two p.m.f.s (or densities) to give the distribution of their sum.

The indicator of an event A is the random variable $\mathbf{1}_A(\omega) = 1$ if $\omega\in A$ and 0 otherwise; it satisfies $\mathbb{E}(\mathbf{1}_A) = \mathbb{P}(A)$, turning probabilities into expectations. This is more than notational convenience: combined with linearity of expectation (A3.15), it gives an extremely powerful technique for computing expected counts without ever finding the distribution of the count itself.

For instance, if $N = \sum_{i=1}^n \mathbf{1}_{A_i}$ counts how many of $A_1,\dots,A_n$ occur,

regardless of any dependence between the $A_i$. This "linearity trick" underlies many combinatorial expectation calculations and is a precursor to the Monte Carlo estimators of A3.114.

• $\mathbf{1}_A$indicator function of event A; equals 1 on A, 0 elsewhere.

Discrete random variables X and Y are conditionally independent given Z if, for every z with $\mathbb{P}(Z{=}z)>0$,

This extends A3.12 from events to random variables and is the formal statement behind the graphical-model notion that two variables are "screened off" from each other by a third. It is the building block of the Markov property (A3.107), where each future state is conditionally independent of the full past given only the present.

• $X\perp Y \mid Z$notation for X and Y conditionally independent given Z.

All random variables in this reference are assumed defined on a fixed probability space $(\Omega, \mathbb{P})$, taken large enough to carry all the randomness needed to generate any required collection of variables, implicitly enlarged whenever necessary. A random vector $X = (X_1,\dots,X_d)$ is simply a measurable map $\Omega \to \mathbb{R}^d$; all the constructions above (joint p.m.f., expectation, conditioning) extend coordinatewise or jointly without change.

Training data is an important notational exception: a realization is written $T = (x_1,\dots,x_N)$, an ordered sample, while $T=(X_1,\dots,X_N)$ denotes the i.i.d. random variables generating it (i.i.d. samples are formalized in A3.55), so that $T = (X_1(\omega),\dots,X_N(\omega)) = T(\omega)$ for some $\omega \in \Omega$. Greek-letter random vectors are set in boldface (e.g. $\boldsymbol{\alpha}, \boldsymbol{\beta}$).

• $T=(x_1,\dots,x_N)$a realized training sample; lowercase, fixed once observed.
• i.i.d.independent and identically distributed.

The k-th moment of a real random variable X is $\mathbb{E}(X^k)$, when it exists; the first moment is the mean. The k-th central moment is $\mathbb{E}\bigl[(X-\mathbb{E}X)^k\bigr]$. Moments of increasing order capture progressively finer features of a distribution's shape: the third standardized central moment is the skewness (asymmetry), and the fourth is the kurtosis (tail heaviness relative to a Gaussian). Not every distribution has finite moments of every order — heavy-tailed distributions may have infinite variance or even infinite mean, which is the reason careful statements of the Law of Large Numbers (A3.72) and CLT (A3.74) include explicit moment-existence hypotheses.

• $\mathbb{E}(X^k)$the k-th (raw) moment of X.

The variance of a real random variable X is the second central moment, measuring the typical squared spread around the mean:

The second equality (expand the square and use linearity of expectation) is the standard computational shortcut. Variance is always non-negative, equals zero iff X is a.s. constant, and scales quadratically: $\operatorname{Var}(aX+b) = a^2\operatorname{Var}(X)$. The standard deviation $\sigma(X)$ has the same units as X itself and is the more interpretable quantity for reporting spread.

• $\operatorname{Var}(X)$variance; expected squared deviation from the mean.
• $\sigma(X)$standard deviation; square root of variance.

For two real random variables, the covariance measures their joint linear variation:

Covariance is bilinear and satisfies $\operatorname{Var}(X+Y) = \operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y)$, which reduces to simple additivity when X, Y are uncorrelated. The correlation coefficient normalizes covariance to a scale-free quantity in $[-1,1]$:

Independence (A3.20) implies $\operatorname{Cov}(X,Y)=0$ (variables are then called uncorrelated), but the converse is false in general: uncorrelated variables can still be dependent through non-linear relationships.

• $\operatorname{Cov}(X,Y)$covariance; bilinear measure of joint linear variation.
• $\rho(X,Y)$correlation; covariance normalized to $[-1,1]$.

For a random vector $X \in \mathbb{R}^d$, the covariance matrix collects all pairwise covariances between coordinates:

The covariance matrix is always symmetric and positive semi-definite ($v^T\Sigma v = \operatorname{Var}(v^TX) \ge 0$ for every vector v); it is positive definite unless some linear combination of the coordinates is a.s. constant. Under an affine transformation $Y=AX+b$, the covariance transforms as $\operatorname{Cov}(Y) = A\Sigma A^T$ — a fact used repeatedly when working with Gaussian vectors (A3.111).

• $\Sigma$covariance matrix; $\Sigma_{ij} = \operatorname{Cov}(X_i,X_j)$.
• $S_+^d$the cone of $d\times d$ symmetric positive semi-definite matrices.

Beyond the basic tower property of A3.21, conditioning interacts with variance through the law of total variance (also called the variance decomposition):

The first term is the average within-group variance (how spread out Y is once X is known) and the second is the variance between groups (how much the conditional mean itself varies with X). This identity is the probabilistic ancestor of the analysis-of-variance (ANOVA) decomposition in statistics, and reappears, in a more general form, when discussing the bias–variance decomposition for estimators (A3.79).

• law of total variancetotal variance splits into within-group and between-group components.

Not every random variable of interest is discrete. A random variable $X : \Omega \to \mathbb{R}^d$ is continuous with probability density function (p.d.f.) f if

for every bounded continuous $\phi : \mathbb{R}^d \to [0,+\infty)$; taking $\phi = \mathbf{1}_A$ (approximated by continuous functions) gives $\mathbb{P}(X\in A) = \int_A f(x)\,dx$. Necessarily $f\ge 0$ and $\int_{\mathbb{R}^d} f(x)\,dx = 1$, the continuous analogue of the p.m.f. normalization (A3.14). A key qualitative difference from the discrete case: $\mathbb{P}(X=x) = 0$ for every individual point x, so the density itself is not a probability — only its integral over a region is.

• $f$, $\phi_X$p.d.f. of X; non-negative, integrates to 1, but is not itself a probability.

For any real random variable X (discrete, continuous, or neither), the c.d.f. $F_X(x) = \mathbb{P}(X\le x)$ is defined unconditionally and is always non-decreasing, right-continuous, with $F_X(-\infty)=0$, $F_X(+\infty)=1$. When X has p.d.f. f, $F_X(x) = \int_{-\infty}^x f(t)\,dt$ and, where $F_X$ is differentiable, $f(x) = F_X'(x)$ — recovering the density from the c.d.f. by differentiation. This is the most general scalar description of a distribution, since $F_X$ always exists and uniquely determines $\mathbb{P}_X$ even when no density exists (A3.96).

• $F_X$c.d.f.; always defined, monotone, right-continuous.

For continuous X with density f, expectation and moments are defined exactly as in A3.31 with $\phi(x)=x$ or $\phi(x)=x^k$:

provided the integral converges absolutely. All the properties of expectation, variance, covariance, and the covariance matrix established for discrete random variables (A3.15, A3.27–A3.29) carry over verbatim — linearity, the variance shortcut formula, bilinearity of covariance — since none of those derivations used discreteness, only linearity and additivity of the underlying integral/sum.

• $\mathbb{E}(X)$mean of a continuous random variable; an integral against its density.

The simplest continuous distribution is the uniform distribution $\operatorname{Unif}(a,b)$ on an interval $[a,b]$, with constant density

It is the continuous analogue of the classical equally-likely model (A3.7): every sub-interval of equal length is equally probable. The uniform distribution on $[0,1]$ plays a special generative role, since any continuous distribution can be simulated from it via the inverse-transform method (A3.44).

• $\operatorname{Unif}(a,b)$uniform on $[a,b]$; constant density.

The exponential distribution $\operatorname{Exp}(\lambda)$ on $[0,\infty)$ models waiting times between rare events, with density and moments

It is the continuous-time analogue of the geometric distribution (A3.17) and is the unique continuous distribution with the memoryless property: $\mathbb{P}(X > s+t \mid X>s) = \mathbb{P}(X>t)$ for all $s,t\ge 0$ — the remaining waiting time does not depend on how long one has already waited. It belongs to the exponential family (A3.41) and is the natural waiting-time distribution underlying the Poisson process.

• $\operatorname{Exp}(\lambda)$exponential with rate $\lambda$; memoryless.

The univariate Gaussian (or normal) distribution $\mathcal{N}(\mu,\sigma^2)$, by far the most important continuous distribution in statistics, has density

The standard normal is $\mathcal{N}(0,1)$, often denoted with density $\varphi(z)$ and c.d.f. $\Phi(z)$; any Gaussian can be standardized via $Z = (X-\mu)/\sigma \sim \mathcal{N}(0,1)$. The Gaussian's ubiquity is not an accident of convenience but a consequence of the Central Limit Theorem (A3.74), which shows it arises as the universal limiting shape for sums of many small, independent contributions.

• $\mathcal{N}(\mu,\sigma^2)$Gaussian with mean $\mu$, variance $\sigma^2$.
• $\Phi$standard normal c.d.f.

For $p\in(0,1)$, the p-th quantile of a random variable X is any $q_p$ satisfying $\mathbb{P}(X\le q_p)\ge p$ and $\mathbb{P}(X\ge q_p)\ge 1-p$; when $F_X$ is continuous and strictly increasing, this is simply $q_p = F_X^{-1}(p)$. The 50th-percentile quantile is the median. Quantiles of the standard normal, denoted $z_p = \Phi^{-1}(p)$, appear directly in the construction of confidence intervals (A3.103) and hypothesis-testing rejection regions (A3.102).

• $q_p$p-th quantile; generalized inverse of the c.d.f.
• $z_p$p-th quantile of the standard normal.

Two continuous random variables X, Y are jointly continuous with joint density $\phi_{X,Y}$ if

for measurable $A \subset \mathbb{R}^2$, with $\phi_{X,Y}\ge 0$ and $\iint \phi_{X,Y}=1$ — the direct two-dimensional analogue of A3.31. The construction extends to any finite-dimensional random vector. Importantly, each coordinate being individually continuous (having a marginal density) does not by itself imply the pair is jointly continuous; joint continuity is a strictly stronger condition.

• $\phi_{X,Y}$joint density of the pair (X,Y).

For jointly continuous X, Y with joint p.d.f. $\phi_{X,Y}$, the marginal density of X is obtained by integrating out Y:

X and Y are independent if their joint density factorizes, $\phi_{X,Y}(x,y) = \phi_X(x)\,\phi_Y(y)$ for all x, y; equivalently, $\mathbb{P}(X\in A, Y \in B) = \mathbb{P}(X\in A)\,\mathbb{P}(Y\in B)$ for all measurable A, B, mirroring A3.20. As in the discrete case, independence implies conditioning has no effect: $\phi_Y(y\mid X=x) = \phi_Y(y)$ (the conditional density is introduced formally in A3.47).

• $\phi_X(x)$marginal density of X; obtained by integrating the joint density over Y.

Many random variables are neither purely discrete nor purely continuous: a typical example is the payout of an insurance claim, which is 0 with positive probability (no claim filed) but continuously distributed given a claim occurs. Such variables are handled by writing the distribution as a mixture,

combining a discrete component (governed by a p.m.f.) and a continuous component (governed by a density) with mixture weight p. This ad hoc construction is unified rigorously by the Radon–Nikodym framework (A3.83), under which p.m.f.s and p.d.f.s are simply densities with respect to different reference measures, and mixed variables are densities with respect to a sum of the two.

• mixturea distribution combining discrete atoms and a continuous density.

Many of the named distributions above belong to a single unifying family. The exponential family has canonical form

where $\theta$ is the natural parameter, $t(x)$ the sufficient statistic (formalized in A3.97), and $A(\theta)$ the log-partition function ensuring normalization. Bernoulli, Binomial, Poisson, Gaussian, Exponential, Gamma, and Beta all belong to this family. Differentiating the normalization identity $\int h(x)e^{\theta^Tt(x)-A(\theta)}dx=1$ gives the useful moment identity $\nabla A(\theta) = \mathbb{E}_\theta[t(X)]$; the family's closure under sufficient-statistic aggregation gives it favorable conjugacy properties for Bayesian inference (A3.116).

• $A(\theta)$log-partition (cumulant) function; $\nabla A(\theta) = \mathbb{E}_\theta[t(X)]$.

If X has density $f_X$ and $Y=g(X)$ for a strictly monotone, differentiable g, the density of Y follows from the c.d.f. method: $F_Y(y) = \mathbb{P}(g(X)\le y) = F_X(g^{-1}(y))$ (or $1-F_X(g^{-1}(y))$ if g is decreasing), and differentiating gives

This one-dimensional change-of-variables formula is the scalar precursor to the general multivariate Jacobian formula of A3.110, and is the basic tool for deriving the distribution of transformed data — e.g. showing that $e^X$ for $X\sim\mathcal{N}(\mu,\sigma^2)$ is log-normal.

• $g^{-1}$inverse of the (monotone) transformation g.

If X has continuous, strictly increasing c.d.f. $F_X$, then $U = F_X(X)$ is distributed $\operatorname{Unif}(0,1)$ — every continuous distribution, once passed through its own c.d.f., becomes uniform. This is the probability integral transform, and it underlies two practical tools: simulating any continuous distribution from uniform random numbers via the inverse method (A3.44), and goodness-of-fit testing, where deviations of transformed data from uniformity reveal model misspecification.

• $F_X(X)$always $\operatorname{Unif}(0,1)$ when $F_X$ is continuous and strictly increasing.

Reversing the probability integral transform of A3.43: if $U \sim \operatorname{Unif}(0,1)$ and $F_X^{-1}$ is the quantile function of a target distribution, then $X = F_X^{-1}(U)$ has exactly that target distribution. This inverse-transform sampling method reduces simulating from any distribution with a computable quantile function to simulating a single uniform random number — the conceptual starting point for the more elaborate sampling schemes (rejection sampling, importance sampling A3.108, MCMC A3.106) needed when the quantile function is not directly available, as is typical for Bayesian posteriors.

• $F_X^{-1}(U)$inverse-transform sample; has distribution $F_X$ when U is uniform.

If X and Y are jointly continuous with p.d.f. $\phi_{X,Y}$, the conditional p.d.f. of Y given X is

provided the denominator (the marginal density of X at x) does not vanish — the exact continuous analogue of the discrete conditional p.m.f. of A3.19, with the marginal sum replaced by a marginal integral. For fixed x, $y\mapsto \phi_Y(y\mid X{=}x)$ is itself a valid density on $R_Y$.

• $\phi_Y(y\mid X{=}x)$conditional density, normalized by the marginal of X.

Writing $\phi_Y(y\mid X)(\omega) = \phi_Y(y \mid X = X(\omega))$, the conditional expectation of continuous Y given continuous X is defined exactly as in the discrete case (A3.20), with the sum replaced by an integral:

As before, this is a random variable depending on $\omega$ only through $X(\omega)$, and the corresponding deterministic function $x \mapsto \mathbb{E}(Y\mid X{=}x)$ is the regression function in the continuous setting — the central object that supervised learning with squared loss aims to estimate (formalized fully in A3.91).

• $\mathbb{E}(Y\mid X{=}x)$continuous regression function; an integral against the conditional density.

Both definitions above (discrete A3.20 and continuous A3.46) satisfy the two properties (CE1)–(CE2) introduced in A3.21, repeated here as they will serve as the defining axioms in the fully general case (A3.87):

• (CE1) $\mathbb{E}(Y\mid X)(\omega)$ depends on $\omega$ only through $X(\omega)$.
• (CE2) For every bounded continuous (or non-negative measurable) $f : R_X \to [0,+\infty)$: $\mathbb{E}\bigl(\mathbb{E}(Y\mid X)\, f(X)\bigr) = \mathbb{E}(Y f(X))$.

One can show that, in both the discrete and continuous settings, the function g satisfying these two properties is unique (a.e.) and coincides with the formulas above. When the joint density $\phi_{X,Y}$ need not be continuous, the identity defining g holds everywhere except possibly on a negligible set — motivating the measure-theoretic treatment of A3.78 onward.

• (CE1)–(CE2)the two axioms uniquely characterizing conditional expectation.

Combining the definition of conditional density with the symmetry of the joint distribution yields the density form of Bayes' rule, generalizing both the elementary event version (A3.10) and the discrete version implicit in A3.19:

In the Bayesian inference setting central to machine learning (A3.99), X plays the role of an unknown parameter and Y the role of observed data, giving rise to the terminology prior $\phi_X(x)$, likelihood $\phi_Y(y\mid X{=}x)$, evidence $\phi_Y(y)$, and posterior $\phi_X(x\mid Y{=}y)$.

• evidencethe normalizing constant $\phi_Y(y) = \int \phi_Y(y\mid X{=}x)\phi_X(x)\,dx$.

If X and Y are independent continuous random variables with densities $f_X, f_Y$, the density of $Z=X+Y$ is given by the continuous convolution, the direct analogue of A3.22:

Two named distributions are convolution-closed in a way that is especially useful: the sum of independent Gaussians is again Gaussian (with means and variances adding), and the sum of independent Gamma random variables with the same rate is again Gamma — both facts are most easily proved using moment generating functions (A3.52) rather than direct integration.

• convolutionintegral formula for the density of a sum of independent variables.

Given i.i.d. continuous $X_1,\dots,X_N$, the order statistics $X_{(1)} \le X_{(2)} \le \cdots \le X_{(N)}$ are the same values sorted in increasing order. The minimum $X_{(1)}$ and maximum $X_{(N)}$ have c.d.f.s

where F is the common c.d.f. — both following immediately from independence. Order statistics underlie the sample median, sample quantiles, and extreme-value theory, and the maximum/minimum's distribution is the starting point for analyzing the worst-case behavior of N i.i.d. draws, relevant to analyzing tail risk and rare-event simulation.

• $X_{(k)}$the k-th smallest value among $X_1,\dots,X_N$.

Continuous random variables X, Y are conditionally independent given Z if their joint conditional density factorizes:

This extends both A3.12 (events) and A3.24 (discrete random variables) to the continuous case, completing the general statement of conditional independence used throughout graphical models, Markov chains (A3.107), and hierarchical Bayesian models, where complex joint distributions are built by chaining together simple conditional-independence assumptions.

• $X\perp Y\mid Z$conditional independence, now stated at the level of densities.

The moment generating function (MGF) of X is $M_X(t) = \mathbb{E}(e^{tX})$, when finite in a neighborhood of $t=0$; differentiating under the expectation (justified rigorously in A3.95) and evaluating at $t=0$ recovers all moments, $M_X^{(k)}(0) = \mathbb{E}(X^k)$. The characteristic function $\varphi_X(t) = \mathbb{E}(e^{itX})$ always exists (no integrability conditions needed) and uniquely determines the distribution of X.

Both transform convolution into multiplication — for independent X, Y, $M_{X+Y}(t) = M_X(t)M_Y(t)$ — which is why they give the cleanest proofs that sums of independent Gaussians are Gaussian, sums of independent Poissons are Poisson, and are the standard technical tool used in proving the Central Limit Theorem (A3.74).

• $M_X(t)$MGF; $\mathbb{E}(e^{tX})$, generates moments via derivatives at 0.
• $\varphi_X(t)$characteristic function; $\mathbb{E}(e^{itX})$, always exists.

The Gamma distribution $\operatorname{Gamma}(\alpha,\beta)$ on $(0,\infty)$, with shape $\alpha$ and rate $\beta$, has density $f(x) \propto x^{\alpha-1}e^{-\beta x}$ and generalizes the Exponential (the case $\alpha=1$); it is the natural model for positive, right-skewed quantities like waiting times and variances. The Beta distribution $\operatorname{Beta}(\alpha,\beta)$ on $(0,1)$ has density

where $B(\alpha,\beta)$ is the Beta function normalizing the density. Both belong to the exponential family (A3.41); the Beta distribution is the canonical prior for a Bernoulli/Binomial success probability, central to the conjugate-prior calculations of A3.117.

• $\operatorname{Beta}(\alpha,\beta)$distribution on (0,1); conjugate prior for Bernoulli success probability.

A mixture distribution is built by first drawing a latent component label $Z\in\{1,\dots,K\}$ with $\mathbb{P}(Z{=}k)=\pi_k$, then drawing X from a component density $\phi_k$ depending on the chosen k. The resulting marginal density of X is a convex combination of the component densities:

Gaussian mixture models, with each $\phi_k = \mathcal{N}(\mu_k,\Sigma_k)$, are the canonical example, used for clustering and density estimation; the latent label Z is unobserved, and estimating mixture parameters from data on X alone is the prototypical use case for the EM algorithm (A3.115).

• $\pi_k$mixture weight; probability of the k-th component.
• $Z$latent (unobserved) component label.

Before formalizing measure theory, it is worth fixing the object that motivates much of it: a sequence $X_1,\dots,X_N$ is independent and identically distributed (i.i.d.) if the $X_i$ are mutually independent (A3.11, extended to random variables) and each has the same marginal distribution. Training data is almost always modeled this way, $T=(X_1,\dots,X_N)$, with realization $T=(x_1,\dots,x_N)$ as introduced in A3.25. Constructing the joint distribution of an i.i.d. sequence rigorously — as a product measure on an infinite product space — requires exactly the machinery built in A3.56–A3.71, which is why this tier is necessary even though its conclusions mostly confirm facts already used informally above.

• i.i.d.independent, identically distributed; the standard sampling assumption.

The informal treatment above silently assumed every subset of $\Omega$ or $\mathbb{R}^d$ can be assigned a probability or a length. This is false: for uncountable spaces like $\mathbb{R}$, there exist pathological subsets (constructed via the axiom of choice) to which no countably-additive, translation-invariant length can consistently be assigned. The resolution is to restrict attention to a well-behaved collection of "measurable" subsets — large enough to contain every set one could practically construct, but small enough to admit a consistent measure. This is the role of the σ-algebra (A3.57) and is the reason probability spaces are formally triples rather than pairs (A3.61).

• measurabilitythe restriction needed to avoid logical inconsistency on uncountable spaces.

A measurable space is a pair $(S,\mathcal{S})$ where $\mathcal{S} \subset \mathcal{P}(S)$ (the power set of S) contains S itself, is stable under complementation ($A \in \mathcal{S} \Rightarrow A^c = S\setminus A \in \mathcal{S}$), and stable under countable unions and intersections. Such a collection $\mathcal{S}$ is called a σ-algebra, and its elements are the measurable subsets of S. Compare to A3.1: a σ-algebra is exactly a collection of sets closed under the elementary set operations, but with closure required for countable (not just finite) combinations — the extra strength needed to support the countable additivity axiom (P3) of A3.5 in full generality.

• $\mathcal{S}$a σ-algebra; closed under complement and countable union/intersection.

Given any collection $\mathcal{C}$ of subsets of S, the σ-algebra generated by $\mathcal{C}$, written $\sigma(\mathcal{C})$, is the smallest σ-algebra containing $\mathcal{C}$ — concretely, the intersection of all σ-algebras containing $\mathcal{C}$ (an intersection of σ-algebras is always itself a σ-algebra). Throughout, measurable spaces will be Polish spaces — complete, separable metric spaces — equipped with their Borel σ-algebra $\mathcal{B}(S) = \sigma(\text{open subsets of } S)$, the smallest σ-algebra containing all open sets. This single construction is flexible enough to make every set arising in ordinary analysis (open, closed, countable unions/intersections thereof) measurable.

• $\sigma(\mathcal{C})$smallest σ-algebra containing $\mathcal{C}$.
• Polish spacea complete, separable metrizable space (always assumed here).

A function $f : S \to S'$ between measurable spaces $(S,\mathcal{S})$ and $(S',\mathcal{S}')$ is measurable if $f^{-1}(A') \in \mathcal{S}$ for every $A' \in \mathcal{S}'$; for $S' = \mathbb{R}^d$ with its Borel σ-algebra, this reduces to requiring that preimages of open sets be measurable, which (since open sets generate the Borel σ-algebra) is equivalent to requiring preimages of every Borel set be measurable. Compositions of measurable functions are measurable, and so are sums, products, and pointwise limits of sequences of real-valued measurable functions — closure properties essential for the construction of the integral in A3.63.

• measurable functionpreimages of measurable sets are measurable.

A (positive) measure $\mu$ on $(S,\mathcal{S})$ assigns to each $A \in \mathcal{S}$ a value $\mu(A) \in [0,+\infty]$, additive over countable disjoint unions — exactly axioms (P1) and (P3) of A3.5, but without the normalization $\mu(S)=1$ required of probability measures. A measure is finite if $\mu(S)<\infty$, and σ-finite if S can be covered by a countable union of measurable sets each of finite measure (the Lebesgue measure on $\mathbb{R}$ is σ-finite but not finite). All measures considered in this reference are assumed σ-finite, a mild technical condition that rules out pathological behavior in the Radon–Nikodym theorem (A3.83) and Fubini's theorem (A3.81).

• $\mu(A)$measure of the set A; non-negative, possibly infinite.
• σ-finiteS coverable by countably many finite-measure pieces.

A probability space is a triple $(\Omega, \mathcal{A}, \mathbb{P})$: $\mathcal{A}$ a σ-algebra on $\Omega$, and $\mathbb{P}$ a measure on $(\Omega,\mathcal{A})$ with $\mathbb{P}(\Omega)=1$. This is the formal completion of the informal pair $(\Omega,\mathbb{P})$ used throughout A3.1–A3.54: measurable subsets of $\Omega$ are called events (matching A3.4, now restricted to $\mathcal{A}$), and an event of probability one is said to occur almost surely (a.s.). Every construction so far — conditional probability, expectation, the axioms of A3.5 — was implicitly taking place inside such a triple, with the σ-algebra suppressed because, for discrete or nicely-behaved continuous spaces, it can usually be taken to be the full power set or the Borel σ-algebra without complication.

• $(\Omega,\mathcal{A},\mathbb{P})$a probability space; $\Omega$ outcomes, $\mathcal{A}$ events, $\mathbb{P}$ the measure.
• a.s."almost surely"; with probability 1.

Formally completing A3.13, a random variable X taking values in a measurable space $(R_X,\mathcal{S}_X)$ is a measurable map $X:\Omega\to R_X$; this guarantees $[X\in C] \in \mathcal{A}$ for every $C \in \mathcal{S}_X$, so that $\mathbb{P}(X\in C)$, also written $\mathbb{P}_X(C)$, is well defined. $\mathbb{P}_X$ is called the law or distribution of X — the measure-theoretic name for the object that A3.14 (p.m.f.) and A3.31 (p.d.f.) each implicitly describe in their respective settings. Measurability of X is precisely the condition needed for $\mathbb{P}(X\in C)$ to make sense as a probability at all.

• $\mathbb{P}_X$the law (pushforward distribution) of X on $R_X$.

A set $A$ is $\mu$-negligible if it is contained in some $B \in \mathcal{S}$ with $\mu(B)=0$; a property holds almost everywhere (a.e.) if the set where it fails is negligible — this generalizes "almost surely" (A3.61) from the special case $\mu=\mathbb{P}$. Countable unions of negligible sets are negligible, a fact used constantly to combine finitely or countably many "exceptional cases" into one.

The integral $\int_S f\, d\mu$ is built up in three stages: first for indicator functions, via $\int_S \mathbf{1}_A\, d\mu = \mu(A)$; then, by linearity, for "simple functions" (finite linear combinations of indicators); then for general non-negative measurable f, as the supremum of integrals of simple functions dominated by f. The integral of a signed $f$ is $\int f^+ d\mu - \int f^- d\mu$ when both terms are finite, in which case f is $\mu$-integrable.

• $\mu$-negligiblecontained in a measurable set of measure zero.
• a.e."almost everywhere"; outside a negligible set.

The Lebesgue measure $\mathcal{L}^d$ on $\mathbb{R}^d$ extends the familiar notion of length/area/volume; for it, $\int_{\mathbb{R}^d} f(x)\,\mathcal{L}^d(dx)$ coincides with the Riemann integral whenever the latter is defined, and we write it simply $\int_{\mathbb{R}^d} f(x)\,dx$ — retroactively justifying every density integral used in Tier 3. The counting measure $\mathrm{card}$ on a finite or countable set S assigns $\mathrm{card}(A) = |A|$, reducing integration to summation: $\int_S f\,d(\mathrm{card}) = \sum_{x\in S} f(x)$, retroactively justifying every p.m.f. sum used in Tier 2.

• $\mathcal{L}^d$Lebesgue measure on $\mathbb{R}^d$; extends the Riemann integral.
• $\mathrm{card}$counting measure; turns integrals into sums on discrete spaces.

Three classical theorems govern when limits can be exchanged with integrals — a question that arises constantly (e.g. when differentiating under the expectation, A3.95). The Monotone Convergence Theorem states that if $0\le f_n \uparrow f$ pointwise, then $\int f_n\,d\mu \to \int f\,d\mu$. Fatou's lemma gives the one-directional inequality $\int \liminf_n f_n\, d\mu \le \liminf_n \int f_n\,d\mu$ for non-negative $f_n$, with no convergence assumed. The Dominated Convergence Theorem states that if $f_n \to f$ pointwise (or a.e.) and $|f_n|\le g$ for some fixed integrable g, then $\int f_n\,d\mu \to \int f\,d\mu$. These three results, in increasing order of how much pointwise structure they exploit, are the standard toolkit for justifying every interchange of limit and integral used elsewhere in this reference.

• DCTDominated Convergence Theorem; the workhorse interchange-of-limits result.

The product measure $\mu_1 \otimes \mu_2$ on $S_1 \times S_2$ satisfies $(\mu_1\otimes\mu_2)(A_1\times A_2) = \mu_1(A_1)\,\mu_2(A_2)$, the rigorous foundation for the joint-density constructions of A3.18 and A3.38. Fubini's theorem justifies iterated integration: for $\mu_1\otimes\mu_2$-integrable f,

with the symmetric statement also holding. The construction extends to finite tensor products $\mu_1\otimes\cdots\otimes\mu_n = \bigotimes_{k=1}^n \mu_k$, which formally underlies the definition of i.i.d. samples (A3.55): the joint law of $(X_1,\dots,X_N)$ i.i.d. with common law $\mathbb{P}_X$ is exactly the product measure $\mathbb{P}_X^{\otimes N}$.

• $\mu_1\otimes\mu_2$product measure on a Cartesian product space.

For measures $\mu, \nu$ on $(S,\mathcal{S})$, $\nu$ is absolutely continuous with respect to $\mu$, written $\nu \ll \mu$, if

Intuitively, $\nu$ cannot place mass anywhere $\mu$ assigns zero measure. Absolute continuity is the precise condition under which a density of one measure with respect to another can exist, and is therefore the conceptual hinge connecting the abstract notion of "measure" to the concrete, computable notion of "density" used throughout Tiers 2–3.

• $\nu \ll \mu$$\nu$ is absolutely continuous w.r.t. $\mu$.

The Radon–Nikodym theorem states that, for σ-finite $\mu$ and finite $\nu$, $\nu \ll \mu$ if and only if $\nu$ has a density with respect to $\mu$: a $\mu$-integrable $\phi : S \to [0,+\infty)$ such that

This is the rigorous foundation for "p.d.f." (density w.r.t. Lebesgue measure, A3.31) and "p.m.f." (density w.r.t. counting measure, A3.14): both are instances of the Radon–Nikodym derivative $\phi = d\nu/d\mu$. The theorem also explains exactly when a random variable fails to have either: precisely when its law is not absolutely continuous with respect to either Lebesgue or counting measure (e.g. the mixed distributions of A3.40, or singular distributions with no density at all).

• $d\nu/d\mu$Radon–Nikodym derivative; the density of $\nu$ relative to $\mu$.

Putting A3.62, A3.64, and A3.68 together: X taking values in $\mathbb{R}^d$ has a p.d.f. precisely when $\mathbb{P}_X \ll \mathcal{L}^d$, the density being the Radon–Nikodym derivative $d\mathbb{P}_X/d\mathcal{L}^d$. A discrete X always has a p.m.f., since every discrete measure is absolutely continuous with respect to $\mathrm{card}$. This single statement subsumes the separate, informal density definitions given in A3.14 and A3.31, and clarifies precisely why "having a density" is a real condition that can fail — it is the condition $\mathbb{P}_X\ll\mathcal{L}^d$, which singular and mixed distributions (A3.40) violate.

• $d\mathbb{P}_X/d\mathcal{L}^d$the p.d.f. of X, identified as a Radon–Nikodym derivative.

A sequence of random variables $(X_n)$ can converge to X in several inequivalent senses, all relevant to asymptotic statistics:

• Almost sure convergence: $X_n \to X$ a.s. if $\mathbb{P}(\lim_n X_n = X) = 1$.
• Convergence in probability: $X_n \xrightarrow{\,p\,} X$ if $\forall \varepsilon>0,\ \mathbb{P}(|X_n - X|>\varepsilon)\to 0$.
• Convergence in $L^p$: $X_n \xrightarrow{L^p} X$ if $\mathbb{E}(|X_n - X|^p) \to 0$.
• Convergence in distribution: $X_n \xrightarrow{d} X$ if $\mathbb{E}(f(X_n)) \to \mathbb{E}(f(X))$ for all bounded continuous f.

These satisfy the implications a.s. $\Rightarrow$ in probability $\Rightarrow$ in distribution, and $L^p$ $\Rightarrow$ in probability; no other implications hold in general. Convergence in distribution is the weakest and is what the Central Limit Theorem (A3.74) delivers, while almost sure convergence is the strongest and is what the Strong Law of Large Numbers (A3.72) delivers.

• $\xrightarrow{\,p\,}$convergence in probability.
• $\xrightarrow{\,d\,}$convergence in distribution (weak convergence).

Two workhorse results let convergence statements be combined and transformed without redoing analysis from scratch. The continuous mapping theorem states that if $X_n \xrightarrow{d} X$ (or in probability, or a.s.) and g is continuous, then $g(X_n) \xrightarrow{d} g(X)$ in the corresponding mode. Slutsky's theorem handles combinations of two sequences converging in different modes: if $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{p} c$ (a constant), then

Together these two results are the standard mechanism for deriving the asymptotic distribution of complicated statistics — e.g. studentized test statistics, where a CLT-based numerator is divided by an estimated (and merely consistent) standard deviation — from the basic building blocks of the LLN (A3.72) and CLT (A3.74).

• Slutsky's theoremcombines distributional and probability convergence.

Let $X_1,X_2,\dots$ be i.i.d. with $\mathbb{E}|X_1|<\infty$ and mean $\mu$, and let $\bar X_N = \frac{1}{N}\sum_{i=1}^N X_i$. The Strong Law of Large Numbers (SLLN) states

The Weak Law (WLLN) gives the corresponding statement for convergence in probability, and follows easily from Chebyshev's inequality (A3.92) when $X_1$ has finite variance — a much shorter argument than the SLLN proof, which needs only a finite mean. The SLLN is the theoretical justification for using the empirical risk (sample average of a loss) as a proxy for the true (population) risk in machine learning (A3.78).

• $\bar X_N$the sample mean of N i.i.d. observations.

If $\sqrt{N}(\hat\theta_N - \theta) \xrightarrow{d} \mathcal{N}(0,\sigma^2)$ and g is differentiable at $\theta$ with $g'(\theta)\ne 0$, the delta method gives the asymptotic distribution of the transformed estimator:

The proof is a direct application of a first-order Taylor expansion of g around $\theta$ combined with Slutsky's theorem (A3.71). The delta method is the standard route to confidence intervals for non-linear functions of an estimator — e.g. obtaining a confidence interval for an odds ratio or a ratio of means from the asymptotic normality of the underlying parameter estimates.

• delta methodpropagates asymptotic normality through a smooth transformation.

Let $X_1,\dots,X_N$ be i.i.d. with mean $\mu$ and finite variance $\sigma^2$. The Central Limit Theorem (CLT) states that

The multivariate version, for i.i.d. $X_i \in \mathbb{R}^d$ with covariance $\Sigma$, states $\sqrt{N}(\bar X_N - \mu) \xrightarrow{d} \mathcal{N}(0,\Sigma)$. The CLT explains the ubiquity of the Gaussian distribution (A3.36) in statistics: it underlies asymptotic confidence intervals (A3.103), the construction of test statistics (A3.102), and the approximate normality of maximum likelihood estimators (A3.96) for large sample sizes.

• $\xrightarrow{\;d\;}$convergence in distribution.

The CLT (A3.74) describes a limit but says nothing about how fast it is approached for finite N. The Berry–Esseen theorem quantifies the rate: for i.i.d. $X_i$ with $\mathbb{E}|X_1|^3 < \infty$,

for a universal constant C. The $O(1/\sqrt N)$ rate explains why normal approximations can already be reasonably accurate at moderate sample sizes, but also why they degrade for heavily skewed or heavy-tailed data, motivating the finite-sample concentration inequalities of A3.93–A3.94 when a guaranteed (non-asymptotic) bound is required.

• Berry–Esseen ratethe $O(1/\sqrt N)$ speed of convergence in the CLT.

Given i.i.d. training data $T=(x_1,\dots,x_N)$ sampled from a distribution $\mathbb{P}_X$, the empirical distribution is the (random) discrete measure

where $\delta_{x_i}$ is the Dirac mass at $x_i$ (the measure with $\delta_{x_i}(A) = \mathbf{1}_{x_i\in A}$). By the SLLN (A3.72), for any fixed measurable g, $\mathbb{E}_{X\sim\widehat{\mathbb{P}}_N}[g(X)] \to \mathbb{E}_{X\sim\mathbb{P}_X}[g(X)]$ a.s. — the empirical distribution converges to the true distribution as more data arrives, in the precise sense of converging integrals against any fixed test function.

• $\widehat{\mathbb{P}}_N$empirical distribution; uniform measure on the observed sample.

A3.76 gives convergence for one fixed test function at a time. The Glivenko–Cantelli theorem strengthens this to a uniform statement for the empirical c.d.f. $\widehat F_N(x) = \widehat{\mathbb{P}}_N(X\le x)$:

The empirical c.d.f. converges to the true c.d.f. uniformly over all thresholds x simultaneously, not just pointwise. This is the simplest instance of a uniform law of large numbers, and is the prototype for the uniform convergence of empirical risk to population risk over an entire hypothesis class — the technical heart of PAC learning theory (A3.119).

• $\widehat F_N$empirical c.d.f.; converges to $F_X$ uniformly, a.s.

For a loss function $\ell(x,\theta)$ and parameter $\theta$, the population risk and empirical risk are

By the SLLN (A3.72), $\widehat{R}_N(\theta) \to R(\theta)$ a.s. for each fixed $\theta$; the uniform version over a class of $\theta$, governed by Glivenko–Cantelli-type results (A3.77) and complexity measures like VC dimension or Rademacher complexity (A3.118), is the subject of statistical learning theory, and underlies the principle of empirical risk minimization (ERM): choosing $\hat\theta = \arg\min_\theta \widehat{R}_N(\theta)$ as a proxy for minimizing $R(\theta)$.

• ERMempirical risk minimization; the core paradigm of supervised learning.

An estimator $\hat\theta = \hat\theta(X_1,\dots,X_N)$ of a parameter $\theta$ is any (measurable) function of the data, hence itself a random variable. Its bias and variance are

An estimator is unbiased if $\operatorname{Bias}(\hat\theta)=0$, and consistent if $\hat\theta \xrightarrow{p} \theta$ as $N \to \infty$. The mean squared error (MSE) decomposes as

the bias–variance decomposition, the central organizing principle for understanding generalization error and model complexity trade-offs in machine learning: flexible models tend to reduce bias at the cost of higher variance, and vice versa.

• $\operatorname{Bias}(\hat\theta)$systematic deviation of the estimator's mean from the true parameter.
• MSEmean squared error; decomposes into variance plus squared bias.

Before maximum likelihood (A3.96), the method of moments offers a simpler, often less efficient, estimation strategy: set the first k sample moments equal to the corresponding theoretical moments (as functions of $\theta\in\mathbb{R}^k$) and solve for $\theta$,

By the SLLN (A3.72), the resulting estimator is consistent whenever the moment-to-parameter map is continuous and invertible, since sample moments converge to population moments. Method-of-moments estimators are typically easy to compute in closed form (no optimization required) and serve as good initializations for the iterative algorithms — Newton's method, EM (A3.115) — used to compute the MLE.

• method of momentsmatches sample moments to theoretical moments to estimate $\theta$.

For general random variables, properties (CE1)–(CE2) of A3.47 become the definition. Let $X:\Omega\to R_X$, $Y:\Omega\to \mathbb{R}^d$ with $\mathbb{E}(|Y|)<\infty$.

Definition. The conditional expectation $\mathbb{E}(Y\mid X)$ is a random variable $Z:\Omega\to\mathbb{R}^d$ such that

• (i) there exists measurable $h:R_X\to\mathbb{R}^d$ with $Z = h\circ X$ a.s.;
• (ii) for every measurable $g:R_X\to[0,+\infty)$: $\mathbb{E}(Y\, g\circ X) = \mathbb{E}(Z\, g\circ X)$.

The function h in (i) is denoted $\mathbb{E}(Y\mid X=\cdot)$. Such a Z always exists and is a.s. unique (existence follows from the Radon–Nikodym theorem, A3.68, applied to a suitable measure built from Y and the law of X). An equivalent formulation restricts g in (ii) to indicators of measurable sets $B\subset R_X$:

As before, the tower property $\mathbb{E}(\mathbb{E}(Y\mid X)) = \mathbb{E}(Y)$ and the "taking out what is known" rule $\mathbb{E}(Y g(X)\mid X) = g(X)\mathbb{E}(Y\mid X)$ continue to hold.

• $Z = \mathbb{E}(Y\mid X)$the a.s.-unique random variable satisfying (i) and (ii).

A3.81 conditions on a random variable X, but the same construction extends to conditioning on an arbitrary sub-σ-algebra $\mathcal{F}\subset\mathcal{A}$: $\mathbb{E}(Y\mid\mathcal{F})$ is the a.s.-unique $\mathcal{F}$-measurable random variable Z with $\mathbb{E}(Y\mathbf{1}_B)=\mathbb{E}(Z\mathbf{1}_B)$ for every $B\in\mathcal{F}$. Taking $\mathcal{F} = \sigma(X)$, the σ-algebra generated by X (A3.58), recovers $\mathbb{E}(Y\mid X)$ exactly, since requirement (i) of A3.81 is precisely $\sigma(X)$-measurability. This more general formulation is indispensable for stochastic processes (A3.107), where one conditions on the entire history up to a time, an object naturally described as a σ-algebra rather than a single random variable.

• $\mathbb{E}(Y\mid\mathcal{F})$conditional expectation given a σ-algebra; subsumes $\mathbb{E}(Y\mid X)$.
• $\sigma(X)$σ-algebra generated by X; the information contained in X.

Conditional expectation inherits the monotonicity and linearity of ordinary expectation, and satisfies a conditional Jensen's inequality: for convex $\gamma:\mathbb{R}^d\to\mathbb{R}$ with $\gamma\circ Y$ integrable,

The unconditional version (take $X$ constant, or apply the tower property) is the familiar $\gamma(\mathbb{E} Y)\le \mathbb{E}(\gamma(Y))$. Taking $\gamma(y)=|y|$ gives $|\mathbb{E}(Y\mid X)| \le \mathbb{E}(|Y|\mid X)$; taking $\gamma(y)=|y|^2$ and a square-integrable Y shows $\mathbb{E}(Y\mid X)$ is also square-integrable with $\mathbb{E}(|\mathbb{E}(Y\mid X)|^2) \le \mathbb{E}(|Y|^2)$, the key inequality used next.

• Jensen's inequalityconvexity of $\gamma$ pulls outside the (conditional) expectation as a lower bound.

The square-integrability fact from A3.83 yields the central interpretation of conditional expectation in statistical learning: $\mathbb{E}(Y\mid X)$ is the best mean-square approximation of Y by a (measurable) function of X. Indeed, for any square-integrable $Z = g(X)$,

and taking expectations on both sides gives $\mathbb{E}|Y-Z|^2 \ge \mathbb{E}|Y-\mathbb{E}(Y\mid X)|^2$ with equality iff $Z=\mathbb{E}(Y\mid X)$ a.s. This identity is the probabilistic origin of the squared-error loss used to motivate regression: the regression function $x\mapsto \mathbb{E}(Y\mid X{=}x)$, first encountered informally in A3.20 and A3.46, is precisely what supervised learning with squared loss aims to estimate.

• regression function$x \mapsto \mathbb{E}(Y\mid X{=}x)$; the L²-optimal predictor of Y from X.

For measurable $A\subset R_Y$, applying conditional expectation to the indicator $\mathbf{1}_A$ defines the conditional probability $\mathbb{P}(Y\in A\mid X) := \mathbb{E}(\mathbf{1}_A \mid X)$, also written $\mathbb{P}_Y(A\mid X)$. For each fixed A this is defined only up to a null set, and assembling these into a single probability distribution $A \mapsto \mathbb{P}_Y(A\mid X)(\omega)$ for (almost) every $\omega$ simultaneously requires care, since there are uncountably many sets A. Under the standing assumption that $R_Y$ is Polish (A3.58), such a regular conditional distribution always exists, satisfying

for every measurable h with $h\circ Y$ integrable. When the pair $(X,Y)$ has a joint density $\phi$ with respect to $\mu_X\otimes\mu_Y$, the regular conditional distribution is again absolutely continuous, with density $\phi(y\mid X{=}X(\omega))$, recovering A3.45 as a special case, and reducing to the discrete formula of A3.19 when Y is discrete and $\mu_Y=\mathrm{card}$.

• regular conditional distributiona jointly-measurable family of probability measures representing $\mathbb{P}_Y(\cdot\mid X)$.

Given a parametric family of densities $\{\phi(\cdot\,;\theta)\}_{\theta\in\Theta}$ and an observed i.i.d. sample $T=(x_1,\dots,x_N)$, the likelihood function is

viewed as a function of $\theta$ with the data held fixed — the opposite role from the density, where $\theta$ is fixed and x varies. Because products of small numbers underflow numerically and sums are easier to differentiate, one almost always works with the log-likelihood:

• $L(\theta)$likelihood; the joint density evaluated at the observed sample, as a function of $\theta$.

Two parameter values $\theta_1,\theta_2$ are compared by their likelihood ratio $L(\theta_1)/L(\theta_2)$, which measures how much more (or less) plausible $\theta_1$ is than $\theta_2$ in light of the data — independent of any overall normalizing constant in $\phi$. The likelihood principle asserts that all the information the data provides about $\theta$ is contained in the likelihood function up to a multiplicative constant: two experiments yielding proportional likelihood functions should lead to identical inferences about $\theta$, regardless of the design that produced the data. This principle underlies both maximum likelihood estimation (A3.96) and the construction of likelihood-ratio test statistics (A3.102) used throughout hypothesis testing.

• $L(\theta_1)/L(\theta_2)$likelihood ratio; relative plausibility of two parameter values.

A parametric family $\{\phi(\cdot;\theta)\}_{\theta\in\Theta}$ is identifiable if distinct parameters always give distinct distributions: $\theta_1\ne\theta_2 \Rightarrow \phi(\cdot;\theta_1) \ne \phi(\cdot;\theta_2)$. Without identifiability, no amount of data — even infinite data — can distinguish between the competing parameter values, since they generate identical observable distributions; the likelihood function (A3.86) is then constant along entire equivalence classes of $\theta$, and the MLE (A3.96) is not unique. Mixture models (A3.54) are a classic source of identifiability failures (e.g. under label-switching of components), and checking identifiability is a necessary first step before any estimation procedure can be expected to recover meaningful parameter values.

• identifiabledistinct parameters always yield distinct distributions.

A statistic $S=S(X_1,\dots,X_N)$ (any measurable function of the data) is sufficient for $\theta$ if the conditional distribution of the full sample given S does not depend on $\theta$ — meaning S retains all the information in the sample relevant to $\theta$, and the rest of the data is "noise" once S is known. Formally, this is exactly the conditional-independence statement $T \perp \theta \mid S$ in the Bayesian sense, or equivalently that the likelihood ratio $L(\theta_1)/L(\theta_2)$ for any $\theta_1,\theta_2$ depends on the data only through S.

• sufficient statistica summary of the data that loses no information about $\theta$.

The Fisher–Neyman factorization theorem gives a practical criterion for sufficiency (A3.89): S is sufficient for $\theta$ iff the joint density factors as

for some functions h (not depending on $\theta$) and g. For exponential family likelihoods (A3.41), the sufficient statistic is exactly $\sum_i t(x_i)$ appearing in the canonical form, and its sample average is sufficient for the natural parameter — explaining why exponential family models admit such compact, finite-dimensional summaries of arbitrarily large datasets, in sharp contrast to general parametric families where no such finite-dimensional sufficient statistic need exist.

• factorization criteriona practical, density-level test for sufficiency.

A sufficient statistic is minimal if it is a function of every other sufficient statistic — the coarsest possible lossless summary of the data. The Rao–Blackwell theorem shows sufficiency is not merely a descriptive notion but is operationally useful for improving estimators: if $\hat\theta$ is any estimator and S is sufficient, then $\tilde\theta = \mathbb{E}(\hat\theta \mid S)$ satisfies

using the law of total variance (A3.30) and the fact that $\tilde\theta$ no longer depends on $\theta$ precisely because S is sufficient. Rao–Blackwellization is therefore a free variance-reduction step — never increasing bias while never increasing (and typically decreasing) variance — and is the conceptual ancestor of variance-reduction techniques in Monte Carlo estimation (A3.114).

• Rao–Blackwellizationconditioning an estimator on a sufficient statistic never increases its variance.

Markov's inequality: for a non-negative random variable X and $a>0$,

Chebyshev's inequality follows by applying Markov to $(X-\mathbb{E}X)^2$:

These elementary tail bounds, polynomial in 1/a, are the ancestors of the sharper exponential concentration inequalities used throughout statistical learning theory to bound the deviation of empirical averages from their means (A3.93–A3.94).

• tail boundan upper bound on $\mathbb{P}(|X-\mathbb{E}X|\ge a)$ as a function of a.

Let $X_1,\dots,X_N$ be independent with $X_i \in [a_i,b_i]$ a.s. Hoeffding's inequality gives an exponential (rather than polynomial) tail bound for the sum $S_N = \sum_i X_i$:

For the sample mean of i.i.d. bounded variables in $[0,1]$, this yields $\mathbb{P}(|\bar X_N - \mu|\ge t) \le 2e^{-2Nt^2}$ — the deviation shrinks exponentially in N, far faster than the polynomial bound from Chebyshev. Such concentration inequalities are the central technical tool for proving generalization bounds in statistical learning theory, e.g. via uniform convergence of empirical risk to population risk over a hypothesis class (A3.119).

• $S_N$sum of independent bounded random variables.

Hoeffding's bound (A3.93) requires boundedness; a more flexible sufficient condition for exponential concentration is the sub-Gaussian property: X (with $\mathbb{E}X=0$) is sub-Gaussian with parameter $\sigma^2$ if $\mathbb{E}(e^{tX}) \le e^{\sigma^2t^2/2}$ for all t, i.e. its MGF (A3.52) is dominated by a Gaussian's. Bounded variables and Gaussians themselves are sub-Gaussian, and sums of independent sub-Gaussians are again sub-Gaussian with variance parameters adding. Bernstein's inequality extends Hoeffding to variables with bounded variance and a control on higher moments, giving a bound that interpolates between Gaussian-like decay for moderate deviations and exponential (Poisson-like) decay for large deviations — yielding the sharper rates used in modern statistical learning theory when variance information beyond a simple bound is available.

• sub-GaussianMGF dominated by a Gaussian's; the general condition behind exponential concentration.

Under regularity conditions allowing the exchange of derivative and integral (justified by the Dominated Convergence Theorem, A3.65), gradients of an expectation with respect to parameters of the integrand satisfy

when the sampling distribution p(x) does not itself depend on $\theta$. This identity is what licenses differentiating the log-likelihood term by term (A3.86), and is a prerequisite for defining the score function and Fisher information rigorously (A3.96–A3.97). When $p$ itself depends on $\theta$, the score-function and reparameterization gradient estimators (A3.109) are needed instead.

• differentiation under the integralexchanging $\nabla_\theta$ and $\mathbb{E}$, justified by dominated convergence.

The maximum likelihood estimator (MLE) is the parameter value that makes the observed data most probable:

Under standard regularity conditions (including identifiability, A3.88), the MLE is consistent ($\hat\theta_{\mathrm{MLE}} \xrightarrow{p} \theta_0$, the true parameter), asymptotically normal ($\sqrt{N}(\hat\theta_{\mathrm{MLE}}-\theta_0) \xrightarrow{d} \mathcal{N}(0, I(\theta_0)^{-1})$, with $I$ the Fisher information of A3.97), and asymptotically efficient (achieves the Cramér–Rao lower bound asymptotically). Many standard losses in machine learning — cross-entropy, squared error under Gaussian noise — are negative log-likelihoods, making ERM (A3.78) with these losses a form of maximum likelihood estimation.

• $\hat\theta_{\mathrm{MLE}}$the maximizer of the log-likelihood over $\Theta$.

For a smooth parametric family $\phi(x;\theta)$, the score function is $s(x;\theta) = \nabla_\theta \log\phi(x;\theta)$, satisfying $\mathbb{E}_\theta[s(X;\theta)]=0$ (differentiate the normalization $\int\phi=1$ under the integral sign, A3.95). The Fisher information is its covariance:

the second equality showing that Fisher information also measures the expected curvature of the log-likelihood. The Cramér–Rao lower bound states that any unbiased estimator $\hat\theta$ of $\theta$ based on N i.i.d. samples satisfies

an inequality between positive semi-definite matrices (A3.29). The MLE asymptotically attains this bound (A3.96), making Fisher information a measure of how much information a single observation carries about $\theta$, and motivating its use in natural gradient descent and information-geometric optimization.

• $I(\theta)$Fisher information matrix; the covariance of the score, or negative expected Hessian of the log-likelihood.

The entropy of a distribution P with density (or p.m.f.) p is

for discrete P this is the familiar Shannon entropy $-\sum_x p(x)\log p(x) \ge 0$, measuring the average "surprise" or uncertainty of an outcome drawn from P — maximized by the uniform distribution and equal to zero exactly when P is a point mass. For continuous P, the same formula defines differential entropy, which (unlike Shannon entropy) can be negative and is not invariant under change of variables (A3.42), reflecting the fact that a continuous density is not itself a probability. Entropy is the foundational quantity from which cross-entropy (A3.99) and the KL divergence (A3.100) are both built.

• $H(P)$entropy; average log-improbability of an outcome under P.

The cross-entropy between two distributions P and Q (with densities p, q) is

the second equality following directly from the definitions and proven in A3.100. Since $H(P)$ does not depend on Q, minimizing cross-entropy $H(P,Q)$ over Q is equivalent to minimizing $D_{\mathrm{KL}}(P\|Q)$ — this is exactly the cross-entropy loss used to train classifiers, where P is the empirical label distribution and Q the model's predicted distribution, and where the connection to maximum likelihood (A3.96) is made precise in A3.101.

• $H(P,Q)$cross-entropy; the standard classification training loss.

For distributions P and Q on the same space, with $P\ll Q$ (A3.67), the Kullback–Leibler (KL) divergence is

(for densities p, q w.r.t. a common dominating measure, A3.68). KL divergence is always non-negative, $D_{\mathrm{KL}}(P\|Q) \ge 0$ with equality iff $P=Q$ a.e. (a direct consequence of Jensen's inequality, A3.83, applied to $-\log$), but it is not symmetric and does not satisfy the triangle inequality, so it is not a true metric. KL divergence is the workhorse loss of variational inference (A3.105), the VAE evidence lower bound, and is intimately related to maximum likelihood: minimizing $D_{\mathrm{KL}}(\widehat{\mathbb{P}}_N \,\|\, \mathbb{P}_\theta)$ over $\theta$ is equivalent to maximizing the log-likelihood (A3.101).

• $D_{\mathrm{KL}}(P\|Q)$KL divergence; expected log-ratio of densities under P.

The connection alluded to in A3.99–A3.100 can be made precise: minimizing $D_{\mathrm{KL}}(\widehat{\mathbb{P}}_N \| \mathbb{P}_\theta)$, where $\widehat{\mathbb{P}}_N$ is the empirical distribution (A3.76), expands as

where $\ell(\theta)$ is the log-likelihood of A3.86. Since the entropy term $H(\widehat{\mathbb{P}}_N)$ does not depend on $\theta$, minimizing the KL divergence over $\theta$ is exactly equivalent to maximizing the log-likelihood — i.e. to computing the MLE (A3.96). This identity is the conceptual bridge between the information-theoretic perspective (minimize divergence to the data) and the classical statistical perspective (maximize likelihood of the data), and as $N\to\infty$, $\widehat{\mathbb{P}}_N \to \mathbb{P}_X$ (A3.77), so the MLE asymptotically targets $\arg\min_\theta D_{\mathrm{KL}}(\mathbb{P}_X \| \mathbb{P}_\theta)$, the best approximation to the truth within the model class.

• MLE ≡ KL projectionmaximizing likelihood is minimizing KL divergence to the empirical distribution.

For jointly distributed X, Y, the mutual information is the KL divergence between the joint and the product of marginals:

with $I(X;Y)=0$ iff X and Y are independent (A3.39) — recovering, via the non-negativity of KL (A3.100), the fact that independence is the unique zero-information case. Mutual information measures the reduction in uncertainty about one variable given the other, equivalently $I(X;Y) = H(X) - H(X\mid Y)$ for an appropriately defined conditional entropy, and underlies information bottleneck approaches and representation learning objectives, where one seeks representations that retain mutual information with a target while discarding it with respect to nuisance variables.

• $I(X;Y)$mutual information; zero iff X, Y independent.

KL divergence is a special case of a broad family. For a convex function $f$ with $f(1)=0$, the f-divergence between P and Q (with density ratio $r=dP/dQ$) is

Convexity of f combined with Jensen's inequality (A3.83) gives $D_f(P\|Q)\ge f(1) = 0$ for every choice of f, generalizing the non-negativity argument used for KL in A3.100. Choosing $f(t)=t\log t$ recovers $D_{\mathrm{KL}}$; other choices give the total variation distance and Hellinger distance, detailed next.

• $D_f(P\|Q)$f-divergence; a convex generalization of KL parameterized by f.

Two important f-divergences (A3.103) are genuine metrics, unlike KL. Choosing $f(t)=\frac{1}{2}|t-1|$ gives the total variation distance

symmetric and satisfying the triangle inequality. Choosing $f(t)=(\sqrt t - 1)^2$ gives the squared Hellinger distance $H^2(P,Q) = \frac{1}{2}\int(\sqrt p - \sqrt q)^2$, related to TV by the inequalities $H^2 \le \mathrm{TV} \le \sqrt{2}\,H$. These divergences generalize the GAN training objective (which corresponds to a specific f-divergence, the Jensen–Shannon divergence) and provide alternative, sometimes better-behaved, distances between distributions for generative modeling, particularly when the supports of P and Q do not fully overlap and KL divergence becomes infinite.

• $\mathrm{TV}(P,Q)$total variation distance; a true metric, unlike KL.

A statistical hypothesis test chooses between a null hypothesis $H_0$ and an alternative $H_1$ based on data, via a test statistic $T(X_1,\dots,X_N)$ and a rejection region. Two error types are distinguished: Type I error (rejecting $H_0$ when true, probability $\alpha$, the significance level) and Type II error (failing to reject $H_0$ when false, probability $\beta$; $1-\beta$ is the test's power). By the Neyman–Pearson lemma, the most powerful test for fixed $\alpha$ between two simple hypotheses thresholds the likelihood ratio of A3.87.

The p-value is the probability, under $H_0$, of observing a test statistic at least as extreme as the one realized:

While less central to modern deep learning, hypothesis testing underlies A/B testing for model comparison, statistical significance of benchmark improvements, and permutation tests for feature importance.

• $\alpha$significance level; the tolerated Type I error rate.

A confidence interval at level $1-\alpha$ for a parameter $\theta$ is a random interval $[\hat\theta_L, \hat\theta_U]$, computed from data, such that

Crucially, the randomness is in the interval, not in the fixed (if unknown) parameter $\theta$ — a frequentist confidence interval is not the same as a Bayesian credible interval (A3.110), which instead asserts $\mathbb{P}(\theta \in [\theta_L,\theta_U] \mid T) \ge 1-\alpha$ for the random parameter under a posterior. For the sample mean of i.i.d. data with the CLT (A3.74), the standard asymptotic confidence interval is $\bar X_N \pm z_{\alpha/2}\, \sigma/\sqrt{N}$, using the standard normal quantile of A3.37.

• $[\hat\theta_L,\hat\theta_U]$a random interval covering $\theta$ with probability at least $1-\alpha$.

In the Bayesian setting of A3.48, with prior $\phi_\Theta$ on the parameter and likelihood $\phi(x\mid\theta)$, the maximum a posteriori (MAP) estimator maximizes the posterior density:

Comparing to A3.96, the MAP estimator is exactly the MLE plus a regularization term $\log\phi_\Theta(\theta)$ coming from the prior. A Gaussian prior $\theta \sim \mathcal{N}(0,\tau^2 I)$ yields the familiar $L^2$ (ridge) penalty, while a Laplace prior yields the $L^1$ (lasso) penalty — placing common regularization schemes in machine learning within a principled Bayesian framework, and recovering the MLE exactly in the limit of a flat (improper uniform) prior.

• $\hat\theta_{\mathrm{MAP}}$posterior mode; MLE regularized by the negative log-prior.

A prior $\phi_\Theta$ is conjugate to a likelihood $\phi(x\mid\theta)$ if the resulting posterior $\phi_\Theta(\theta\mid T)$ belongs to the same parametric family as the prior. For exponential family likelihoods (A3.41), conjugate priors always exist and admit closed-form posterior updates, because the sufficient statistic (A3.89–A3.90) summarizes the data's effect on the posterior in finitely many numbers. The canonical example: with a $\operatorname{Beta}(\alpha,\beta)$ prior (A3.53) on a Bernoulli success probability p and k successes in N trials,

Conjugacy allows exact, tractable sequential Bayesian updating as data arrives, avoiding numerical integration, and forms the basis of many online learning and bandit algorithms (e.g. Thompson sampling).

• conjugate priora prior family closed under Bayesian updating with a given likelihood.

The Gaussian family (A3.36) supplies a second canonical conjugacy example, central to Bayesian linear regression and Gaussian process inference (A3.111): with known variance $\sigma^2$ and a Gaussian prior $\theta \sim \mathcal{N}(\mu_0,\tau_0^2)$ on the mean, observing i.i.d. data $x_1,\dots,x_N \sim \mathcal{N}(\theta,\sigma^2)$ gives a Gaussian posterior

The posterior mean $\mu_N$ is a precision-weighted average of the prior mean and the sample mean, smoothly interpolating between the prior (when N is small or $\tau_0$ is small) and the MLE $\bar x_N$ (as $N\to\infty$) — a concrete illustration of how the MAP estimator (A3.107) blends prior belief with data evidence as more observations accumulate.

• precisioninverse variance; precisions of independent Gaussian sources add.

A credible interval at level $1-\alpha$ is the Bayesian counterpart to the frequentist confidence interval of A3.106: an interval $[\theta_L,\theta_U]$, computed from the posterior $\phi_\Theta(\theta\mid T)$, satisfying

Here the parameter $\theta$ is treated as random (under the posterior) and the data T is fixed at its observed value — the interpretation is the directly intuitive one ("given what I observed, there is at least $1-\alpha$ posterior probability that $\theta$ lies in this interval"), unlike the more subtle repeated-sampling interpretation required for confidence intervals. The narrowest such interval for a given coverage level is the highest posterior density (HPD) interval.

• credible intervala region of posterior probability mass $1-\alpha$.

Given a posterior $\phi_\Theta(\theta\mid T)$ over parameters, the posterior predictive distribution for a new observation $x_{\mathrm{new}}$ marginalizes over parameter uncertainty rather than plugging in a single point estimate:

This fully Bayesian averaging — rather than evaluating at $\hat\theta_{\mathrm{MAP}}$ or $\hat\theta_{\mathrm{MLE}}$ — naturally propagates parameter uncertainty into predictive uncertainty, and is the formal justification behind Bayesian neural networks, Gaussian process regression, and deep ensembles as approximations to this integral. When the integral is intractable, it is estimated by Monte Carlo (A3.114) using posterior samples drawn via MCMC (A3.113).

• predictive distributionprediction averaged over the posterior, not a single point estimate.

A3.95 covered differentiating an expectation when the sampling distribution does not depend on $\theta$. When it does, two estimators of the gradient are used in practice: the score-function (REINFORCE) estimator,

valid for any (even non-differentiable) f but typically high-variance, derived by writing $\nabla_\theta p = p\,\nabla_\theta \log p$ (the score function of A3.97) and differentiating under the integral; and the reparameterization (pathwise) estimator, valid when $X = g(\varepsilon;\theta)$ for a base noise variable $\varepsilon$ independent of $\theta$:

typically much lower-variance and the engine behind training variational autoencoders. The Fisher information of A3.97 follows as a special case of differentiating an expectation twice using the score-function identity.

• score function$\nabla_\theta \log p(x;\theta)$; zero in expectation under $p(\cdot;\theta)$.

Let $g:\mathbb{R}^d\to\mathbb{R}^d$ be a diffeomorphism (smooth bijection, smooth inverse) and $Y=g(X)$ for X with density $p_X$. Generalizing the one-dimensional formula of A3.42 to d dimensions, Y has density

where $dg(x)$ is the Jacobian matrix of g. This identity is the analytical engine of normalizing flows, which model complex distributions as the pushforward of a simple base distribution through a sequence of invertible, easy-to-Jacobian transformations, and underlies the reparameterization trick (A3.112) used in VAEs.

• $|\det\,dg(x)|$absolute Jacobian determinant; local volume distortion of g.

A random vector $X\in\mathbb{R}^d$ is Gaussian with mean $\mu$ and (positive-definite) covariance $\Sigma$ (A3.29), written $X\sim\mathcal{N}(\mu,\Sigma)$, if its density is

Two defining closure properties make the Gaussian family central to machine learning: any affine map of a Gaussian is Gaussian, $AX+b \sim \mathcal{N}(A\mu+b,\, A\Sigma A^T)$ (consistent with the covariance transformation rule of A3.29); and the marginal and conditional distributions of any sub-block of jointly Gaussian variables are themselves Gaussian. Partitioning $X=(X_1,X_2)$ with corresponding mean and covariance blocks, the conditional distribution is

This closed-form conditioning, an explicit instance of the general conditional expectation of A3.81 in the Gaussian case, is the foundation of Gaussian process regression and Kalman filtering.

• $\Sigma_{12}\Sigma_{22}^{-1}$the "regression coefficient" mapping $X_2$ into the conditional mean of $X_1$.

A sequence of random variables $(X_n)_{n\ge 0}$ is a Markov chain if it satisfies the Markov property, the dynamic analogue of the conditional independence introduced in A3.24 and A3.51: the future depends on the past only through the present,

For a time-homogeneous chain on a discrete state space, the dynamics are fully described by the transition matrix $P_{ij} = \mathbb{P}(X_{n+1}=j\mid X_n=i)$. A distribution $\pi$ is stationary if $\pi P = \pi$; under mild conditions (irreducibility, aperiodicity) the chain's distribution converges to a unique stationary $\pi$ regardless of initial state. Markov chains underlie hidden Markov models, MCMC sampling algorithms (A3.116), and the theoretical analysis of stochastic gradient descent as a stochastic process.

• $P_{ij}$transition probability from state i to state j.
• $\pi$stationary distribution; satisfies $\pi P = \pi$.

Markov Chain Monte Carlo (MCMC) methods construct a Markov chain (A3.115) whose stationary distribution is a target density $\pi(x)$ (typically a Bayesian posterior, known only up to a normalizing constant — recall the evidence term of A3.48 is often intractable), then approximate expectations under $\pi$ by averaging along a simulated trajectory. The Metropolis–Hastings algorithm proposes a move $x' \sim q(\cdot \mid x)$ from the current state and accepts it with probability

a construction that guarantees $\pi$ is stationary via the detailed balance condition $\pi(x)\,P(x\to x') = \pi(x')\,P(x'\to x)$. MCMC sidesteps the need to compute the (often intractable) normalizing constant in Bayes' rule (A3.48), making it foundational to practical Bayesian machine learning.

• $a(x,x')$Metropolis–Hastings acceptance probability.
• detailed balancea sufficient condition for $\pi$ to be the stationary distribution.

To estimate $\mathbb{E}_{X\sim p}[f(X)]$ when sampling from p directly is difficult, but a more convenient proposal distribution q (with $p \ll q$, A3.67) is available, importance sampling reweights samples drawn from q:

where $w(x)=p(x)/q(x)$ is the importance weight. The estimator is unbiased (it is exactly a Monte Carlo average, A3.118, of a transformed function) but its variance can be large (even infinite) if q places little mass where $p\cdot f$ is large; choosing a good proposal is critical, and Rao–Blackwellization-style variance reduction (A3.91) is a frequent supplement. Importance sampling underlies off-policy evaluation in reinforcement learning, variational inference bounds, and rare-event simulation.

• $w(x)=p(x)/q(x)$importance weight correcting for sampling from q instead of p.

Underlying both MCMC (A3.116) and importance sampling (A3.117) is the basic Monte Carlo estimator: given i.i.d. samples $x_1,\dots,x_M$ from a distribution p, the sample average $\widehat\mu_M = \frac{1}{M}\sum_m f(x_m)$ estimates $\mathbb{E}_{X\sim p}[f(X)]$. This is exactly the SLLN (A3.72) applied to the random variables $f(x_m)$, with convergence rate governed by the CLT (A3.74):

a rate independent of the dimension of X — the key advantage of Monte Carlo methods over deterministic numerical integration in high dimensions, where grid-based quadrature suffers exponentially from the curse of dimensionality. The error constant depends on $\operatorname{Var}(f(X))$, which is exactly what importance sampling and Rao–Blackwellization aim to reduce.

• $\widehat\mu_M$Monte Carlo estimator; converges at the dimension-free rate $O(1/\sqrt M)$.

A sequence $X_1,X_2,\dots$ is exchangeable if its joint distribution is invariant under any finite permutation of indices — a strictly weaker assumption than i.i.d. (A3.55). De Finetti's theorem states that any infinite exchangeable sequence of random variables can be represented as a mixture of i.i.d. sequences:

for some prior measure $\mu$ on a parameter space $\Theta$. This theorem provides the deepest theoretical justification for the entire Bayesian paradigm (A3.48, A3.107–A3.111): treating parameters as random with a prior is not merely a modeling convenience but is forced upon us by the weak, intuitive assumption that the order of observations should not matter.

• exchangeablejoint law invariant under permutation of the sequence.

The generalization gap for a hypothesis $\hat\theta$ trained on data T is the difference between population and empirical risk (A3.78), $R(\hat\theta) - \widehat R_N(\hat\theta)$. Probably Approximately Correct (PAC) learning theory bounds, with probability at least $1-\delta$ over the draw of the training sample, the gap uniformly over a hypothesis class $\mathcal{H}$:

where $\mathcal{C}(\mathcal{H},N)$ is a complexity term (e.g. based on VC dimension or Rademacher complexity) that grows with the expressiveness of $\mathcal{H}$ and shrinks with sample size. Such bounds, built from Hoeffding-type concentration inequalities (A3.93) combined with a union bound (A3.6) or chaining argument over $\mathcal{H}$, formalize the bias–variance trade-off (A3.79) and explain why minimizing empirical risk (A3.78) is a sound strategy for minimizing true risk, provided the hypothesis class is not too complex relative to N — closing the loop from the union bound of A3.6, through the law of large numbers of A3.72, to the full statistical learning theory underpinning modern machine learning.

• $\mathcal{C}(\mathcal{H},N)$a complexity measure of the hypothesis class, controlling the generalization gap.

The chi-squared distribution with k degrees of freedom arises as the sum of squares of k independent standard normal random variables. It is a special case of the gamma distribution and plays a central role in hypothesis testing, confidence interval construction, and goodness-of-fit tests.

Its mean is k and variance 2k. For large k, the distribution approximates a normal distribution. In statistical learning, the chi-squared distribution underlies likelihood ratio tests and the Wilks phenomenon for nested model comparison.

• $k$degrees of freedom (positive integer).

If X follows a normal distribution with mean μ and variance σ², then Y = e^X is log-normally distributed. It models positive, right-skewed quantities such as incomes, biological sizes, and failure times in reliability engineering.

The parameters μ and σ are the mean and standard deviation of the underlying normal variable. The log-normal distribution is heavy-tailed and commonly used in Bayesian modelling as a prior for positive scale parameters.

• $\mu$location parameter (log-scale mean).
• $\sigma$scale parameter (log-scale standard deviation, positive).

The inverse Gaussian (or Wald) distribution describes the first-passage time of a Brownian motion with drift. It is a two-parameter family that supports positive reals and is widely used in survival analysis, accelerated life testing, and as a mixing distribution in hierarchical models.

The parameter μ > 0 is the mean, and λ > 0 controls the shape. As λ grows, the distribution approaches a normal; smaller λ yields heavier tails.

• $\mu$mean parameter (positive).
• $\lambda$shape parameter (positive).

The asymmetric Laplace distribution generalises the symmetric Laplace to allow different rates of exponential decay on each side of a location parameter. It is used in quantile regression, Bayesian inference for median regression, and financial modelling of skewed returns.

where μ is the location, σ > 0 the scale, and 0 < p < 1 determines the skewness (p = 0.5 recovers symmetry). This formulation is particularly convenient for likelihood-based quantile estimation.

• $\mu$location parameter.
• $\sigma$scale parameter (positive).
• $p$skewness parameter (probability, 0 < p < 1).

The Pareto distribution (type I) models quantities bounded below by a known minimum x_m and is characterised by a power-law tail. It appears in wealth distribution, city sizes, and natural phenomena exhibiting scale invariance.

With x_m > 0 fixed, the shape parameter α > 0 controls tail heaviness; the mean exists only for α > 1 and the variance for α > 2. It is the canonical distribution for extreme value theory.

• $\alpha$shape parameter (positive), tail index.
• $x_m$fixed minimum (scale parameter, positive).

The Weibull distribution is widely used in reliability engineering and survival analysis to model lifetimes. When the shape parameter k is known, the remaining scale parameter controls the time scale of failures.

Given a fixed k > 0, the distribution is a one-parameter exponential family in λ. The hazard function is increasing for k > 1, decreasing for k < 1, and constant (exponential) for k = 1.

• $\lambda$scale parameter (positive).
• $k$known shape parameter (positive, fixed).

The Laplace (double-exponential) distribution, with known location μ, is the maximum entropy distribution on the real line for a fixed mean absolute deviation. It serves as a robust alternative to the normal in regression and Bayesian lasso.

With μ fixed, the scale parameter b > 0 controls dispersion. The Laplace distribution generates sparsity when used as a prior for coefficients in linear models, leading to the L1-penalised likelihood interpretation.

• $\mu$known location parameter.
• $b$scale parameter (positive).

The continuous Bernoulli distribution extends the Bernoulli to the unit interval (0, 1), parameterised by λ ∈ (0,1). It is used in variational autoencoders and as a probabilistic model for continuous proportions.

The normalising constant can be expressed as C(λ) = \frac{2\tanh^{-1}(1-2\lambda)}{1-2\lambda} for λ ≠ 0.5, and C(0.5)=2. The distribution is not exponential family except when the normaliser is treated as fixed.

• $\lambda$shape parameter, 0 < λ < 1.

If X follows a gamma distribution with shape α and rate β, then Y = 1/X is inverse-gamma distributed. It is the conjugate prior for the variance of a normal distribution and for scale parameters in Bayesian hierarchical models.

Both parameters are positive. The inverse gamma has a heavy right tail and finite mean only for α > 1. It appears naturally as the marginal posterior of a variance parameter under a normal-inverse-gamma conjugate model.

• $\alpha$shape parameter (positive).
• $\beta$scale parameter (positive).

The generalised inverse Gaussian (GIG) distribution encompasses the inverse Gaussian, gamma, and inverse gamma as special cases. It is a flexible three-parameter family for positive data, often used as the mixing distribution in normal variance-mean mixtures.

Here K_λ is a modified Bessel function of the second kind. The domain of parameters allows various tail behaviours; for instance, the hyperbolic distribution arises as a normal variance-mean mixture with a GIG mixing law.

• $\lambda$tail parameter (real).
• $\chi$scale parameter (non-negative).
• $\psi$scale parameter (non-negative).

The scaled inverse chi-squared distribution is a reparameterisation of the inverse gamma, widely used as a conjugate prior for a normal variance σ² in Bayesian analysis. It is parameterised by degrees of freedom ν and a scale σ₀².

The distribution has mode at σ₀² when ν > 2 and is the sampling distribution of σ₀²/χ²_ν. It provides a conditionally conjugate prior for variance components in hierarchical linear models.

• $\nu$degrees of freedom (positive).
• $\sigma_0^2$scale parameter (positive).

The multivariate normal (Gaussian) distribution generalises the univariate normal to k dimensions and is the most important continuous distribution in multivariate statistics, machine learning, and spatial modelling.

where μ is the mean vector and Σ the positive-definite covariance matrix. All marginal and conditional distributions are normal; linear transformations remain normal. It underpins Gaussian processes, Kalman filters, and principal component analysis.

• $\boldsymbol{\mu}$mean vector (length k).
• $\boldsymbol{\Sigma}$covariance matrix (k×k, positive definite).

The categorical distribution models a single trial with K possible outcomes. It is parameterised by a probability vector p = (p₁,…,p_K) where p_i ≥ 0 and ∑ p_i = 1. All variants (generalised Bernoulli, one-hot encoding) reduce to this basic form.

The distribution is the building block of multinomial models, softmax classification, and discrete choice models. In the exponential family form, the natural parameter is the log-odds with respect to a reference category.

• $p_i$probability of category i (sum to 1).
• $K$number of categories (positive integer).

The multinomial distribution generalises the binomial to n independent categorical trials over K outcomes. The probability mass function gives the count vector (x₁,…,x_K) with ∑ x_i = n.

All variants—including the multinomial as a sum of independent categoricals, the binomial as the special case K=2, and the conditional multinomial—share this core structure. It is the canonical model for counts in contingency tables and document-term matrices.

• $p_i$cell probability (i = 1,…,K).
• $n$total number of trials (positive integer).

The Dirichlet distribution is the conjugate prior to the categorical and multinomial distributions. It is defined on the (K‑1)-dimensional probability simplex and is parameterised by a concentration vector α = (α₁,…,α_K) with all α_i > 0.

where B(α) is the multivariate Beta function. All variants—symmetric Dirichlet (α_i all equal), the Beta distribution (K=2), and Dirichlet process priors—extend naturally from this form. It is central to topic models, Bayesian mixture models, and compositional data analysis.

• $\boldsymbol{\alpha}$concentration parameter vector (all entries positive).

The Wishart distribution is a multivariate generalisation of the chi-squared distribution, defined over symmetric positive-definite matrices. It arises as the distribution of the sample covariance matrix for multivariate normal observations and serves as the conjugate prior for the precision matrix in Bayesian multivariate analysis.

Here n ≥ p are the degrees of freedom, V is a p×p positive-definite scale matrix, and Γ_p is the multivariate gamma function. The Wishart plays a fundamental role in covariance estimation, Gaussian graphical models, and random matrix theory.

• $n$degrees of freedom (integer, n ≥ p).
• $\mathbf{V}$scale matrix (p×p, positive definite).