REFERENCE SHEET

Estimation & likelihood

Choose a model, then distinguish an unknown parameter from a statistic computed on data.

Before this Random variables & moments

MSE(θ̂) = Var(θ̂) + Bias(θ̂)²

Definitions

Estimator

A statistic θ̂(X₁,…,Xₙ) used to estimate a parameter θ. Its sampling distribution describes its behavior under repeated samples.

Bias and consistency

Bias is Eθ[θ̂]−θ. Consistency means θ̂ₙ→θ in probability as n grows.

Likelihood

L(θ;x) is the joint mass or density at the observed data, viewed as a function of θ. Maximum likelihood chooses a maximizing parameter, if one exists.

Key theorems

Bias–variance decomposition

With finite second moments, mean squared error E[(θ̂−θ)²] equals variance plus squared bias.

Unbiased sample variance

For iid data with finite variance σ² and n≥2, s²=Σᵢ(Xᵢ−X̄)²/(n−1) satisfies E[s²]=σ².

Notable applications

Parameter fitting

Likelihood turns model assumptions into an optimization objective.

Regularization

Accepting some bias can reduce variance enough to improve prediction error.

Work it out

Try before revealing

01 / Derive

Find the MLE of p from k successes in n independent Bernoulli trials, allowing p∈[0,1].

Hint

Maximize k log p+(n−k) log(1−p); handle endpoints separately.

Solution

For 0<k<n the derivative vanishes at p=k/n and the log-likelihood is strictly concave. If k=0 or n, the maximum is at 0 or 1. Thus p̂=k/n.

02 / Compute

Estimator A has variance 4 and bias 0; B has variance 1 and bias 1. Which has smaller MSE?