REFERENCE SHEET

Linear regression

Fit conditional means while keeping the assumptions behind inference visible.

Before this Orthogonality, least squares & SVD, Estimation & likelihood

y = Xβ + ε; β̂ = (XᵀX)⁻¹Xᵀy (full column rank)

Definitions

Design matrix

An n×p matrix whose rows describe observations and whose columns are predictors, optionally including a column of ones.

OLS

Ordinary least squares minimizes ‖y−Xβ‖² over β.

Residual

The fitted error e=y−Xβ̂. It is not the unobserved model error ε.

Key theorems

OLS unbiasedness

With fixed full-column-rank X and E[ε|X]=0, E[β̂|X]=β.

Gauss–Markov

If additionally Var(ε|X)=σ²I, OLS has minimum covariance in the positive-semidefinite order among estimators linear in y and unbiased conditional on X. Normal errors are not required.

Residual orthogonality

Every OLS fit satisfies Xᵀe=0. With an intercept, residuals sum to zero.

Notable applications

Prediction

Approximate a response from measured features, assessing error on data not used to fit.

Adjustment

Include measured covariates to model conditional associations; causal interpretation requires additional design or assumptions.

Work it out

Try before revealing

01 / Compute

Fit y=a+bx to points (0,1),(1,3),(2,5).

Hint

The points lie exactly on a line.

Solution

a=1 and b=2, with all residuals zero.

02 / Prove

Why do OLS residuals sum to zero when X contains an intercept?