REFERENCE SHEET

Orthogonality, least squares & SVD

Approximate a vector by its closest point in a subspace.

Before this Vector spaces & linear maps

AᵀAx̂ = Aᵀb; A = UΣVᵀ

Definitions

Inner product

Over ℝ, a symmetric positive-definite bilinear form. It defines a norm by ‖v‖=√⟨v,v⟩ and orthogonality by ⟨u,v⟩=0.

Orthogonal projection

The unique closest vector to b in a finite-dimensional inner-product subspace S. Its residual is orthogonal to S.

Singular values

The nonnegative square roots of the eigenvalues of AᵀA for a real matrix A.

Key theorems

Projection theorem

A vector p∈S minimizes ‖b−p‖ exactly when b−p is orthogonal to S.

Normal equations

Every least-squares minimizer solves AᵀAx=Aᵀb. The coefficient vector is unique if A has full column rank.

Singular value decomposition

Every real m×n matrix has A=UΣVᵀ with orthogonal U,V and rectangular diagonal Σ with nonnegative entries. Complex matrices use conjugate transpose.

Notable applications

Regression

Least squares fits a response vector to the column space of a design matrix.

Compression

Truncating the SVD to the k largest singular values gives a best rank-at-most-k approximation in Frobenius norm.

Work it out

Try before revealing

01 / Compute

Project b=(2,0) onto span{(1,1)}.

Hint

Use ⟨b,u⟩u/⟨u,u⟩.

Solution

The projection is (1,1); the residual (1,−1) is orthogonal to (1,1).

02 / Compute

Find the singular values of diag(3,−2,0).

Hint

Use AᵀA.