Suppose A is a tall matrix with linearly independent columns so that Ax = b is over-determined. Let ˆx denote the least squares approximate solution to the set of equations Ax = b. We want to show that ||Aˆx − b|| ≤ ||b||.
First explain why the inequality is true, i.e. ||Aˆx − b|| ≤ ||b||.