Measure anything twice and you get two different numbers. So when an experiment says "these quantities should be related by a line", the data never land exactly on one, and the system $Ax = b$ that asks for the line has more equations than unknowns and no solution at all. Giving up isn't an option: the scientist still needs a slope, the engineer still needs a calibration. Least squares is the answer everybody uses. Instead of solving $Ax = b$, find the $x$ that misses by the least. It's the most-used computation in statistics, and it runs on everything this course has built: column spaces, transposes, projections, and right angles.
Three dots on a graph that refuse to sit on one line. No line goes through all three, but some line comes closest, and that's the one a scientist would draw. This page finds it, and the surprise is where the answer lives: the three heights of the dots make one point in space, every possible line makes a point on one particular plane, and the best line is the spot on the plane right underneath the data point. Fitting a line is dropping a perpendicular.
On Orthogonal Projection, the closest point of a subspace to a vector $b$ was its projection, the foot of the perpendicular. That was a theorem about geometry. This page turns it into a way to answer unanswerable systems.
Every line misses at least one of the three points. Scoring a line by the squares of its misses, added up, picks out a single winner, $y = \frac76 + \frac12x$. Nothing about it was found by eye: the rest of this page shows how to compute it in a few lines of arithmetic.
Start with the smallest possible case: one unknown. Here's a coin experiment from Hefferon's book. Flip a penny $30$ times, then $60$, then $90$, and count the heads.
A penny comes up heads $16$ times in $30$ flips, $34$ times in $60$ flips, and $51$ times in $90$ flips. If it lands heads a fraction $m$ of the time, then $30m = 16$, $60m = 34$, and $90m = 51$. No $m$ solves all three. What $m$ fits best?
The three equations say $ma = b$, with $a = (30, 60, 90)$ and $b = (16, 34, 51)$. Every $ma$ lies on the line through $a$, and $b$ doesn't (the ratios $\frac{16}{30} \approx 0.533$ and $\frac{34}{60} \approx 0.567$ already disagree). The point of that line closest to $b$ is the projection of $b$ onto $a$:
$$m = \frac{b \cdot a}{a \cdot a} = \frac{480 + 2040 + 4590}{900 + 3600 + 8100} = \frac{7110}{12600} = \frac{79}{140} \approx 0.564.$$A little over one half, so the penny looks fair enough.
Here it is on the graph of heads against flips. Every line through $0$ misses at least one count (the misses are drawn five times taller than they really are, or you couldn't see them). Add up the squares of the misses for each slope and you get a parabola; its bottom is $m = 0.564$.
That's the whole idea in miniature: when a system can't be solved, solve the closest thing to it instead. With more unknowns we need a name for "closest".
Let $A$ be an $m \times n$ matrix and $b$ a vector in $\R^m$. A least-squares solution of $Ax = b$ is a vector $\hat{x}$ in $\R^n$ such that $\|b - A\hat{x}\| \le \|b - Ax\|$ for every $x$ in $\R^n$.
Let $A$ be a matrix with three rows and one or two columns, and $b$ a point of $\R^3$. A least-squares solution of $Ax = b$ is an $\hat{x}$ that puts $A\hat{x}$ as close to $b$ as possible: $\|b - A\hat{x}\| \le \|b - Ax\|$ for every $x$.
If $Ax = b$ happens to have a solution, the smallest possible distance is $0$, and the least-squares solutions are exactly the ordinary ones. So nothing is lost: least squares only changes the answer when there was no answer. The distance being minimized gets its own name, because we'll measure it constantly.
Every $Ax$ lands on the line or plane the columns of $A$ make, so the question is which point of that line or plane is closest to $b$. If $b$ is already on it, the distance can be $0$ and the least-squares solution is an ordinary solution. The gap between $b$ and $Ax$ gets its own name.
For a system $Ax = b$ and any vector $x$, the residual vector is $r = b - Ax$.
Its entries are the misses, one per equation, and its length $\|b - Ax\|$ is the error of $x$. Since $\|r\|^2 = r_1^2 + r_2^2 + \cdots$, making the error smallest is the same as making the sum of the squares of the misses smallest. That's where the name "least squares" comes from. For the penny, $r = b - \frac{79}{140}a \approx (-0.93, 0.14, 0.21)$.
In 1801 the asteroid Ceres was spotted, tracked for a few weeks, and then lost in the Sun's glare. The 24-year-old Gauss took the handful of messy observations and predicted where it would show up again, using the method of least squares. Astronomers pointed their telescopes there that December, and there it was.
Before solving anything, practice scoring a guess. In free response, type one number.
Loading practice engine...
With one unknown, the possible $Ax$'s made a line and the projection formula found the closest one. With two unknowns, $Ax = x_1a_1 + x_2a_2$ ranges over every combination of the columns: the column space $\col(A)$, a plane when there are two independent columns. The closest point of it to $b$ is the foot of the perpendicular.
The foot is the one point $A\hat{x}$ where the leftover $b - A\hat{x}$ is orthogonal to the whole plane, which means orthogonal to each column. "Orthogonal to column $j$" is the dot product $a_j \cdot (b - A\hat{x}) = 0$, and stacking those dot products is exactly what $A^t$ does. That turns geometry into a system we can solve.
A vector $\hat{x}$ is a least-squares solution of $Ax = b$ exactly when it solves the normal equations $$A^tA\hat{x} = A^tb.$$ The normal equations always have at least one solution.
For any vectors, $(Ay) \cdot z = y \cdot (A^tz)$, since both are $y^tA^tz$. So if $A^t(b - A\hat{x}) = 0$, the residual $b - A\hat{x}$ is orthogonal to every vector $Ay$ of $\col(A)$.
Suppose $\hat{x}$ solves the normal equations, and take any $x$. Then $b - Ax = (b - A\hat{x}) + A(\hat{x} - x)$, and the two pieces are orthogonal, so by the Pythagorean theorem $$\|b - Ax\|^2 = \|b - A\hat{x}\|^2 + \|A(\hat{x} - x)\|^2 \ge \|b - A\hat{x}\|^2.$$ So $\hat{x}$ is a least-squares solution.
The normal equations have a solution: the orthogonal projection $p$ of $b$ onto $\col(A)$ lies in $\col(A)$, so $p = Ax_0$ for some $x_0$, and $b - p$ is orthogonal to $\col(A)$, in particular to each column, so $A^t(b - Ax_0) = 0$.
Conversely, let $\hat{x}$ be any least-squares solution. Then $\|b - A\hat{x}\| \le \|b - Ax_0\|$, while the Pythagorean equation above (with $x_0$ in the role of $\hat{x}$) gives $\|b - A\hat{x}\|^2 = \|b - Ax_0\|^2 + \|A(x_0 - \hat{x})\|^2$. Together these force $A(x_0 - \hat{x}) = 0$, so $A\hat{x} = Ax_0$ and $A^t(b - A\hat{x}) = A^t(b - Ax_0) = 0$. $\blacksquare$
This is a visual argument, not a formal proof. It leans on one fact from geometry class: the shortest way from a point to a plane is straight down, at a right angle. Watch the leftover $r = b - Ax$ and its dot products with the two columns. Anywhere but the foot, $r$ leans along the plane, so at least one dot product isn't $0$. At the foot both are $0$ at once, and those two dot products are the two entries of $A^tr$. So the closest point is where $A^t(b - A\hat{x}) = 0$, which rearranges to $A^tA\hat{x} = A^tb$. And a plane always has a point straight below $b$, so there's always a solution.
The full proof is in the n version.
That's the payoff: however many rows $A$ has, $A^tA$ is a small square matrix, one row and column per unknown, so the normal equations are a small square system. Here's the opening figure's line, computed.
Find the least-squares line $y = c + mx$ through $(0, 2)$, $(1, 0)$, and $(2, 3)$.
A line through all three would need $c + m \cdot 0 = 2$, $c + m \cdot 1 = 0$, $c + m \cdot 2 = 3$, which is $Ax = b$ with
$$A = \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}, \qquad x = \begin{pmatrix} c \\ m \end{pmatrix}, \qquad b = \begin{pmatrix} 2 \\ 0 \\ 3 \end{pmatrix}.$$Then $A^tA = \begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}$ and $A^tb = (5, 6)$, so the normal equations are $3c + 3m = 5$ and $3c + 5m = 6$. Subtracting, $2m = 1$, so $m = \frac12$, and then $3c = 5 - \frac32$ gives $c = \frac76$. The best line is $y = \frac76 + \frac12x$.
Check: $A\hat{x} = \left(\frac76, \frac53, \frac{13}{6}\right)$, so $r = b - A\hat{x} = \left(\frac56, -\frac53, \frac56\right)$, and $A^tr = \left(\frac56 - \frac53 + \frac56, \ 0 - \frac53 + \frac53\right) = (0, 0)$. The error is $\|r\|^2 = \frac{25 + 100 + 25}{36} = \frac{25}{6}$, the total the opening figure settled on.
Now both pictures side by side. On the left, the line and its three misses. On the right, $b = (2, 0, 3)$ and the plane of the columns, where the line $(c, m)$ is the point $A(c, m)$. The three misses are the three steps (along $x$, then $y$, then $z$) from that point to $b$, so shrinking the misses is shrinking the distance to $b$.
That last check, $A^tr = 0$, costs ten seconds and catches almost every arithmetic slip. Your turn. In free response, type the entries in order, separated by commas.
Loading practice engine...
The normal equations always have a solution. Whether they have only one depends on the columns of $A$, and the reason is one line of algebra: $A^tAx = 0$ forces $\|Ax\|^2 = x \cdot A^tAx = 0$, so $A^tA$ kills exactly the vectors $A$ kills.
The normal equations always have a solution. Whether they have only one depends on the columns of $A$: do they spread out into a plane, or lie along one line?
Let $A$ be an $m \times n$ matrix. The following are equivalent:
When they hold, $\hat{x} = (A^tA)^{-1}A^tb$. When they fail, every $b$ has infinitely many least-squares solutions, all with the same $A\hat{x}$.
First, $\N(A^tA) = \N(A)$. If $Ax = 0$ then $A^tAx = 0$. If $A^tAx = 0$ then $\|Ax\|^2 = (Ax) \cdot (Ax) = x \cdot (A^tAx) = 0$, so $Ax = 0$.
(2) $\iff$ (3): the $n \times n$ matrix $A^tA$ is invertible exactly when $\N(A^tA) = \{0\}$, which is $\N(A) = \{0\}$, which says the columns of $A$ are independent.
(3) $\implies$ (1): by the normal equations theorem, the least-squares solutions are the solutions of $A^tA\hat{x} = A^tb$, and with $A^tA$ invertible there is exactly one, $\hat{x} = (A^tA)^{-1}A^tb$.
If (2) fails, pick $z \neq 0$ with $Az = 0$. For any least-squares solution $\hat{x}$ (one exists), every $\hat{x} + tz$ also solves the normal equations, since $A^tA(\hat{x} + tz) = A^tA\hat{x}$, so (1) fails, and $A(\hat{x} + tz) = A\hat{x}$. Every least-squares solution has the same $A\hat{x}$ by the last step of the normal equations proof. $\blacksquare$
Let $A$ have three rows and two columns. If the columns don't lie on one line, then $Ax = b$ has exactly one least-squares solution, and it is $\hat{x} = (A^tA)^{-1}A^tb$. If the columns lie on one line, $Ax = b$ has infinitely many least-squares solutions, and they all give the same point $A\hat{x}$.
This is a visual argument, not a formal proof. Whatever the columns are, the possible $Ax$'s form a line or a plane through $0$, and there's exactly one point of it straight below $b$. The question is how many recipes $x = (x_1, x_2)$ cook up that point. If the columns lie on one line, many do: in the figure, $(2, 0)$, $(0, 1)$, $(-2, 2)$, and $(4, -1)$ all land on the same point. Swing the second column off the line and the two columns make a plane, where every point has exactly one pair of coordinates, so exactly one recipe works.
The full proof is in the n version.
The formula $\hat{x} = (A^tA)^{-1}A^tb$ is handy for theory, but by hand it's faster to row reduce the normal equations than to invert anything. Here's what dependent columns look like.
Find every least-squares solution of $Ax = b$ for $A = \begin{pmatrix} 1 & 2 \\ 1 & 2 \\ 0 & 0 \end{pmatrix}$ and $b = (1, 3, 2)$.
The second column is twice the first, so expect infinitely many. $A^tA = \begin{pmatrix} 2 & 4 \\ 4 & 8 \end{pmatrix}$ and $A^tb = (4, 8)$: both normal equations say $x_1 + 2x_2 = 2$. Every $\hat{x} = (2 - 2t, t)$ is a least-squares solution, and every one gives $A\hat{x} = (x_1 + 2x_2)(1, 1, 0) = (2, 2, 0)$, with residual $r = (-1, 1, 2)$, which is orthogonal to $(1, 1, 0)$ as it should be.
This is the example the picture above draws: the columns lie on the line through $(1, 1, 0)$, the foot from $b$ is $(2, 2, 0)$, and the recipes $(2, 0)$, $(0, 1)$, $(-2, 2)$, $(4, -1)$ are four of the $(2 - 2t, t)$'s.
So $A\hat{x}$, the best approximation of $b$, is always unique; only the recipe for it can fail to be. For fitting lines, that failure has a plain meaning, coming up next.
The most common least-squares problem in the world: data points $(x_1, y_1), \ldots, (x_k, y_k)$, and a line $y = c + mx$ meant to go near all of them. Each point misses the line by $y_i - (c + mx_i)$, measured straight up or down.
For data points $(x_1, y_1), \ldots, (x_k, y_k)$ whose $x$'s are not all equal, the line of best fit is the line $y = c + mx$ that makes $$(y_1 - c - mx_1)^2 + \cdots + (y_k - c - mx_k)^2$$ as small as possible.
That sum is $\|b - Ax\|^2$ for the matrix $A$ whose rows are $(1, x_i)$, the vector $b = (y_1, \ldots, y_k)$, and $x = (c, m)$. So the line of best fit is a least-squares solution, and "the $x$'s are not all equal" is exactly what makes the two columns of $A$ independent, so there's only one best line. Its normal equations always have the same shape:
$$\begin{pmatrix} k & \sum x_i \\ \sum x_i & \sum x_i^2 \end{pmatrix}\begin{pmatrix} c \\ m \end{pmatrix} = \begin{pmatrix} \sum y_i \\ \sum x_iy_i \end{pmatrix}.$$Four points this time.
Find the line of best fit through $(1, 2)$, $(2, 3)$, $(3, 5)$, and $(4, 6)$.
The sums are $k = 4$, $\sum x_i = 10$, $\sum x_i^2 = 30$, $\sum y_i = 16$, and $\sum x_iy_i = 2 + 6 + 15 + 24 = 47$. The normal equations are $4c + 10m = 16$ and $10c + 30m = 47$. Multiply the first by $\frac52$ to get $10c + 25m = 40$ and subtract: $5m = 7$, so $m = \frac75$, and then $4c = 16 - 14$ gives $c = \frac12$. The line is $y = \frac12 + \frac75x$.
The misses are $2 - 1.9 = 0.1$, $3 - 3.3 = -0.3$, $5 - 4.7 = 0.3$, and $6 - 6.1 = -0.1$, with squares adding up to $0.2$.
Four points means $b = (2, 3, 5, 6)$ has four entries, one more than space can draw. The plot of the points is still flat, though, and the recipe doesn't care how many points there are. Here's a different picture instead: the best line passes through the average point $(2.5, 4)$ (the Pro Tip below says why), so spin lines around that point and watch the total of the squared misses. It traces a parabola in the slope, with its bottom at $\frac75$.
Notice the misses added up to $0$. That's no accident, and it gives a shortcut.
The first normal equation, $kc + m\sum x_i = \sum y_i$, says the misses of the best line add up to $0$. Divide it by $k$: the line passes through the average point, (average of the $x$'s, average of the $y$'s). Shifting to that point gives the formula statistics books print: the slope is $\dfrac{\sum (x_i - \text{avg}_x)(y_i - \text{avg}_y)}{\sum (x_i - \text{avg}_x)^2}$, and then $c = \text{avg}_y - m \cdot \text{avg}_x$. For the four points: averages $2.5$ and $4$, slope $\frac{3 + 0.5 + 0.5 + 3}{2.25 + 0.25 + 0.25 + 2.25} = \frac75$.
The least-squares line is the one your eye would draw, if your eye could add squares. Try it: the shaded squares are the misses squared, and the score is their total.
The third mission shows the catch: squaring makes a big miss count enormously, so one wild data point can drag the whole line toward itself. Least squares trusts every point equally, outliers included.
Why squares, though? Why not just add up the sizes of the misses? Because squares are smooth, so calculus and dot products work on them, and because they give one formula with one answer. Adding sizes is a fine idea too. It just needs a computer and a lot more patience.
Your turn. In free response, type one number; decimals and fractions are both fine.
Loading practice engine...
Nothing in the normal equations cares that the columns were $1$ and $x$. Least squares only needs the unknowns to appear linearly: each one multiplies something you can compute from the data. A parabola $y = c + vt + at^2$ is curved in $t$, but it's linear in $c$, $v$, and $a$, so its matrix just has three columns, $1$, $t$, and $t^2$.
A ball is thrown straight up, and its height is measured once a second: $0.5$, $22.5$, $29.5$, and $31.5$ meters at $t = 0, 1, 2, 3$. Fit $y = c + vt + at^2$ by least squares, and estimate gravity.
The rows of $A$ are $(1, t, t^2)$ for each time, and $b$ lists the heights:
$$A = \begin{pmatrix} 1 & 0 & 0 \\ 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \end{pmatrix}, \quad A^tA = \begin{pmatrix} 4 & 6 & 14 \\ 6 & 14 & 36 \\ 14 & 36 & 98 \end{pmatrix}, \quad A^tb = \begin{pmatrix} 84 \\ 176 \\ 424 \end{pmatrix}.$$Row reducing the normal equations gives $c = 1$, $v = 25$, $a = -5$. (Check the first row: $4(1) + 6(25) + 14(-5) = 84$.) So $y = 1 + 25t - 5t^2$, with misses $-0.5$, $1.5$, $-1.5$, $0.5$. Physics says the height is $h_0 + v_0t - \frac{g}{2}t^2$, so $g \approx -2a = 10$ meters per second squared, close to the true $9.8$.
Here's the throw, the four measurements, and the parabola least squares picked. Each red tick is a miss.
Four measurements and three unknowns: $b$ has four entries and the column space is a three-dimensional slice of four-dimensional space, which nobody can draw. The algebra is exactly the same as the plane in space, though: drop the perpendicular, and the normal equations find its foot.
Exponentials need one trick. In $y = ae^{kx}$ the unknown $k$ sits in an exponent, which isn't linear. Take the logarithm and it is: $\ln y = \ln a + kx$, a line in the points $(x, \ln y)$. Any base works the same way. Hefferon has a nice example with the planets.
The average distances of the planets from the sun, in units of Earth's distance, are Mercury $0.39$, Venus $0.72$, Earth $1.00$, Mars $1.52$, Jupiter $5.20$, Saturn $9.54$, Uranus $19.2$. Number them $1$ to $8$, giving number $5$ to the asteroid belt (the leftovers of a planet that never formed), so Jupiter is $6$. Fit $d = a \cdot 10^{kn}$, and predict Neptune, planet $9$.
Fit a line to the points $(n, \log_{10} d)$: $(1, -0.41)$, $(2, -0.14)$, $(3, 0)$, $(4, 0.18)$, $(6, 0.72)$, $(7, 0.98)$, $(8, 1.28)$. Seven points, so let a computer form the normal equations. It gives $\log_{10} d \approx -0.678 + 0.237n$, which means $d \approx 0.21 \times 1.73^n$: each planet is about $1.73$ times farther out than the one before. For $n = 9$, $\log_{10} d \approx 1.457$, so $d \approx 28.7$. Neptune is really at $30.1$. Not bad for a rule nobody can explain! (It falls apart at Pluto: the rule says $49.5$, the truth is $39.5$.)
Watch the curve straighten out when the heights switch to logarithms.
The log turned a curve into a line, and least squares took it from there. One warning about that trick.
Fitting a line to $(x, \ln y)$ makes the misses in $\ln y$ small, not the misses in $y$. Those aren't the same goal: in logs, a miss of $1$ on a point near $y = 3$ counts as much as a miss of $30$ near $y = 100$. Usually that's what you want for growth data, but the answer is close to, not equal to, the best exponential measured in $y$ itself.
So which model? Look at the data first: does it bend, and how? Then let the error decide between models with the same number of unknowns.
A model with more unknowns never fits worse: a parabola can always pretend to be a line by setting its $x^2$ coefficient to $0$. So when two models score about the same, the simpler one is the better bet. Now fit one yourself. In free response, type one number.
Loading practice engine...
The normal equations have one weakness, and it's forming $A^tA$ at all. If the columns of $A$ point in nearly the same direction, $A$ is close to having dependent columns, and $A^tA$ is even closer to singular: it squares the condition number, the factor by which rounding errors get magnified, so a computer can lose twice as many digits as the problem deserves. That's why statistics software doesn't solve the normal equations. It uses the factorization $A = QR$ from Gram–Schmidt & QR instead.
Let $A$ have linearly independent columns, and let $A = QR$ where $Q$ has orthonormal columns and $R$ is square and upper triangular. Then $R$ is invertible, and the least-squares solution of $Ax = b$ is the one solution of $$R\hat{x} = Q^tb,$$ found by back substitution.
If $Rx = 0$ then $Ax = QRx = 0$, so $x = 0$ because the columns of $A$ are independent; a square matrix with only the trivial solution is invertible, so $R$ and $R^t$ are invertible. Orthonormal columns mean $Q^tQ = I$, so $A^tA = R^tQ^tQR = R^tR$ and $A^tb = R^tQ^tb$. The normal equations become $R^tR\hat{x} = R^tQ^tb$, and multiplying by $(R^t)^{-1}$ leaves $R\hat{x} = Q^tb$. Since $A^tA$ is invertible, this one solution is the least-squares solution. $\blacksquare$
This is a visual argument, not a formal proof. The columns of $Q$ are two perpendicular arrows of length $1$ lying in the plane of $A$'s columns, so they lay a square grid on that plane. On a square grid, finding the coordinates of the foot below $b$ takes no system at all: each coordinate is a shadow, $q_1 \cdot b$ and $q_2 \cdot b$, and that pair is $Q^tb$. All that's left is translating grid steps into steps along $a_1$ and $a_2$, and that's what the triangular $R$ does.
The full proof is in the n version.
The same line as before, the QR way, as a check that both roads lead to the same place.
Gram–Schmidt turns $A = \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}$ into $A = QR$ with $Q = \begin{pmatrix} 1/\sqrt3 & -1/\sqrt2 \\ 1/\sqrt3 & 0 \\ 1/\sqrt3 & 1/\sqrt2 \end{pmatrix}$ and $R = \begin{pmatrix} \sqrt3 & \sqrt3 \\ 0 & \sqrt2 \end{pmatrix}$. Use it to fit the line through $(0, 2)$, $(1, 0)$, $(2, 3)$ again.
With $b = (2, 0, 3)$, $Q^tb = \left(\frac{2 + 0 + 3}{\sqrt3}, \frac{-2 + 0 + 3}{\sqrt2}\right) = \left(\frac{5}{\sqrt3}, \frac{1}{\sqrt2}\right)$. Back substitution in $R\hat{x} = Q^tb$: the bottom row says $\sqrt2\,m = \frac{1}{\sqrt2}$, so $m = \frac12$. The top row says $\sqrt3\,c + \sqrt3\,m = \frac{5}{\sqrt3}$, so $c + m = \frac53$ and $c = \frac76$. Same line, $y = \frac76 + \frac12x$, and $A^tA$ never appeared.
This is the example the picture above draws: the grid of $q_1$ and $q_2$, the shadows $\frac{5}{\sqrt3} \approx 2.89$ and $\frac{1}{\sqrt2} \approx 0.71$, and the recipe $\frac76a_1 + \frac12a_2$ landing on the same foot.
By hand the square roots make QR look like more work, and for three data points it is. For a million rows and a computer it's the safer road. Your turn, with friendlier numbers: type the entries in order, separated by commas.
Loading practice engine...
For decades, running a mile in under four minutes was thought impossible, and people argued about whether it would ever happen. Hefferon's book has the world record on January 1 of each decade from 1870 to 1950: $268.8$, $264.5$, $258.4$, $255.6$, $255.6$, $252.6$, $250.4$, $246.4$, and $241.4$ seconds. Could a 1950 sports fan have predicted when the record would hit $240$?
Write $t$ for years since 1900. The line $y = c + mt$ through all nine points is least squares with nine rows of $(1, t)$, a job for a computer, which gives $c \approx 257.9$ and $m \approx -0.305$: the record was dropping about $0.3$ seconds a year. Setting $257.9 - 0.305t = 240$ gives $t \approx 58.7$, late in 1958.
Roger Bannister ran $3$:$59.4$ on May 6, 1954, four years ahead of the line. A prediction from a line of best fit is a guess with a good argument behind it, not a promise. The same tool shows up wherever there's data: Hefferon fits baseball salaries against wins and finds that an extra million dollars bought a team about a quarter of a win in 2002.
Run that line forward to the year 2746 and the mile record hits zero seconds. After that, runners finish before they start. Lines of best fit are great at describing the data you have and terrible at the far future.
Where this goes next: the matrix $A^tA$ that kept showing up is symmetric, and symmetric matrices have their own beautiful theory, coming up in the Spectral Theorem. Later still, principal component analysis fits not a line to points in the plane but the best low-dimensional picture of data with hundreds of columns, and it's least squares again underneath.
Every question so far came right after its idea, which is a hint a quiz never gives you. Down here the hint is gone. First, true or false: a claim from anywhere on this page, and you decide whether it holds.
Loading practice engine...
Next, a mixed set: every question type on this page, shuffled. Type a number or the entries when it asks you to.
Loading practice engine...
Made it this far? Time for something with a little heat: a smudged lab notebook, and a line of best fit that knows what the smudge said. In free response, type a number.
Loading practice engine...