Linear Least Squares Fitting

What is Linear Least Squares Fitting?

Let (x₁, y₁), (x₂, y₂)... (x_N, y_N) be experimental data points as shown in the scatter plot below and suppose we want to predict the dependent variable y for different values of the independent variable x using a linear model of the form

y = a x + b

scatter plot of data points — Figure 1. scatter plot

A widely used procedure in mathematics is to minimize the sum D of the squares of the vertical distances d1, d2, ... between the mathematical model y = f(x) and the experimental points as shown in the graph below.

least square fitting a model to data — Figure 2. Least square fitting a model to data

Let \( f(x) = a x + b \) be the linear model to be used. Hence the vertical distances \(d_1, d_2, ... \) are given by

\( d_1 = |y_1 - (a x_1 + b))| \) , \( d_2 = |y_2 - (a x_2 + b))| \) , ...

The sum D of the squares of the vertical distances d1, d2, ... may be written as
\[ D = \sum_{i=1}^{N} (y_i - (a x_i + b))^2 \] The values of a and b that minimize D are the values that make the partial derivatives of D with respect to a and b simultaneously equal to 0. Hence we first calculate the two derivatives: \[ \dfrac {\partial D}{\partial a} = \sum_{i=1}^{N} - 2 x_i(y_i - a x_i - b) \] \[ \dfrac {\partial D}{\partial b } = \sum_{i=1}^{N} - 2 (y_i - a x_i - b) \] then solve for \( a \) and \( b \) the system of equations
\begin{cases} \sum_{i=1}^{N} - 2 x_i(y_i - a x_i - b) = 0 \\ \sum_{i=1}^{N} - 2 (y_i - a x_i - b) = 0 \end{cases} Divide both sides of each equation by - 2 and simplify to rewrite the system of equations as
\begin{cases} \sum_{i=1}^{N} x_i(y_i - a x_i - b) = 0 \\ \sum_{i=1}^{N} (y_i - a x_i - b) = 0 \end{cases} It is a system of equations with the two unknowns a and b. Expands the sums.
\begin{cases} \sum_{i=1}^{N} x_iy_i - a \sum_{i=1}^{N} x_i x_i - \sum_{i=1}^{N} x_i b = 0 \\ \sum_{i=1}^{N} y_i - a\sum_{i=1}^{N} x_i - \sum_{i=1}^{N} b = 0 \end{cases} We now rewrite the system of equations terms containing a and b on the left and all other terms on the right as follows
\begin{cases} a \sum_{i=1}^{N} x_i^2 + b \sum_{i=1}^{N} x_i = \sum_{i=1}^{N} x_iy_i \\ a\sum_{i=1}^{N} x_i + b N = \sum_{i=1}^{N} y_i \end{cases} The above system in matrix form is written as \[ \begin{bmatrix} \sum_{i=1}^{N} x_i^2 & \sum_{i=1}^{N} x_i \\ \sum_{i=1}^{N} x_i & N \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} \sum_{i=1}^{N} x_iy_i \\ \sum_{i=1}^{N} y_i \end{bmatrix} \] The above system may be solved using the inverse matrix as follows \[ \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} \sum_{i=1}^{N} x_i^2 & \sum_{i=1}^{N} x_i \\ \sum_{i=1}^{N} x_i & N \end{bmatrix}^{-1} \begin{bmatrix} \sum_{i=1}^{N} x_iy_i \\ \sum_{i=1}^{N} y_i \end{bmatrix} \] where we use the inverse of a 2 by 2 matrix to find \[ \begin{bmatrix} \sum_{i=1}^{N} x_i^2 & \sum_{i=1}^{N} x_i \\ \sum_{i=1}^{N} x_i & N \end{bmatrix}^{ -1} = \dfrac{1}{N \sum_{i=1}^{N} x_i^2 - (\sum_{i=1}^{N} x_i)^2} \begin{bmatrix} N & - \sum_{i=1}^{N} x_i \\ - \sum_{i=1}^{N} x_i & \sum_{i=1}^{N} x_i^2 \end{bmatrix} \] Finally \[ \begin{bmatrix} a \\ b \end{bmatrix} = \dfrac{1}{N \sum_{i=1}^{N} x_i^2 - (\sum_{i=1}^{N} x_i)^2} \begin{bmatrix} N & - \sum_{i=1}^{N} x_i \\ - \sum_{i=1}^{N} x_i & \sum_{i=1}^{N} x_i^2 \end{bmatrix} \begin{bmatrix} \sum_{i=1}^{N} x_iy_i \\ \sum_{i=1}^{N} y_i \end{bmatrix} \] where \[ a = \dfrac{N \sum_{i=1}^{N} x_iy_i - (\sum_{i=1}^{N} x_i)(\sum_{i=1}^{N} y_i)} {N \sum_{i=1}^{N} x_i^2 - (\sum_{i=1}^{N} x_i)^2} \]
\[ b = \dfrac{ - (\sum_{i=1}^{N} x_i) (\sum_{i=1}^{N} x_i y_i) + (\sum_{i=1}^{N} x_i^2)( \sum_{i=1}^{N}y_i) } {N \sum_{i=1}^{N} x_i^2 - (\sum_{i=1}^{N} x_i)^2} \]

Example of Linear Least Squares Fitting Application

Find the linear least square fit y = a x + b for the experimental data points given by: {(1 , 2) , (3 , 4) , (2 , 6) , (4 , 8) , (5 , 12) , (6 , 13) , (7 , 15)}
Solution
Set up a table with the quantities included in the above formulas for m and b.

\(x_i\)	\(y_i\)	\(x_i\) \(y_i\)	\(x_i^{2}\)
1	2	2	1
3	4	12	9
2	6	12	4
4	8	32	16
5	12	60	25
6	13	78	36
7	15	105	49

The total number of points is N = 7.

\( \sum_{i=1}^{7} x_i \) = 1 + 3 + 2 + 4 + 5 + 6 + 7 = 28

\( \sum_{i=1}^{7} y_i \) = 2 + 4 + 6 + 8 + 12 + 13 + 15 = 60

\( \sum_{i=1}^{7} x_iy_i \) = 2 + 12 + 12 + 32 + 60 + 78 + 105 = 301

\( \sum_{i=1}^{7} x_i^2 \) = 1 + 9 + 4 + 16 + 25 + 36 + 49 = 140

Substitute to obtain

\( a = \dfrac{7 \times 301 - 28 \times 60} {7 \times 140 - 28^2} = 2.17857142857 \)

\( b = \dfrac{ - 28 \times 301 + 140 \times 60 } {7 \times 140 - 28^2} = -0.14285714285 \)