Linear Least Squares Fitting

Optimization, Normal Equations, Mathematical Derivation, and Interactive Calculator

What is Linear Least Squares Fitting?

Let \( (x_1, y_1), (x_2, y_2), \dots, (x_N, y_N) \) be experimental data points as shown in the scatter plot below and suppose we want to predict the dependent variable \( y \) for different values of the independent variable \( x \) using a linear model of the form:

\[ y = ax + b \]
scatter plot of data points
Figure 1. Scatter plot of experimental data

A widely used procedure in mathematics is to minimize the sum \( D \) of the squares of the vertical distances \( d_1, d_2, \dots \) between the mathematical model \( y = f(x) \) and the experimental points as shown in the graph below.

least square fitting a model to data
Figure 2. Least squares fitting model to data

Mathematical Derivation

Let \( f(x) = ax + b \) be the linear model to be used. Hence the vertical distances \( d_1, d_2, \dots \) are given by:

\[ d_1 = |y_1 - (ax_1 + b)|, \quad d_2 = |y_2 - (ax_2 + b)|, \quad \dots \]

The sum \( D \) of the squares of the vertical distances may be written as:

\[ D = \sum_{i=1}^{N} (y_i - (ax_i + b))^2 \]

The values of \( a \) and \( b \) that minimize \( D \) are the values that make the partial derivatives of \( D \) with respect to \( a \) and \( b \) simultaneously equal to 0. Hence, we calculate the partial derivatives:

\[ \dfrac{\partial D}{\partial a} = \sum_{i=1}^{N} -2x_i(y_i - ax_i - b) \] \[ \dfrac{\partial D}{\partial b} = \sum_{i=1}^{N} -2(y_i - ax_i - b) \]

Setting both derivatives to zero and dividing by \( -2 \) yields the system of normal equations:

\[ \begin{cases} \sum_{i=1}^{N} x_i(y_i - ax_i - b) = 0 \\ \sum_{i=1}^{N} (y_i - ax_i - b) = 0 \end{cases} \]

Expanding the sums:

\[ \begin{cases} \sum_{i=1}^{N} x_iy_i - a \sum_{i=1}^{N} x_i^2 - b \sum_{i=1}^{N} x_i = 0 \\ \sum_{i=1}^{N} y_i - a \sum_{i=1}^{N} x_i - Nb = 0 \end{cases} \]

Rearranging terms with \( a \) and \( b \) on the left:

\[ \begin{cases} a \sum_{i=1}^{N} x_i^2 + b \sum_{i=1}^{N} x_i = \sum_{i=1}^{N} x_iy_i \\ a \sum_{i=1}^{N} x_i + Nb = \sum_{i=1}^{N} y_i \end{cases} \]

In matrix form:

\[ \begin{bmatrix} \sum_{i=1}^{N} x_i^2 & \sum_{i=1}^{N} x_i \\ \sum_{i=1}^{N} x_i & N \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} \sum_{i=1}^{N} x_iy_i \\ \sum_{i=1}^{N} y_i \end{bmatrix} \]

Using the inverse matrix formula for a \( 2 \times 2 \) matrix, we solve for \( a \) and \( b \):

\[ a = \dfrac{N \sum_{i=1}^{N} x_iy_i - (\sum_{i=1}^{N} x_i)(\sum_{i=1}^{N} y_i)}{N \sum_{i=1}^{N} x_i^2 - (\sum_{i=1}^{N} x_i)^2} \] \[ b = \dfrac{-(\sum_{i=1}^{N} x_i)(\sum_{i=1}^{N} x_iy_i) + (\sum_{i=1}^{N} x_i^2)(\sum_{i=1}^{N} y_i)}{N \sum_{i=1}^{N} x_i^2 - (\sum_{i=1}^{N} x_i)^2} \]

Example of Linear Least Squares Fitting Application

Find the linear least squares fit \( y = ax + b \) for the experimental data points given by:

\[ \{(1, 2), (3, 4), (2, 6), (4, 8), (5, 12), (6, 13), (7, 15)\} \]
Show Solution to Example

Set up a table with the necessary quantities:

\( x_i \) \( y_i \) \( x_i y_i \) \( x_i^2 \)
1 2 2 1
3 4 12 9
2 6 12 4
4 8 32 16
5 12 60 25
6 13 78 36
7 15 105 49

The total number of points is \( N = 7 \).

\( \sum_{i=1}^{7} x_i = 1 + 3 + 2 + 4 + 5 + 6 + 7 = 28 \)

\( \sum_{i=1}^{7} y_i = 2 + 4 + 6 + 8 + 12 + 13 + 15 = 60 \)

\( \sum_{i=1}^{7} x_iy_i = 2 + 12 + 12 + 32 + 60 + 78 + 105 = 301 \)

\( \sum_{i=1}^{7} x_i^2 = 1 + 9 + 4 + 16 + 25 + 36 + 49 = 140 \)

Substituting into the formulas:

\[ a = \dfrac{7 \times 301 - 28 \times 60}{7 \times 140 - 28^2} = \dfrac{2107 - 1680}{980 - 784} = \dfrac{427}{196} \approx 2.1786 \] \[ b = \dfrac{-28 \times 301 + 140 \times 60}{7 \times 140 - 28^2} = \dfrac{-8428 + 8400}{196} = \dfrac{-28}{196} \approx -0.1429 \]

Linear Least Squares Fitting Calculator

Given experimental points, this calculator computes coefficients \( a \) and \( b \), the line equation \( y = ax + b \), and the correlation coefficient. It also plots the experimental points alongside the best-fit line.

Enter points as \((x_1, y_1), (x_2, y_2), \dots\) separated by commas, or use the default dataset.

Hover over the top-right corner of the plot to download the graph in PNG format.

More References and Links