Straight Line Least Squares Fit

Michael O. Duffy

28 Oct 2009

Assume that you have n points:

(x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ) (1)

The points are distributed randomly in such a way that you can’t draw a single straight line that
passes through all the points. You’d like to know the ”best” equation for a line to approximate the
data you have:

y = mx + b (2)

where m is the slope and b is the y-intercept.

Substituting the n points into the equation for the line gives n equations with two unknowns: the
slope and y-intercept:

y1 = mx1 + b
y2 = mx2 + b
.. (3)
.
yn = mxn + b

We can rewrite this in terms of a matrix equation:

   
x1 1  y1 
 x2    
1 m  y2
  

= (4)

 .. ..
 b

 . 
 . 

 
xn 1 yn
 

Remember that multiplying two matricies Amn and Bnp together means adding the products of
each element in a row of Amn in succession, multiplying it by the value in the corresponding column
in Bnp :

1
 2

n
X
Cmp = Ami Bip (5)
i=1

We’ll assume that the coefficients of the ”best” line are those that minimize the sum of squares of
the errors between the data point yi and mxi + b, the value predicted by the line at the point xi .
For now, we’ll take it on faith that we can calculate those ”best” coefficients by pre-multiplying
both sides by the transpose of the matrix:

   
x1 1  y1 
 
. . . x n  x2 1 m
      y2
 
x1 x2 x1 x2 . . . xn
 
 .. = .. (6)
1 1 ... 1  .  b 1 1 ... 1 


 . 


xn 1 yn
 

I can multiply these matricies and reduce them to a 2x2 matrix equation, with the product of a 2x2
matrix and a 2x1 vector of unknowns equal to a 2x1 right-hand side vector:

Ac̃ = ỹ (7)

where

Pn 2
Pn 
i=1 (x i ) i=1 (x i )
A = Pn (8)
i=1 (xi ) n

and

 
m
c̃ = (9)
b

and

Pn 
Pi=1 (xi yi )
ỹ = n (10)
i=1 (yi )

Now I can calculate the solution by multiplying both sides of the matrix equation by the inverse of
A:

A−1 Ac̃ = A−1 ỹ (11)

The product of A−1 A is the identity matrix:

A−1 A = I (12)
 3

where

 
1 0
I= (13)
0 1

Since the product I c̃ = c̃, we have an expression for the vector of unknown coefficients:

c̃ = A−1 ỹ (14)

I can write out the expression for A−1 using determinants:

n
X n
X
det(A) = n (xi )2 − ( (xi ))2 (15)
i=1 i=1

We can write out the inverse of A explicitly:

 Pn 
Pn − i=1 (x i )
n Pn 2
− i=1 (xi ) i=1 (xi )
A−1 = (16)
det(A)

We can check this result by multiplying the matrix by its inverse and seeing if we get the identity
matrix back:

Pn  Pn Pn
(xi )2
 
Pnn −Pn i=1 (xi2) Pi=1 i=1 (xi )
n
− i=1 (xi ) (xi ) (xi ) n
A−1 A = Pni=1 2 Pn i=1 2
(17)
n i=1 (xi ) − ( i=1 (xi ))

Performing the multiplication gives the desired result:

 Pn
n i=1 (xi )2 − ( ni=1 (xi ))2
P 
0 P
n i=1 (xi )2 − ( ni=1 (xi ))2
Pn
0
A−1 A = (18)
n i=1 (xi )2 − ( ni=1 (xi ))2
Pn P

Solving for the coefficients:

 Pn  Pn 
Pnn −
Pn i=1 (xi2) Pi=1 (xi yi )
n
− i=1 (xi ) i=1 (xi ) i=1 (yi )
c̃ = (19)
det(A)

Simplifying:

Pn
(xi yi ) − ( ni=1 (x
 P Pn 
Pn n i=1P i ))( i=1 (yPi ))
−( i=1 (xi ))( ni=1 (xi yi )) + ( ni=1 (xi )2 )( ni=1 (yi ))
P
c̃ = (20)
det(A)
 4

Define the following symbols:

n
X
SX = (xi ) (21)
i=1
Xn
SY = (yi ) (22)
i=1
Xn
SXX = (xi )2 (23)
i=1
Xn
SXY = (xi yi ) (24)
i=1

Now we can write the final expressions for the slope and y-intercept of the ”best” line:

det(A) = nSXX − SX 2 (25)

nSXY − (SX)(SY )
m = (26)
nSXX − SX 2
(SXX)(SY ) − (SX)(SXY )
b = (27)
nSXX − SX 2