Numerical Analysis

Numerical Analysis
Root Finding

Root Finding Topics

Bi-section Method
Newtons method
Uses of root finding for sqrt() and reciprocal sqrt()
Secant Method
Generalized Newtons method for systems of nonlinear equations
The Jacobian matrix

Fixed-point formulas, Basins of Attraction and

Fractals.
3

Motivation
Many problems can be re-written into a
form such as:
f(x,y,z,) = 0
f(x,y,z,) = g(s,q,)

Motivation
A root, r, of function f occurs when f(r) = 0.
For example:
f(x) = x2 2x 3
has two roots at r = -1 and r = 3.
f(-1) = 1 + 2 3 = 0
f(3) = 9 6 3 = 0

We can also look at f in its factored form.

f(x) = x2 2x 3 = (x + 1)(x 3)
January 30, 2016

Factored Form of Functions

The factored form is not limited to
polynomials.
Consider:
f(x)= x sin x sin x.

A root exists at x = 1.
f(x) = (x 1) sin x

Or,
f(x) = sin x => x (x 1) (x 2)
January 30, 2016

Examples
Find x, such that
xp = c, xp c = 0
Calculate the sqrt(2)

x 22 22=00 x 2 x 2

Ballistics
Determine the horizontal distance at which the
projectile will intersect the terrain function.

January 30, 2016

Root Finding Algorithms

Closed or Bracketed techniques
Bi-section
Regula-Falsi

Open techniques
Newton fixed-point iteration
Secant method

Multidimensional non-linear problems

The Jacobian matrix

Fixed-point iterations
Convergence and Fractal Basins of Attraction
January 30, 2016

Bisection Method
Based on the fact that the function will
change signs as it passes thru the root.
f(a)*f(b) < 0

Once we have a root bracketed, we simply

evaluate the mid-point and halve the
interval.

January 30, 2016

Bisection Method
c=(a+b)/2
f(a)>0

f(c)>0

f(b)<0
January 30, 2016

10

Bisection Method
Guaranteed to converge to a root if one
exists within the bracket.

a=c
f(a)>0

f(c)<0
January 30, 2016

f(b)<0
11

Bisection Method
Slowly converges to a root

b=c
f(b)<0
a

January 30, 2016

12

Bisection Method
Simple algorithm:
Given: a and b, such that f(a)*f(b)<0
Given: error tolerance, err
c=(a+b)/2.0; // Find the midpoint
While( |f(c)| > err ) {
if( f(a)*f(c) < 0 ) // root in the left half
b = c;
else
// root in the right half
a = c;
c=(a+b)/2.0; // Find the new midpoint
}
return c;
January 30, 2016

13

Relative Error
We can develop an upper bound on the
relative error quite easily.
ba
x c

,a x
a
x

January 30, 2016

14

Absolute Error
What does this mean in binary mode?
err0 |b-a|
erri+1 erri/2 = |b-a|/2i+1

We gain an extra bit each iteration!!!

To reach a desired absolute error tolerance:
erri+1 errtol

January 30, 2016

ba
errtol
n
2
b a
n log 2

err
tol

15

Absolute Error
The bisection method converges linearly or
first-order to the root.
If we need an accuracy of 0.0001 and our
initial interval (b-a)=1, then:
2-n < 0.0001 14 iterations

Not bad, why do I need anything else?

January 30, 2016

16

A Note on Functions
Functions can be simple, but I may need to
evaluate it many many times.
Or, a function can be extremely
complicated. Consider:
Interested in the configuration of air vents (position,
orientation, direction of flow) that makes the
temperature in the room at a particular position
(teachers desk) equal to 72.
Is this a function?
January 30, 2016

17

A Note on Functions
This function may require a complex threedimensional heat-transfer coupled with a
fluid-flow simulation to evaluate the
function. hours of computational time on
a supercomputer!!!
May not necessarily even be computational.
Techniques existed before the Babylonians.
January 30, 2016

18

Root Finding Algorithms

Closed or Bracketed techniques
Bi-section
Regula-Falsi

Open techniques
Newton fixed-point iteration
Secant method

Multidimensional non-linear problems

The Jacobian matrix

Fixed-point iterations
Convergence and Fractal Basins of Attraction
January 30, 2016

OSU/CIS 541

19

Regula Falsi
In the book under computer problem 16 of
section 3.3.
Assume the function is linear within the
bracket.
Find the intersection of the line with the xaxis.

January 30, 2016

20

Regula Falsi
f (a ) f (b)
y ( x) f b
( x b)
a b
f (a ) f (b)
y (c ) 0 f b
(c b )
a b
a b
0 f (b)
c b
f (a ) f (b)
f (b) a b
c b
f (a ) f (b)
f(c)<0
a

January 30, 2016

21

Regula Falsi
Large benefit when the root is much closer
to one side.
Do I have to worry about division by zero?

January 30, 2016

22

Regula Falsi
More generally, we can state this method as:
c=wa + (1-w)b

For some weight, w, 0w 1.

If |f(a)| >> |f(b)|, then w < 0.5
Closer to b.

January 30, 2016

23

Bracketing Methods

Bracketing methods are robust

Convergence typically slower than open methods
Use to find approximate location of roots
Polish with open methods
Relies on identifying two points a,b initially such
that:
f(a) f(b) < 0

Guaranteed to converge
January 30, 2016

24

Root Finding Algorithms

Closed or Bracketed techniques
Bi-section
Regula-Falsi

Open techniques
Newton fixed-point iteration
Secant method

Multidimensional non-linear problems

The Jacobian matrix

Fixed-point iterations
Convergence and Fractal Basins of Attraction
January 30, 2016

OSU/CIS 541

25

Newtons Method
Open solution, that requires only one
current guess.
Root does not need to be bracketed.
Consider some point x0.
If we approximate f(x) as a line about x0, then
we can again solve for the root of the line.

l ( x) f ( x0 )( x x0 ) f ( x0 )

January 30, 2016

26

Newtons Method
Solving, leads to the following iteration:
l ( x) 0
f ( x0 )
x1 x0
f ( x0 )
f ( xi )
xi 1 xi
f ( xi )
January 30, 2016

27

Newtons Method
This can also be seen from Taylors Series.
Assume we have a guess, x0, close to the
actual root. Expand f(x) about this point.
x xi x
x 2
f ( xi x ) f ( xi ) xf ( xi )
f ( xi ) L 0
2!

If dx is small, then dxn quickly goes to zero.

x xi 1 xi
January 30, 2016

f ( xi )
f ( xi )
28

Newtons Method
Graphically, follow the tangent vector down
to the x-axis intersection.

xi

January 30, 2016

xi+1

29

Newtons Method
Problems

diverges
x0

2
January 30, 2016

30

Newtons Method
Need the initial guess to be close, or, the
function to behave nearly linear within the
range.

January 30, 2016

31

Finding a square-root
Ever wonder why they call this a squareroot?
Consider the roots of the equation:
f(x) = x2-a

This of course works for any power:

p

a x a 0, p R

January 30, 2016

32

Finding a square-root
Example: 2 = 1.4142135623730950488016887242097
Let x0 be one and apply Newtons method.
f ( x) 2 x

x 2
xi 1 xi i

2 xi

1
xi
2

xi

x0 1
1
2
3
x1 1 1.5000000000
2
1
2
1 3 4
17
x2
1.4166666667
2 2 3
12
January 30, 2016

33

Finding a square-root
Example: 2 = 1.4142135623730950488016887242097
Note the rapid convergence
1 17 24
577
x3

1.414215686
2 12 17
408
x4 1.4142135623746
((
x5 1.4142135623730950488016896
((
x6 1.4142135623730950488016887242097

Note, this was done with the standard

Microsoft calculator to maximum precision.
January 30, 2016

34

Finding a square-root
Can we come up with a better initial guess?
Sure, just divide the exponent by 2.
Remember the bias offset
Use bit-masks to extract the exponent to an
integer, modify and set the initial guess.

For 2, this will lead to x0=1 (round down).

January 30, 2016

35

Convergence Rate of Newtons

en x xn or x xn en
0 f ( x ) f ( xn en )
1
f ( xn en ) f ( xn ) en f ( xn ) en2 f ( n ), for some n x , xn
2
1
f ( xn ) en f ( xn ) en2 f ( n )
2

Now,

en 1 x xn 1 x xn

f ( xn )
f ( xn )
en
f ( xn )
f ( xn )

en f ( xn ) f ( xn )
f ( xn )

1 f ( n ) 2
en 1
en
2 f ( xn )
January 30, 2016

36

Convergence Rate of Newtons

Converges quadratically.

if en 10

en 1 c10

January 30, 2016

then,
2 k

37

Newtons Algorithm
Requires the derivative function to be evaluated,
hence more function evaluations per iteration.
A robust solution would check to see if the
iteration is stepping too far and limit the step.
Most uses of Newtons method assume the
approximation is pretty close and apply one to
three iterations blindly.

January 30, 2016

38

Division by Multiplication
Newtons method has many uses in
computing basic numbers.
For example, consider the equation:
1
a 0
x

Newtons method gives the iteration:

1
a
x
xk 1 xk k
xk xk axk2
1
2
xk
xk 2 axk

January 30, 2016

39

Reciprocal Square Root

Another useful operator is the reciprocalsquare root.
Needed to normalize vectors
Can be used to calculate the square-root.
1
a
a
a

January 30, 2016

40

Reciprocal Square Root

1
Let f ( x) 2 a 0
x
2

f ( x) 3
x

Newtons iteration yields:

xk
xk3
xk 1 xk a
2
2
1
xk 3 axk2
2
January 30, 2016

41

1/Sqrt(2)
Lets look at the convergence for the
reciprocal square-root of 2.
x0 1

x1 0.5 1 3 2 12 0.5
x2 0.5 0.5 3 2 0.5

0.625

x3 0.693359375
x4 0.706708468496799468994140625

If we could
only start
here!!

x 5 0.707106444695907075511730676593228
x 6 0.707106781186307335925435931237738
x 7 0.7071067811865475244008442397
2481
(
January 30, 2016

42

1/Sqrt(x)
What is a good choice for the initial seed
point?
Optimal the root, but it is unknown
Consider the normalized format of the number:

2e 127 (1.m) 2

What is the reciprocal?

What is the square-root?
January 30, 2016

43

1/Sqrt(x)
Theoretically,

1
1
1

127
2
x 2 1.m 2 2

e
1
127
3127

1.m 2 2 2

1.m

1
2

127 e
2

New bit-pattern
for the exponent

Current GPUs provide this operation in as little as

2 clock cycles!!! How?
How many significant bits does this estimate have?
January 30, 2016

44

1/Sqrt(x)
GPUs such as nVidias FX cards provide a
23-bit accurate reciprocal square-root in
two clock cycles, by only doing 2 iterations
of Newtons method.
Need 24-bits of precision =>
Previous iteration had 12-bits of precision
Started with 6-bits of precision
January 30, 2016

45

1/Sqrt(x)
Examine the mantissa term again (1.m).
Possible patterns are:
1.000, 1.100, 1.010, 1.110,

Pre-compute these and store the results in a table.

Fast and easy table look-up.
A 6-bit table look-up is only 64 words of on chip
cache.
Note, we only need to look-up on m, not 1.m.
This yields a reciprocal square-root for the first
seven bits, giving us about 6-bits of precision.
January 30, 2016

46

1/Sqrt(x)
Slight problem:

The 1.m produces a result between 1 and 2.

Hence, it remains normalized, 1.m.
1
For x , we get a number between and 1.
Need to shift the exponent.

January 30, 2016

47

Root Finding Algorithms

Closed or Bracketed techniques
Bi-section
Regula-Falsi

Open techniques
Newton fixed-point iteration
Secant method

Multidimensional non-linear problems

The Jacobian matrix

Fixed-point iterations
Convergence and Fractal Basins of Attraction
January 30, 2016

OSU/CIS 541

48

Secant Method
What if we do not know the derivative of
f(x)?
Secant line
Tangent vector

xi
January 30, 2016

xi-1
49

Secant Method
As we converge on the root, the secant line
approaches the tangent.
Hence, we can use the secant line as an
estimate and look at where it intersects the
x-axis (its root).

January 30, 2016

50

Secant Method
This also works by looking at the definition of the
f ( x h) f ( x)
derivative: f ( x)
lim
h
h 0

f ( xk )

f ( xk ) f ( xk 1 )
xk xk 1

Therefore, Newtons method gives:

xk xk 1
xk 1 xk
f ( xk )
f ( xk ) f ( xk 1 )

Which is the Secant Method.

January 30, 2016

51

Convergence Rate of Secant

Using Taylors Series, it can be shown
(proof is in the book) that:
ek 1 x xk 1
1 f ( k )

ek ek 1 c ek ek 1
2 f ( k )

January 30, 2016

52

Convergence Rate of Secant

This is a recursive definition of the error
term. Expressed out, we can say that:
ek 1 C ek

Where =1.62.
We call this super-linear convergence.
January 30, 2016

53

Root Finding Algorithms

Closed or Bracketed techniques
Bi-section
Regula-Falsi

Open techniques
Newton fixed-point iteration
Secant method

Multidimensional non-linear problems

The Jacobian matrix

Fixed-point iterations
Convergence and Fractal Basins of Attraction
January 30, 2016

OSU/CIS 541

54

Higher-dimensional Problems
Consider the class of functions
f(x1,x2,x3,,xn)=0,
where we have a mapping from n.
We can apply Newtons method separately
for each variable, xi, holding the other
variables fixed to the current guess.

January 30, 2016

55

Higher-dimensional Problems
This leads to the iteration:

f x1 , x2 ,K , xn
xi xi
f xi x1 , x2 ,K , xn

Two choices, either I keep of complete set of old

guesses and compute new ones, or I use the new ones
as soon as they are updated.
Might as well use the more accurate new guesses.
Not a unique solution, but an infinite set of solutions.
January 30, 2016

56

Higher-dimensional Problems
Example:
x+y+z=3

Solutions:
x=3, y=0, z=0
x=0, y=3, z=0

January 30, 2016

57

Systems of Non-linear Equations

Consider the set of equations:
f1 x1 , x2 ,K , xn 0

f 2 x1 , x2 ,K , xn 0
M
f n x1 , x2 ,K , xn 0

January 30, 2016

58

Systems of Non-linear Equations

Example:
x yz 3
x2 y 2 z 2 5

Plane intersected with a

sphere, intersected with a
more complex function.

e x xy xz 1

Conservation of mass coupled with

conservation of energy, coupled with
solution to complex problem.
January 30, 2016

59

Vector Notation
We can rewrite this using vector notation:
r r r
f ( x) 0

f f1 , f 2 ,K , f n
x x1 , x2 ,K , xn

January 30, 2016

60

Newtons Method for

Non-linear Systems
Newtons method for non-linear systems
can be written as:
x

( k 1)

(k )

f x
(k )

f x( k )

(k )

where f x is the Jacobian matrix

January 30, 2016

61

The Jacobian Matrix

The Jacobian contains all the partial derivatives of
the set of functions.
f1
x
1

f1
x2

f 2
J x1
M

f n
x1

f 2
x2

M O
f n
x2

f1
xn

f 2
xn

f n
xn

Note, that these are all functions and need to be

evaluated at a point to be useful.
January 30, 2016

62

The Jacobian Matrix

Hence, we write
f1 ( i )
x x
1

f1 ( i )
x L
x2

f 2 ( i )
x

(i )
J (x ) x1

f n x (i )
x1

f 2 ( i )
x L

x2

January 30, 2016

f n ( i )
x L

x2

f1 (i )
x
xn

f 2 (i )
x

xn

f n (i )
x

xn

63

Matrix Inverse
We define the inverse of a matrix, the same
as the reciprocal.
a

1
1
a

1
0
AA 1 I
M

January 30, 2016

0
1
M
0

L
L
O
L

0
0

64

Newtons Method
If the Jacobian is non-singular, such that its
inverse exists, then we can apply this to
Newtons method.
We rarely want to compute the inverse, so
instead we look at the problem.
x

( i 1)

January 30, 2016

x f x
x(i ) h(i )
(i )

(i )

f x(i )

65

Newtons Method
Now, we have a linear system and we solve
for h.
J x ( k ) h ( k ) f x ( k )

x (i 1) x (i ) h ( i )

Repeat until h goes to zero.

We will look at solving linear systems later in the course.
January 30, 2016

66

Initial Guess
How do we get an initial guess for the root
vector in higher-dimensions?
In 2D, I need to find a region that contains
the root.
Steepest Decent is a more advanced topic
not covered in this course. It is more stable
and can be used to determine an
approximate root.
January 30, 2016

67

Root Finding Algorithms

Closed or Bracketed techniques
Bi-section
Regula-Falsi

Open techniques
Newton fixed-point iteration
Secant method

Multidimensional non-linear problems

The Jacobian matrix

Fixed-point iterations
Convergence and Fractal Basins of Attraction
January 30, 2016

OSU/CIS 541

68

Fixed-Point Iteration
Many problems also take on the specialized
form: g(x)=x, where we seek, x, that
satisfies this equation.
f(x)=x
g(x)

January 30, 2016

69

Fixed-Point Iteration
Newtons iteration and the Secant method
are of course in this form.
In the limit, f(xk)=0, hence xk+1=xk

January 30, 2016

70

Fixed-Point Iteration
Only problem is that that assumes it converges.
The pretty fractal images you see basically encode
how many iterations it took to either converge (to
some accuracy) or to diverge, using that point as
the initial seed point of an iterative equation.
The book also has an example where the roots
converge to a finite set. By assigning different
colors to each root, we can see to which point the
initial seed point converged.
January 30, 2016

71

Fractals
Images result
when we deal
with 2dimensions.
Such as
complex
numbers.
Color
indicates how
quickly it
converges or
diverges.
January 30, 2016

72

Fixed-Point Iteration
More on this when we look at iterative
solutions for linear systems (matrices).

January 30, 2016

73