1

IMPROVING BACKPROPAGATION ALGORITHM TO

OVERCOME LOCAL MINIMA PROBLEM

Prepared By : Ahmed Kamal Abdel-fatah
4
th
Year Computer and Automatic Control Engineering

1 Problem
Artificial Neural Network (ANN) is computational models whose
architecture and operation is inspired by human knowledge of biological
nervous system.

The BP algorithm is the most popular, effective and easy to produce
model for complex ANN. BP algorithm is a supervised learning method that
attempts to minimise the error of the network. The very general nature of the
backpropagation training method means that a backpropagation net can be
used to solve problems in many areas.

In spite of these important properties, a major criticism is commonly moved
against backpropagation algorithm. Backpropagation algorithm is actually a
gradient method and therefore there is no guarantee at all that the global
minimum of error surface can be reached. [Figure 1]


Figure 1. Local Minima and Global Minima
2
2 Solution
There are several solutions to this problem. One is very simple and that is to
reset the weights to different random numbers and try training again
(this can also solve several other problems).

Another solution is to add momentum to the weight change. This
means that the weight change this iteration depends not just on the
current error, but also on previous changes.

For example:
W + = W + Current change + (Change on previous iteration * constant)

Constant is less than 1.

As the system falls into a local minima, the small changes in the weight vector
do not take out the system from this hole and bring it to global minimum.
Therefore we need radical changes in weight vector so that the system gets out
of this local minimum.

The mutation operation that is commonly used in genetic algorithms is
another solution to this problem. Each time we apply the input-output training
set to the system, we also apply a mutation operation to the original synaptic
weights in the system.


3 References
[1] A.Yiicetiirkl, A.Herdagdelenz, K.Uyantkz , 1999 A SOLUTION TO THE
PROBLEM OF LOCAL MINIMA IN BACKPROPAGATION ALGORITHM

[2] C. MacLeod , An Introduction to Practical Neural Networks and Genetic
Algorithms for Engineers and Scientists , Chapter 3 , p.9