ARTIFICIAL NEURAL

NETWORKS
Scholastic Video Book Series
Artificial Neural Networks
Part 3
(Back Propagation)
(with English Narrations)

http://scholastictutors.webs.com

(http://scholastictutors.webs.com/Scholastic-Book-NeuralNetworks-Part03-2013-10-22.pdf)

Scholastic Tutors (Oct, 2013)

ISVT 911-0-20-131022-1 1
 International Baccalaureate (IB)
Artificial Neural Networks - #3
Back-Propagation

Click here to see the video http://youtu.be/I2I5ztVfUSE

http://scholastictutors.webs.com
(ANN-003) 2
 Back-Propagation Algorithm
Step 1: Calculate the local gradients (do1, do2, dh1 and dh2) for the nodes
in the network. 1
(v )
+1 1 exp v
+1 dh1 b3 do1
b1 = () 1 ()
y1
w11 w31 y3 Desired = d3
x1 n1 n3
w41
1 = 1 3 + 1 31 + 2 32 (3 3)
w21
dh2 do2
w12 2 = 1 4 + 1 41 + 2 42 (4 4)
w32 y4
w22 n2
x2 n4 Desired = d4
w42
y2
+1 b2 b4
+1
1 = (1 1 + 1 11 + 2 12) 1 31 + 2 41
2 = (1 2 + 1 21 + 2 22) 1 32 + 2 42
http://scholastictutors.webs.com
 Back-Propagation
Algorithm
Step 2: Adjust the weights of the network using
the learning rule:
+ = + +
31 + 1 = 31 + 31 1 + 1 1
41 + 1 = 41 + 41 1 + 2 1
32 + 1 = 32 + 32 1 + 1 2
42 + 1 = 42 + 42 1 + 2 2
11 + 1 = 11 + 11 1 + 1 1
21 + 1 = 21 + 21 1 + 2 1
12 + 1 = 12 + 12 1 + 1 2
22 + 1 = 22 + 22 1 + 2 2
3 + 1 = 3 + 3 1 + 1 1
4 + 1 = 4 + 4 1 + 2 1
1 + 1 = 1 + 1 1 + 1 1
2 + 1 = 2 + 2 1 + 2 1
http://scholastictutors.webs.com
 Back-Propagation - Example
Lets perform a complete forward and backward sweep of the feedforward
network (2-2-1 architecture) shown below using the back propagation
algorithm discussed.
b1=0.1 dh1 Assume,
+1 v1 y1 +1 target output = 0.9,
h = 0.25, and a=0.0001
w11= -0.2
x1=0.1 Neuron 1
b3= 0.2
do1
w31= 0.2 Output = y3
w12= 0.1
v3 Neuron 3

w21= -0.1 w32= 0.3

Also assume that the
activation function in all 3
w22= 0.3 neurons are sigmoid functions
x2=0.9 Neuron 2
given by:
1
b2= 0.1 v2 (v )
1 exp v
+1 y2
dh2 5
http://scholastictutors.webs.com
 Forward Pass

1 = 1 1 + 1 11 + 2 12 = 1 0.1 + 0.1 0.2 + 0.9 0.1 = 0.17

1
1 = 1 = 0.17 = 1+exp(0.17) = 0.542
2 = 1 2 + 1 21 + 2 22 = 1 0.1 + 0.1 0.1 + 0.9 0.3 = 0.36
1
2 = 2 = 0.36 = 1+exp(0.36) = 0.589
3 = 1 3 + 1 31 + 2 32 = 1 0.2 + 0.542 0.2 + 0.589 0.3 = 0.485
1
3 = 3 = 0.485 = 1+exp(0.485) = 0.619
Therefore: = 3 3 = 0.9 0.619 = 0.281
http://scholastictutors.webs.com
 Backward Pass

1 = 0.17, 1 = 0.542
2 = 0.36, 2 = 0.589
3 = 0.485, 3 = 0.619
= 0.281

1 = 3 3 3 = 0.4851 0.281 =
0.4851 1 0.4851 0.281 = 0.619 1 0.619 0.281 = 0.0663

1 = 1 1 31 = 0.17 0.0663 0.2 =

0.17 1 0.17 0.01362 = 0.542 1 0.542 0.01362=0.0033

2 = 2 1 32 = 0.36 0.0663 0.3 =

0.36 1 0.36 0.01989 = 0.589 1 0.589 0.01989=0.0049

http://scholastictutors.webs.com
 Backward Pass

1 = 0.542, 2 = 0.589
1 = 0.0663,
1 = 0.0033
2 = 0.0049

Step 2: Adjust the weights of the network using

the learning rule:
+ = + +
31 +1 = 31 + 31 1 + 1 1
31 +1 = 0.2 + 0.0001 0.2 + 0.25 0.0663 0.542 = 0.2090
32 +1 = 32 + 32 1 + 1 2
32 +1 = 0.3 + 0.0001 0.3 + 0.25 0.0663 0.589 = 0.3098
Click here to see the video http://youtu.be/I2I5ztVfUSE
 Backward Pass

1 = 0.542, 2 = 0.589
1 = 0.0663,
1 = 0.0033
2 = 0.0049

Step 2: Adjust the weights of the network using

the learning rule:
+ = + +
11 +1 = 11 + 11 1 + 1 1
11 +1 = 0.2 + 0.0001 0.2 + 0.25 0.0033 0.1 = 0.1999
21 +1 = 21 + 21 1 + 2 1
21 +1 = (0.1) + 0.0001 0.1 + 0.25 0.0049 0.1 = 0.0999

http://scholastictutors.webs.com
 Backward Pass

1 = 0.542, 2 = 0.589
1 = 0.0663,
1 = 0.0033
2 = 0.0049

Step 2: Adjust the weights of the network using

the learning rule:
+ = + +
12 +1 = 12 + 12 1 + 1 2
12 +1 = 0.1 + 0.0001 0.1 + 0.25 0.0033 0.9 = 0.1008
22 +1 = 22 + 22 1 + 2 2
22 +1 = 0.3 + 0.0001 0.3 + 0.25 0.0049 0.9 = 0.3011
Click here to see the video http://youtu.be/I2I5ztVfUSE
 Backward Pass

1 = 0.542, 2 = 0.589
1 = 0.0663,
1 = 0.0033
2 = 0.0049

Step 2: Adjust the weights of the network using the

learning rule: Click here to see the video http://youtu.be/I2I5ztVfUSE
+ = + +
3 +1 = 3 + 3 1 + 1 1
3 +1 = 0.2 + 0.0001 0.2 + 0.25 0.0663 1 = 0.2166
1 +1 = 1 + 1 1 + 1 1
1 +1 = 0.1 + 0.0001 0.2 + 0.25 0.0033 1 = 0.1008
3 +1 = 3 + 3 1 + 2 1
3 +1 = 0.1 + 0.0001 0.1 + 0.25 0.0049 1 = 0.1012
 After one Complete Forward &
Backward Pass

b1=0.1008 dh1
+1 v1 y1 +1
w11= -0.1999
x1=0.1 Neuron 1 b3= 0.2166
do1
w31= 0.2090
w12= 0.1008 Output = y3
v3 Neuron 3
w21= -0.0999
w32= 0.3098

x2=0.9 w22= 0.3011

Neuron 2

b2= 0.1012 v2 y2
+1
dh2

Click here to see the video http://youtu.be/I2I5ztVfUSE 12

 After one Complete Forward &
Backward Pass

1 = 0.17 1 = 0.1715
1 = 0.542 1 = 0.5428
2 = 0.36 2 = 0.3622
2 = 0.589 2 = 0.5896
3 = 0.4851 3 = 0.5127
3 = 0.619 3 = 0.6254
= 3 3 = 0.9 0.619 = 0.281 = 3 3 = 0.9 0.6254 =
0.2746

Click here to see the video http://youtu.be/I2I5ztVfUSE

 After a few more Complete
Forward & Backward Passes

After second pass = 0.2683

After third pass = 0.2623
After forth pass = 0.2565
After 100 passes = 0.0693
After 200 passes = 0.0319
After 500 passes = 0.0038

Error is getting reduced after each pass.

Click here to see the video http://youtu.be/I2I5ztVfUSE

Artificial Neural
International Networks -(IB)
Baccalaureate #3
Back Propagation

Click here to see the video http://youtu.be/I2I5ztVfUSE

END of the Book

If you like to see similar solutions to any Mathematics problems please
contact us at: home.video.tutor@gmail.com with your request.

http://scholastictutors.webs.com
(ANN-003) 15
 Scholastic Video Book Series
Artificial Neural Networks
Part 3
(Back Propagation)
(with English Narrations)
(END)

Videos at: http://www.youtube.com/user/homevideotutor

(http://scholastictutors.webs.com/Scholastic-Book-NeuralNetworks-Part03-2013-10-22.pdf)

Scholastic Tutors (Sep, 2013)

ISVT 911-0-20-131022-1 16