Intro.

ANN & Fu::y Systems

Lecture 39 Hopfield Network
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
2
utline
Fundamentals of Hopfield Net
Analog mplementation
Associate Retrieval
Solving ptimization Problem
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
3
Fundamentals of Hopfield Net
Proposed by J.J. Hopfield. A fully Connected, feed-back, fixed
weight network.
Each neuron accepts its input from the outputs of all other
neurons and the its own input:
Net function
utput:



# u I
I v w u
=
+ =

=
|
,
|

>
=
0 1
0 1

u
u
v

'
1
'
2
'
3
I
1
I
2
I
3
T1
T2
T3
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
4
iscrete Time Formulation
efine V = [V
1
, V
2
, , V
n
]
T
, T = [T
1
, T
2
, , T
n
]
T
, =
[
1
,
2
, ,
n
]
T
, and
Then V(t+1) = sgn{ WV(t) + (t) T(t)}

=
0
0
0
0
3 2 1
3 32 31
2 23 21
1 13 12

n n n
n
n
n
w w w
w w w
w w w
w w w
W
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
5
Example
Let
Then
0 ;
0 1 1 1
1 0 1 1
1 1 0 1
1 1 1 0
;
1
1
1
1
) 0 ( =

= % I W v

= =
1
1
1
1
}
1
3
1
1
sgn )| 0 ( sgn| ) 1 ( v W v
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
6
Example (continued)
[1 1 1 1]
T
and [1 1 1 1]
T
are the two stable
attractors. Note that
) 1 (
1
1
1
1
}
3
3
3
3
sgn )| 1 ( sgn| ) 2 ( v v W v =

= =
. J

=
0 1 1 1
1 0 1 1
1 1 0 1
1 1 1 0
1
1
1
1
1 1 1 1
1
1
1
1
W
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
7
bservations
Let v* = [ 1 1 1 1]
T
. For any v(0) such that v
T
(0)v* = 0,
therwise, v(t) will oscillate between v(0).
Exercise: try v(0) = [ 1 1 1 1]
T
or [ 1 1 1 1]
T
.
iscussion:
Synchronous update: All neurons are updated together.
Suitable for digital implementation
Asynchronous update: Some neurons are updated faster
than others. Not all neurons are updated simultaneously.
Most natural for analog implementation.

) ( lim v t v
t
=

Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
8
Lyapunov function for Stability
Consider a scalar function E(V) satisfying:
(i) E(V*) = 0
(ii) E(V) > 0 for V = V*
(iii) dE/dV = 0 at V = V*, and dE/dV < 0 for V = V*
f such an E(V) can be found, it is called a
Lyapunov function, and the system is
asymptotically stable (i.e. V V* as t ).
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
9
Hopfield Net Energy Function
Hence, Hopfield net dynamic equation is to minimize
E(;) along descending gradient direction.
Stability of Hopfield Net f w
ij
= w
ji
& w
ii
= 0, the
output will converge to a local minimum (instead of
oscillating).
# u I Wv v E
d: : f
#
v I v v w v E
v

) (
) (
1
2
1
) (
0
1
,
+ = \
+ =




=
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
10
Associative Retrieval
Want to store a set of binary input vector {b
m
; 1 A m A
M} such that when a perturbed b'
m
is presented as
(input), the binary output V= b
m
.
Weight Matrix: Assume binary values ]1
. J ,
%

m
%
m m
%
m m
U
b b d, b b W
| 0 0 0 |
1
=
=

=
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
11
Example
b
1
= [ 1 1 1 1]
T
, b
2
= [1 1 1 1]
T
Let = V(0) = [ 1 1 1 1]
T
, then

=
0
0
0
;
0 0 2 2
0 0 0 0
2 0 0 2
2 0 2 0

U W

'
+

'

'
+

'

= =
1
1
1
1
4
0
0
0
) 2 ( ,
1
1
1
1
0
0
0
4
) ( ) 1 ( f J f I % f J
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
12
Hopfield Net Solution to TSP
(Hopfield and Tank) Use an n by n matrix to
represent a tour. V
ij
i-th city as the j-th stop. Each
entry is a neuron!
A 0 1 0 0 0 5
B 0 0 0 1 0 4
C 0 0 0 0 1 3
0 0 1 0 0 2
E 1 0 0 0 0 1
City/
tour
1 2 3 4 5
Intro. ANN & Fu::y Systems
(C) 2001 by Yu Hen Hu
13
Energy Function
First three terms makes V a permutation matrix. Last term
minimizes the tour distance
Validity of the solution e.g. the A, B, C, coefficients in the
TSP problem.
Quality of the solution the initial condition will affect the



=
+
= =
+ +
'
+

'

+
+ =
y
y y y


y
y


v v v d
D
N v
C
v v
B
v v
A
E
1 , 1 , , ,
2
,
, , , ,
2 2
2 2

49erwise 0 ; iI 1 were
) 1 ( ) 1 (
1 , 1 , ,
= = =
+ =
+

D C B A W

y y y y
0
0 0 0 0 0 0 0