Boltzmann Machine

Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Boltzmann Machine
A Brief Introduction
Stochastic Hopfield Nets

with Hidden Units
Boltzmann Machine
Learning Algorithm for
Boltzmann Machine
Reference
Ritajit Majumdar
Arunabha Saha
Calcutta University
October 23, 2013
Ritajit Majumdar Arunabha Saha (CU)
Boltzmann Machine
October 23, 2013
1 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Hopfield Net
with Hidden Units
Boltzmann Machine
Stochastic Hopfield Nets with Hidden Units

Boltzmann Machine
Reference
Boltzmann Machine
Learning Algorithm for Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
2 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Hopfield Network
Outline
Hopfield Net
with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Figure : Two dimensional representation of motion in state space
Boltzmann Machine
October 23, 2013
3 / 28
Boltzmann Machine
Hopfield Net
Ritajit Majumdar
Arunabha Saha
Outline
A Hopfield Net is composed of binary threshold units with

recurrent connections between them.
Hopfield Net
Recurrent networks of non-linear units are hard to analyze,

since they can behave in many different ways -
Boltzmann Machine
Settle to a stable state.
Oscillate
Follow chaotic trajectory.

with Hidden Units

Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
4 / 28
Boltzmann Machine
Hopfield Net
Ritajit Majumdar
Arunabha Saha
Outline

Hopfield Net

Boltzmann Machine
Oscillate

with Hidden Units

Boltzmann Machine
Reference
John Hopfield introduced a global energy function for

network with symmetric connections.
Boltzmann Machine
October 23, 2013
4 / 28
Boltzmann Machine
Hopfield Net
Ritajit Majumdar
Arunabha Saha
Outline

Hopfield Net

Boltzmann Machine
Oscillate

with Hidden Units

Boltzmann Machine
Reference

1
Each binary configuration of the whole network has an

Energy.
Boltzmann Machine
October 23, 2013
4 / 28
Boltzmann Machine
Hopfield Net
Ritajit Majumdar
Arunabha Saha
Outline

Hopfield Net

Boltzmann Machine
Oscillate

with Hidden Units

Boltzmann Machine
Reference

1
Each binary configuration of the whole network has an

Energy.
The binary threshold decision rule causes the network to

settle to a minimum of this energy function.
Boltzmann Machine
October 23, 2013
4 / 28
Boltzmann Machine
Energy Function
Ritajit Majumdar
Arunabha Saha
Outline
The global energy is defined as X

X
E =
si bi
si sj wij
i
Hopfield Net
(1)
Boltzmann Machine
i<j
th
where bi is the bias of the i unit, s is 0 or 1 depending on

whether the unit is turned off or on respectively. And wij is the
weight of the connection between units i and j.
1 For

with Hidden Units

Boltzmann Machine
Reference
Bipolar Inputs the states will be 1 and 1 respectively.
Boltzmann Machine
October 23, 2013
5 / 28
Boltzmann Machine
Energy Function
Ritajit Majumdar
Arunabha Saha
Outline

X
E =
si bi
si sj wij
i
Hopfield Net
(1)
Boltzmann Machine
i<j
th

with Hidden Units


Boltzmann Machine
Reference
From this energy function, each unit computes locally how

changing their state will affect the global energy.
1 For
Boltzmann Machine
October 23, 2013
5 / 28
Boltzmann Machine
Energy Function
Ritajit Majumdar
Arunabha Saha
Outline

X
E =
si bi
si sj wij
i
Hopfield Net
(1)
Boltzmann Machine
i<j
th

with Hidden Units

Boltzmann Machine

Reference
From this energy function, each unit computes locally how

changing their state will affect the global energy.
The energy gap is defined as 4 Ei = E (si = 0) E (si = 1) = bi +
sj wij
(2)
1 For
Boltzmann Machine
October 23, 2013
5 / 28
Settling to an Energy Minima
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
The net is initially in a random

state i.e., the units are on or off
randomly. The binary threshold
decision rule updates units one
at a time in a random order.
Update each unit to
whichever of its two states
minimizes the global
energy.
Boltzmann Machine
Boltzmann Machine
Reference
Use binary threshold units,

i.e., the states can be
either 0 or 1.
Boltzmann Machine
October 23, 2013
6 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units

Update each unit to
energy.
Boltzmann Machine
Boltzmann Machine
Reference

either 0 or 1.
Boltzmann Machine
October 23, 2013
7 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units

Boltzmann Machine
Boltzmann Machine
Reference
Update each unit to

energy.
either 0 or 1.
-E = goodness = 4
Boltzmann Machine
October 23, 2013
8 / 28
Using this type of Computation as memory
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
Hopfield proposed that memories could be energy minima

of a neural net.
I
The binary threshold decision rule can be used to clean

up incomplete or corrupted memory.
Boltzmann Machine
Boltzmann Machine
Boltzmann Machine
Reference
October 23, 2013
9 / 28
Using this type of Computation as memory
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
Hopfield proposed that memories could be energy minima

of a neural net.
I
The binary threshold decision rule can be used to clean

up incomplete or corrupted memory.
Boltzmann Machine
Boltzmann Machine
Reference
Using energy minima to represent memories gives a

content-addressable memory.
I
I
An item can be accessed just by knowing parts of it.

It is robust against hardware damage.
Boltzmann Machine
October 23, 2013
9 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Stochastic Hopfield Nets with Hidden

Units
Outline
Hopfield Net
with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
10 / 28
Hopfield Net with Hidden Units
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Why add hidden units?

with Hidden Units
In Hopfield Net, there is no hidden layer of units. By adding

hidden layers, the attention can be shifted from just storing
memories to various types of interpretations of the inputs.
Boltzmann Machine
Boltzmann Machine

Boltzmann Machine
Reference
October 23, 2013
11 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net

with Hidden Units

Boltzmann Machine
Boltzmann Machine
Reference
Use the net to construct

interpretations of sensory
input.
Boltzmann Machine
October 23, 2013
11 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net

with Hidden Units

Boltzmann Machine
Boltzmann Machine
Reference

input.
The input is represented
by the visible units.
Boltzmann Machine
October 23, 2013
11 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net

with Hidden Units

Boltzmann Machine
Boltzmann Machine
Reference

input.
The input is represented
by the visible units.
The interpretation is
represented by the states
of the hidden units.
Boltzmann Machine
October 23, 2013
11 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline
The Binary Threshold Decision

rule always goes downhill, i.e.,
reduces energy.
Hopfield Net
with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline

reduces energy.
Hopfield Net
Hence it is impossible to escape

from a local minima.

Boltzmann Machine

with Hidden Units
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline

reduces energy.
Hopfield Net


Boltzmann Machine

with Hidden Units
Boltzmann Machine
Reference
Solution - Use random noise to

escape from poor, shallow
minima.
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline

reduces energy.
Hopfield Net


Boltzmann Machine

with Hidden Units
Boltzmann Machine
Reference

minima.
I
Start with a lot of noise to

escape the energy barriers of
poor local minima.
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline

reduces energy.
Hopfield Net


Boltzmann Machine

with Hidden Units
Boltzmann Machine
Reference

minima.
I

poor local minima.
Slowly reduce the noise so
that the system ends up in a
deep minima.
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Noisy Networks
Ritajit Majumdar
Arunabha Saha
Outline

reduces energy.
Hopfield Net


Boltzmann Machine

with Hidden Units
Boltzmann Machine
Reference

minima.
I

poor local minima.
Slowly reduce the noise so
that the system ends up in a
deep minima.
This process is called
Simulated Annealing.
Boltzmann Machine
October 23, 2013
12 / 28
Boltzmann Machine
Stochastic Binary Units
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?
Hopfield Net
with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
13 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?

1
Hopfield Net
Replace the binary threshold units by binary stochastic

units that make biased random decisions.

with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
13 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?

1
Hopfield Net


with Hidden Units
The temperature controls the amount of noise.

Boltzmann Machine
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
13 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?
Hopfield Net


with Hidden Units

Boltzmann Machine
Unit i then turns on with the probability given by the

logistic function -
Reference
prob(si = 1) =
1
1 + e
4Ei
T
Boltzmann Machine
Boltzmann Machine
(3)
October 23, 2013
13 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
How to add noise?
Hopfield Net


with Hidden Units

Boltzmann Machine
Unit i then turns on with the probability given by the

logistic function -
Reference
prob(si = 1) =
T =0
T
T =1
1
1 + e
4Ei
T
Boltzmann Machine
(3)
Deterministic (Hopfield Net)

Complete Chaos
Approaches Boltzmann Distribution
Boltzmann Machine
October 23, 2013
13 / 28
What is Boltzmann Machine?
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
Boltzmann Machine
Boltzmann Machine

Boltzmann Machine
A Hopfield Net consisting of Binary Stochastic Neuron with

hidden units is called Boltzmann Machine.
Reference
A Boltzmann Machine is a network of symmetrically

connected, neuron like units that make stochastic decisions
about whether to be on or off.
Boltzmann Machine
October 23, 2013
14 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Boltzmann Machine

with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
15 / 28
Boltzmann Machine
Modeling Binary Data
Ritajit Majumdar
Arunabha Saha
The objective of Boltzmann Machine is -
Outline
Hopfield Net
Modeling Binary Data

with Hidden Units
Given a training set of binary vectors, fit the model that will
assign a probability to every possible binary vector.
When unit i is given oppurtunity to update its state, it first
computes its total input zi ,
X
zi = bi +
sj wij
(4)
Boltzmann Machine
Boltzmann Machine
Reference
Unit i turns on with probability prob(si = 1) =
1
1 + e zi
(5)
If the units are updated sequencially in random order, the

network will eventually reach a Boltzmann Distribution, also
called its stationary distribution.
Boltzmann Machine
October 23, 2013
16 / 28
Boltzmann Machine
How Boltzmann Machine Generates Data
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
In Boltzmann Machine, everything is defined in terms of the

energies of joint configurations of the visibile (v) and hidden
(h) units.
Boltzmann Machine
Boltzmann Machine
Reference
The energies of joint configurations are related to their

probabilities as p(v , h) e E (v ,h)
Boltzmann Machine
(6)
October 23, 2013
17 / 28
Probabilities in term of Energies
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
The probability of a joint

configuration over both
visibile and hidden units
depends on the energy of
that joint configuration
compared with the
energies of all other joint
configurations.

with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Boltzmann Machine
October 23, 2013
18 / 28
Probabilities in term of Energies
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
The probability of a joint

configuration over both
visibile and hidden units
depends on the energy of
that joint configuration
compared with the
energies of all other joint
configurations.

with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
The probability of a
configuration of the
visibile units is the sum of
the probabilities of all the
joint configurations that
contain it.
Boltzmann Machine
October 23, 2013
18 / 28
Boltzmann Machine
An Example
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Figure : Credit: Geoffrey Hinton, Neural Networks for Machine

Learning
Boltzmann Machine
October 23, 2013
19 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Learning Algorithm

with Hidden Units
Boltzmann Machine
Boltzmann Machine
A surprising feature of this rule is that it uses only locally

available information. The change of weight depends only
on the behaviour of the two units it connects, even
though the change optimizes a global measure.
Reference
- Ackley, Hinton 1985
Boltzmann Machine
October 23, 2013
20 / 28
Boltzmann Machine
Goal of Learning
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
Learning Algorithm for Boltzmann Machine is an unsupervised

learning algorithm. Unlike Backpropagation Algorithm, where
the training set consists of input vector and desired output, in
Boltzmann Machine only the input vector is provided.
Boltzmann Machine
Boltzmann Machine
Reference
We want to maximize the product of the probabilities the

Boltzmann Machine assigns to the binary vectors in
training set.
It is equivalent to maximizing the probability that we
would obtain exactly the N training cases.
Boltzmann Machine
October 23, 2013
21 / 28
Why Learning could be difficult
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Consider a chain of hidden units with visibile units attached at

two ends -

with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Training: We want the two visibile units to be in opposite

states.
Solution: The product of all the weights must be negative. If
all are positive, then turning on one unit will turn on the next
unit and eventually the two visibile units will be in same state.
Boltzmann Machine
October 23, 2013
22 / 28
Why Learning could be difficult
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Consider a chain of hidden units with visibile units attached at

two ends -
Hopfield Net
with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
Difficulty: To modify w1 and w5, we need to know w3 (and

the weights of other hidden units too).
Because, if w3 is negative, then we need to modify w1 in a
different way than what we would do if w3 is positive.
So to change one weight in a right direction, we need to know
all the other weights.
Boltzmann Machine
October 23, 2013
23 / 28
Boltzmann Machine
Learning Algorithm
Ritajit Majumdar
Arunabha Saha
Outline
Correlation
Hopfield Net
Everything that one weight needs to know of other weights and

the data is contained in the difference of two correlations logp(v )
= hsi sj iv hsi sj imodel
wij
(7)

with Hidden Units
Boltzmann Machine
Boltzmann Machine
Reference
where h.i is the expectation value.

The first term in R.H.S. denotes the expectation value of
product of states at equilibrium when the state vector (or
data) v is clamped on the visibile units.
The second term in R.H.S. denotes the expectation value
of product of states at equilibrium without any clamping.
So we can make the change in weight 4wij hsi sj iv hsi sj imodel
Boltzmann Machine
(8)
October 23, 2013
24 / 28
Boltzmann Machine
Why is it so?
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
We know the probability of a global configuration at

equilibrium is -

with Hidden Units
Boltzmann Machine
p(v , h) e E (v ,h)

Boltzmann Machine
So, the logarithm of probability is a linear function of the

energy.
Reference
And energy, on its own term, is a linear function of weights and

states.
P
P
E = si bi
si sj wij
i
i<j
Hence,
E
= si sj
wij
Boltzmann Machine
(9)
October 23, 2013
25 / 28
Boltzmann Machine
Why is it so?
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
Differnetiating equation 1, we get E

wij

with Hidden Units
Boltzmann Machine
= si sj

Boltzmann Machine
Reference
The process of settling to equilibrium state propagates

information about the weights.
No need of backpropagation.
The following two stages are required -
I
I
4
The machine needs to settle to equilibrium with data.

The machine needs to settle to equilibrium without data.
However, in both cases the learning process is similar with

different boundary conditions.
Boltzmann Machine
October 23, 2013
26 / 28
Boltzmann Machine
Reference
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
Geoffrey Hinton
Neural Networks for Machine Learning
www.coursera.org
Boltzmann Machine
Boltzmann Machine
Reference
Geoffrey Hinton
http://www.scholarpedia.org/article/Boltzmann_
machine
Geoffrey Hinton, David Ackley
A learning Algorithm for Boltzmann Machine
Cognitive Science 9, 147-169 (1985)
Boltzmann Machine
October 23, 2013
27 / 28
Boltzmann Machine
Ritajit Majumdar
Arunabha Saha
Outline
Hopfield Net
with Hidden Units
Boltzmann Machine
THANK YOU
Boltzmann Machine

Boltzmann Machine
Reference
October 23, 2013
28 / 28

Boltzmann Machine

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Boltzmann Machine

Uploaded by

Copyright:

Available Formats

Boltzmann Machine

Stochastic Hopfield Nets

October 23, 2013

Ritajit Majumdar Arunabha Saha (CU)

October 23, 2013

Stochastic Hopfield Nets with Hidden Units

Learning Algorithm for

Learning Algorithm for Boltzmann Machine

Ritajit Majumdar Arunabha Saha (CU)

October 23, 2013

Figure : Two dimensional representation of motion in state space

Ritajit Majumdar Arunabha Saha (CU)

October 23, 2013

A Hopfield Net is composed of binary threshold units with

Recurrent networks of non-linear units are hard to analyze,

Settle to a stable state.

Ritajit Majumdar Arunabha Saha (CU)

Stochastic Hopfield Nets

Learning Algorithm for

October 23, 2013

A Hopfield Net is composed of binary threshold units with

Recurrent networks of non-linear units are hard to analyze,

Settle to a stable state.

Stochastic Hopfield Nets

Learning Algorithm for

John Hopfield introduced a global energy function for

Ritajit Majumdar Arunabha Saha (CU)

October 23, 2013

A Hopfield Net is composed of binary threshold units with

Recurrent networks of non-linear units are hard to analyze,

Settle to a stable state.

Stochastic Hopfield Nets

Learning Algorithm for

John Hopfield introduced a global energy function for

Each binary configuration of the whole network has an

Ritajit Majumdar Arunabha Saha (CU)

October 23, 2013

A Hopfield Net is composed of binary threshold units with

Recurrent networks of non-linear units are hard to analyze,

Settle to a stable state.

Stochastic Hopfield Nets

Learning Algorithm for

John Hopfield introduced a global energy function for

Each binary configuration of the whole network has an

The binary threshold decision rule causes the network to

Ritajit Majumdar Arunabha Saha (CU)

October 23, 2013

The global energy is defined as X

where bi is the bias of the i unit, s is 0 or 1 depending on

Stochastic Hopfield Nets

Learning Algorithm for

Bipolar Inputs the states will be 1 and 1 respectively.

Ritajit Majumdar Arunabha Saha (CU)

October 23, 2013

The global energy is defined as X

Stochastic Hopfield Nets

where bi is the bias of the i unit, s is 0 or 1 depending on

Learning Algorithm for

From this energy function, each unit computes locally how

Bipolar Inputs the states will be 1 and 1 respectively.

Ritajit Majumdar Arunabha Saha (CU)

October 23, 2013

The global energy is defined as X

Stochastic Hopfield Nets