Jaringan Syaraf Tiruan

Computational Intelligence S2

1
What are Neural Networks ?
1. Neural Networks (NNs) are networks of neurons, for example, as found in real (i.e.
biological) brains.
2. Artificial Neurons are crude approximations of the neurons found in brains. They may
be physical devices, or purely mathematical constructs.
3. Artificial Neural Networks (ANNs) are networks of Artificial Neurons, and hence
constitute crude approximations to parts of real brains. They may be physical devices, or
simulated on conventional computers.
4. From a practical point of view, an ANN is just a parallel computational system consisting
of many simple processing elements connected together in a specific way in order to
perform a particular task.
5. One should never lose sight of how crude the approximations are, and how over-
simplified our ANNs are compared to real brains.
2
Why are Artificial Neural Networks
worth studying?
1. They are extremely powerful computational devices (Turing equivalent,
universal computers).
2. Massive parallelism makes them very efficient.
3. They can learn and generalize from training data so there is no need for
enormous feats of programming.
4. They are particularly fault tolerant this is equivalent to the graceful
degradation found in biological systems.
5. They are very noise tolerant so they can cope with situations where normal
symbolic systems would have difficulty.
6. In principle, they can do anything a symbolic/logic system can do, and more.
(In practice, getting them to do it can be rather difficult)
3
What are Artificial Neural
Networks used for?
As with the field of AI in general, there are two basic goals for neural network research:

Brain modelling : The scientific goal of building models of how real brains work.
This can potentially help us understand the nature of human intelligence, formulate better teaching strategies, or
better remedial actions for brain damaged patients.

Artificial System Building : The engineering goal of building efficient systems for real world
applications. This may make machines more powerful, relieve humans of tedious tasks, and may even
improve upon human performance.

These should not be thought of as competing goals. We often use exactly the same networks and
techniques for both. Frequently progress is made when the two approaches are allowed to feed into
each other. There are fundamental differences though, e.g. The need for biological plausibility in brain
modelling, and the need for computational efficiency in artificial system building.

4
Learning in Neural Networks
There are many forms of neural networks. Most operate by passing neural
activations through a network of connected neurons.
One of the most powerful features of neural networks is their ability to learn
and generalize from a set of training data. They adapt the strengths/weights
of the
connections between neurons so that the final output activations are correct.
There are three broad types of learning:
1. Supervised Learning (i.e. learning with a teacher)
2. Reinforcement learning (i.e. learning with limited feedback)
3. Unsupervised learning (i.e. learning with no help)

5
A Brief History of the Field
1943 McCulloch and Pitts proposed the McCulloch-Pitts neuron model
1949 Hebb published his book The Organization of Behavior, in which the Hebbian
learning rule was proposed.
1958 Rosenblatt introduced the simple single layer networks now called Perceptrons.
1969 Minsky and Paperts book Perceptrons demonstrated the limitation of single layer
perceptrons, and almost the whole field went into hibernation.
1982 Hopfield published a series of papers on Hopfield networks.
1982 Kohonen developed the Self-Organising Maps that now bear his name.
1986 The Back-Propagation learning algorithm for Multi-Layer Perceptrons was
rediscovered and the whole field took off again.
1990s The sub-field of Radial Basis Function Networks was developed.
2000s The power of Ensembles of Neural Networks and Support Vector Machines
becomes apparent.
6
Some Current Artificial Neural
Network Applications
Brain modelling
Models of human development help children with developmental problems
Simulations of adult performance aid our understanding of how the brain works
Neuropsychological models suggest remedial actions for brain damaged patients
Real world applications
Financial modelling predicting stocks, shares, currency exchange rates
Other time series prediction climate, weather, airline marketing tactician
Computer games intelligent agents, backgammon, first person shooters
Control systems autonomous adaptable robots, microwave controllers
Pattern recognition speech recognition, hand-writing recognition, sonar signals
Data analysis data compression, data mining, PCA, GTM
Noise reduction function approximation, ECG noise reduction
Bioinformatics protein secondary structure, DNA sequencing

7
 The Nervous System
The human nervous system can be broken down into three stages that may be
represented in block diagram form as:

The receptors collect information from the environment e.g. photons on the retina.
The effectors generate interactions with the environment e.g. activate muscles.
The flow of information/activation is represented by arrows feedforward and
feedback.
Naturally, in this module we will be primarily concerned with the neural network in
themiddle.
8
 Levels of Brain Organization
The brain contains both large scale and small scale anatomical structures and different functions
take place at higher and lower levels.

There is a hierarchy of interwoven levels of organization:

1. Molecules and Ions
2. Synapses
3. Neuronal microcircuits
4. Dendritic trees
5. Neurons
6. Local circuits
7. Inter-regional circuits
8. Central nervous system

The ANNs we study in this module are crude approximations to levels 5 and 6.

9
Brains versus Computers : Some
numbers
1. There are approximately 10 billion neurons in the human cortex, compared with
10 of thousands of processors in the most powerful parallel computers.
2. Each biological neuron is connected to several thousands of other neurons,
similar to the connectivity in powerful parallel computers.
3. Lack of processing units can be compensated by speed. The typical operating
speeds of biological neurons is measured in milliseconds (10-3 s), while a silicon
chip can operate in nanoseconds (10-9 s).
4. The human brain is extremely energy efficient, using approximately 10-16 joules
per operation per second, whereas the best computers today use around 10-6
joules per operation per second.
5. Brains have been evolving for tens of millions of years, computers have been
evolving for tens of decades.
10
Structure of Human Brain

11
 Basic Components of Biological
Neurons
1. The majority of neurons encode their activations or outputs as a series of brief
electrical pulses (i.e. spikes or action potentials).
2. The neurons cell body (soma) processes the incoming activations and converts
them into output activations.
3. The neurons nucleus contains the genetic material in the form of DNA. This exists
in most types of cells, not just neurons.
4. Dendrites are fibres which emanate from the cell body and provide the receptive
zones that receive activation from other neurons.
5. Axons are fibres acting as transmission lines that send activation to other neurons.
6. The junctions that allow signal transmission between the axons and dendrites are
called synapses. The process of transmission is by diffusion of chemicals called
neurotransmitters across the synaptic cleft.
12
Schematic Diagram of a
Biological Neuron

13
 The McCulloch-Pitts Neuron
This vastly simplified model of real neurons is also known as a Threshold Logic Unit :

1. A set of synapses (i.e. connections) brings in activations from other neurons.

2. 2. A processing unit sums the inputs, and then applies a non-linear activation
function (i.e. squashing/transfer/threshold function).
3. 3. An output line transmits the result to other neurons.

14
 Some Useful Functions
A function y = f(x) describes a relationship (input-output mapping) from x to y.

Example 1 The threshold or sign function (bipolar) sgn(x) is defined as

Example 2 The logistic or sigmoid function Sigmoid(x) is defined as

This is a smoothed (differentiable) form of the threshold function.

15
 The McCulloch-Pitts Neuron Equation
Using the above notation, we can now write down a simple equation for the output out
of a McCulloch-Pitts neuron as a function of its n inputs :

where is the neurons activation threshold.

We can easily see that:

Note that the McCulloch-Pitts neuron is an extremely simplified model of real biological
neurons. Some of its missing features include: non-binary inputs and outputs, non-
linear summation, smooth thresholding, stochasticity, and temporal information
processing.
Nevertheless, McCulloch-Pitts neurons are computationally very powerful. One can
show that assemblies of such neurons are capable of universal computation.
16
 Networks of McCulloch-Pitts Neurons
One neuron cant do much on its own. Usually we will have many neurons
labelled by indices and activation flows between them via synapses with
strengths and

17
 The Perceptron
We can connect any number of McCulloch-Pitts neurons together in any way we like.
An arrangement of one input layer of McCulloch-Pitts neurons feeding forward to one output
layer of McCulloch-Pitts neurons is known as a Perceptron.

Already this is a powerful computational device. Later we shall see variations that make it even
more powerful.

18
 Implementing Logic Gates with
M-P Neurons
We can use McCulloch-Pitts neurons to implement the basic logic gates.
All we need to do is find the appropriate connection weights and neuron
thresholds to produce the right outputs for each set of inputs.
We shall see explicitly how one can construct simple networks that perform NOT,
AND, and OR.
It is then a well known result from logic that we can construct any logical function
from these three operations.
The resulting networks, however, will usually have a much more complex
architecture than a simple Perceptron.
We generally want to avoid decomposing complex problems into simple logic
gates, by finding the weights and thresholds that work directly in a Perceptron
architecture.
19
 Implementation of Logical NOT,
AND, and OR
In each case we have inputs and outputs , and need to determine the weights and thresholds. It
is easy to find solutions by inspection:

20
 ANN
Architectures/Structures/Topolo
gies
Mathematically, ANNs can be represented as weighted directed graphs. For our
purposes, we can simply think in terms of activation flowing between processing
units via one-way connections. Three common ANN architectures are:
Single-Layer Feed-forward NNs One input layer and one output layer of processing units.
No feed-back connections. (For example, a simple Perceptron.)
Multi-Layer Feed-forward NNs One input layer, one output layer, and one or more
hidden layers of processing units. No feed-back connections. The hidden layers sit in
between the input and output layers, and are thus hidden from the outside world. (For
example, a Multi-Layer Perceptron.)
Recurrent NNs Any network with at least one feed-back connection. It may, or may not,
have hidden units. (For example, a Simple Recurrent Network.)

Further interesting variations include: short-cut connections, partial connectivity,

time delayed connections, Elman networks, Jordan networks, moving windows, 21
Examples of Network
Architectures

22
 Other Types of
Activation/Transfer Function
Sigmoid Functions These are smooth (differentiable) and monotonically increasing.

The logistic function (sigmoid biner)

Hyperbolic tangent (sigmoid bipolar)

sigmoid_bipolar

Piecewise-Linear Functions Approximations of a sigmoid functions.

23
 Other Types of
Activation/Transfer Function
Symetric saturating linear

Threshold-bipolar

Symetric Hard-Limit

24
 Learning Process
Pada otak manusia, informasi yang dilewatkan dari satu neuron ke neuron yang
lainnya berbentuk rangsangan listrik melalui dendrit.
Jika rangsangan tersebut diterima oleh suatu neuron, maka neuron tersebut akan
membangkitkan output ke semua neuron yang berhubungan dengannya sampai informasi
tersebut sampai ke tujuannya yaitu terjadinya suatu reaksi.
Jika rangsangan yang diterima terlalu halus, maka output yang dibangkitkan oleh neuron
tersebut tidak akan direspon.
Tentu saja sangatlah sulit untuk memahami bagaimana otak manusia bisa belajar.
Selama proses pembelajaran, terjadi perubahan yang cukup berarti pada bobot-
bobot yang menghubungkan antar neuron.
Apabila ada rangsangan yang sama dengan rangsangan yang telah diterima oleh neuron,
maka neuron akan memberikan reaksi dengan cepat.
Namun apabila kelak ada rangsangan yang berbeda dengan apa yang telah diterima oleh
neuron, maka neuron akan segera beradaptasi untuk memberikan reaksi yang sesuai.
25
 Learning Process
Jaringan syaraf akan mencoba untuk mensimulasikan kemampuan otak manusia
untuk belajar. Jaringan syaraf tiruan juga tersusun atas neuron-neuron dan dendrit.
Tidak seperti model biologis, jaringan syaraf memiliki struktur yang tidak dapat
diubah, dibangun oleh sejumlah neuron, dan memiliki nilai tertentu yang
menunjukkan seberapa besar koneksi antara neuron (yang dikenal dengan nama
bobot).
Perubahan yang terjadi selama proses pembelajaran adalah perubahan nilai bobot.
Nilai bobot akan bertambah, jika informasi yang diberikan oleh neuron yang
bersangkutan tersampaikan.
Sebaliknya jika informasi tidak disampaikan oleh suatu neuron ke neuron yang lain, maka
nilai bobot yang menghubungkan keduanya akan dikurangi.
Pada saat pembelajaran dilakukan pada input yang berbeda, maka nilai bobot akan
diubah secara dinamis hingga mencapai suatu nilai yang cukup seimbang. Apabila
nilai ini telah tercapai mengindikasikan bahwa tiap-tiap input telah berhubungan
dengan output yang diharapkan. 26
 Supervised
Learning

Paradigma Unsupervised
Learning Learning

Hybrid

Proses
Learning Error
Correction

Boltzmann
Learning
Jenis Aturan
Learning
Hebian
Learning

Competitive
Learning
27
 Supervised Learning
Metode pembelajaran pada jaringan syaraf disebut terawasi jika output yang diharapkan
telah diketahui sebelumnya.
Contoh: andaikan kita memiliki jaringan syaraf yang akan digunakan untuk mengenali
pasangan pola, misalkan pada operasi AND:

Pada proses pembelajaran, satu pola input akan diberikan ke satu neuron pada lapisan input.
Pola ini akan dirambatkan di sepanjang jaringan syaraf hingga sampai ke neuron pada lapisan
output. Lapisan output ini akan membangkitkan pola output yang nantinya akan dicocokkan
dengan pola output targetnya.
Apabila terjadi perbedaan antara pola output hasil pembelajaran dengan pola target, maka
disini akan muncul error. Apabila nilai error ini masih cukup besar, mengindikasikan bahwa
masih perlu dilakukan lebih banyak pembelajaran lagi.
28
 Supervised Learning (Hebb
Rule)
Hebb rule adalah metode pembelajaran yang paling sederhana.
Pada metode ini pembelajaran dilakukan dengan cara memperbaiki nilai bobot sedemikian
rupa sehingga jika ada 2 neuron yang terhubung, dan keduanya pada kondisi hidup (on)
pada saat yang sama, maka bobot antara keduanya dinaikkan.
Apabila data direpresentasikan secara bipolar, maka perbaikan bobotnya adalah:

= bobot sinapsis yang menghubungkan neuron i dan neuron j

= nilai neuron input i
= nilai neuron output j
= learning rate
Misalkan kita gunakan pasangan vektor input s dan vektor output sebagai pasangan vektor
yang akan dilatih. Sedangkan vektor yang hendak digunakan untuk testing adalah vektor
29 x.
Supervised Learning (Algorithm
of Hebb Rule)

30
 Supervised Learning (Example 1
of Hebb Rule)
Misalkan kita ingin membuat jaringan syaraf untuk melakukan pembelajaran
terhadap fungsi OR dengan input dan target bipolar sebagai berikut:

31
Supervised Learning (Example
of Hebb Rule)

32
 Supervised Learning (Example 2
of Hebb Rule)
Misalkan kita ingin membuat jaringan syaraf untuk melakukan pembelajaran
terhadap fungsi AND dengan 2 input (-1 dan 1) dan target bipolar sesuai aturan
operator AND serta bobot awal dan bobot bias adalah 0.

33
 Supervised Learning (Example 3
of Hebb Rule)
Misalkan kita ingin membuat jaringan syaraf untuk melakukan pembelajaran
terhadap fungsi OR dengan 3 input (setiap input mempunyai kemungkinan nilai 0
dan 1) dan bobot awal dan bobot bias adalah 0 (tentukan fungsi yang sesuai).

34
Perceptron Learning Procedure
with Error Correction Rule

35
Perceptron Rule Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t z) xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000

36
Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t z) xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000

37
Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t z) xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111

38
Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t z) xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111 3 1 0 0 0 0
011 1 0 1111 39
Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t z) xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111 3 1 0 0 0 0
011 1 0 1111 3 1 0 -1 -1 -1
001 1 0 1000
40
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Example Assume a learning rate c of 1 and initial weights all 0: Dw = c(t z) x i i

Training set 0 0 1 -> 0

1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

PatternTarget Weight Vector Net Output DW

0 0 1 10 0000 0 0 0 0 0 0
1 1 1 11 0000 0 0 1 1 1 1
1 0 1 11 1111 3 1 0 0 0 0
0 1 1 10 1111 3 1 0 -1 -1 -1
0 0 1 10 1000 0 0 0 0 0 0
1 1 1 11 1000 1 1 0 0 0 0
1 0 1 11 1000 1 1 0 0 0 0
0 1 1 10 1000 0 0 0 0 0 0
41