Introduction To Neural Networks: by Suneel

StudentRockStars.
com
StudentRockStars.com
Introduction to Neural Networks

By Suneel
What is a Neural Network?

The question 'What is a neural network? is ill-posed. StudentRockStars.com -- Pinkus (1999) A method of computing, based on the interaction of multiple connected processing elements
Characteristics of Biological Neural Networks

1) Massive connectivity 2) Nonlinear, Parallel, Robust and Fault Tolerant 3) Capability to adapt to surroundings
4) Ability to learn and generalize from known examples 5) Collective behavior is different from individual behavior
Artificial Neural Networks mimics some of the properties of the biological neural networks
Massively parallel computers are
approaching power of human brain Can we develop neural networks which StudentRockStars.com work well on these machines? It is well known that artificial neural networks do not scale well
What can a Neural Net do?

Compute a known function Approximate an unknown function Pattern StudentRockStars.com Recognition Signal Processing Learn to do any of the above
Neural Computing
Neuron is fundamental information processing unit of brain Neuroscience
The objective is to understand the human brain Biologically realistic models of neurons Biologically realistic connection topologies
Neural networks
The objective is to develop computation methods Highly simplified artificial neurons Connection topologies that are aimed at computational effectiveness
Human Brain vs Supercomputers

Human Brain ~100 billion neurons (1011) (for comparison, a rat
cortex has about 30million) StudentRockStars.com ~1000 synapses per neuron (1014 total ) This is roughly 100 terabytes (1014) of data storage It is also capable of roughly 1000 teraflops (1015 operations per second)
IBM BlueGene/L Computer (DOE) has: 65,536 processors ( 1013 transistors? ) 33 terabytes RAM (1013) 137 teraflops (1014 operations per second) StudentRockStars.com NASAs Columbia computer has: 10,240 Itanium processors (1.5 GHz) 10 terabytes RAM (1013) 40 teraflops (1013 operations per second) Theoretically more capable than the brain of a monkey ... and near human capability ...
Man versus Machine (hardware)
Von Neumann Numbers Human brain computer (a.d. 2005) # elements 1010 - 1012 107 - 108 neurons transistors StudentRockStars.com 10 # connections / 104 element switching frequency 103 Hz 109 - 1010 Hz energy / operation power consumption reliability of system 10-16 Joule 10 Watt high 10-6 Joule 100 - 500 Watt reasonable reasonable
reliability of elements low
Man versus Machine (information processing)

Features Human Brain
Data analog StudentRockStars.com representation Memory distributed localized localization Control distributed localized Processing Skill acquisition parallel learning
Von Neumann computer digital
sequential programming
Basic Concepts
A Neural Network generally
maps a set of inputs to a set of outputs

Number of inputs/outputs is
Input 0
Input 1
...
Input n
variable
Output 0 Output 1
Neural Network
The Network itself is composed
of an arbitrary number of nodes with an arbitrary topology
...
Output m
Applications of ANNs
ANNs have been widely used in various
domains for:
Pattern recognition StudentRockStars.com Associative memory Function approximation
Organization of the Brain

Central nervous system Interregional circuits Local circuits Neurons StudentRockStars.com Dendritic trees Neural microcircuits Synapses Molecules
Biological Neuron
A neuron has four main regions: Cell body (soma) DendritesStudentRockStars.com Axon Presynaptic terminal excitatory inhibitory
The Biological Neuron
The human brain is made of about 100 billions of such neurons.
Neurons: Dendrites and Axons StudentRockStars.com (Information travels thru electrical signals)
Synapses
You are your synapses (LeDoux) Your memories, emotions, etc. are stored in
your synapses StudentRockStars.com Learning occurs via changes to the synapses Some of the synapses are set at birth, while others are trained The human ability to hear, remember, fear, or desire are all related to the synapses.
Moores Law
For about 50 years we have seen computer
performance double every 18 months Intel expects this to continue until at least 2011 StudentRockStars.com Number of transistors per chip area has doubled every 18 months: - 1974, Intel 8088, 29,000 transistors -2000, Intel Pentium 4, 42,000,000 transistors -2011, Intel, 20 billion transistors/chip expected (maybe 128processors per chip)
Future Computers
Intel expects the first petaflop (1015 ops/second)
to appear by about 2009 this is human-brainlevel computing power (it might cost $150M and require 8 MWatts of power) StudentRockStars.com By 2011 there could be several petaflop computers By 2011 there could be supercomputers several times more powerful than human brains
Artificial Neural Networks
Artificial Neural Networks

Layered net with n input nodes m output nodes zero or more
x1
1
x2
hidden layers (one shown)

m
xn
Advantages of ANNs
Efficient
Inherent massively parallel

Robust
Can deal with incomplete and/or noisy data

Fault-tolerant
Still works when part of the net fails

User-friendly
Learning instead of programming
Disadvantages of ANNs
Difficult to design
The are no clear design rules for arbitrary applications
Hard orStudentRockStars.com impossible to train Difficult to assess internal operation

It is difficult to find out whether, and if so what tasks are performed by different parts of the net
Unpredictable
It is difficult to estimate future network
Model of a Neuron
Weigthed connecting links Adder
Activation function
vk = wkj xj
j=1
yk = f (vk + bk)
Activation Functions
1) Threshold Function f(v) = 1 if v 0 = 0 otherwise 2) Piecewise-Linear Function f(v) =StudentRockStars.com 1 if v = v if > v > - = 0 otherwise 3) Sigmoid Function f(v) = 1/{1 + exp(- av)} etc..
Discrete asymmetric transfer f(c, x) = ( x > c ? 1 : 0 ) Heaviside step-function:

1.2 1 0.8 c=2
StudentRockStars.com 0.6
0.4 0.2 0 -0.2 -10 -5 0 5 10
Piecewise-Linear Transfer
f(c, z) = ( z < c ? 1 : ( z > c ? 1 : z / c ) )
1
0.5 0 -0.5
-1 -10 -5 0 5 10
Continuous asymmetric transfer

sigmoid function logistic function
1.2 1 0.8 c = 0.5 c = 1.0 c = 2.0
StudentRockStars.com 0.6
0.4 0.2 0 -0.2 -10 -5 0 5 10
NEURAL NETWORK VIEWED AS DIRECTED GRAPHS

A
C
FEED BACK
Multilayer feedforward network
Feed Back Network
NETWORK ARCHITECTURES
Network architectures typically used
Single layer feedforward Multilayer feedforward StudentRockStars.com Recurrent networks, etc.
Different Network Topologies

Single layer feed-forward networks Input layer projecting into the output layer
Single layer network
Input layer
Output layer

Multi-layer feed-forward networks One or more hidden layers. Input projects only from previous layers onto a layer.
StudentRockStars.com or 2-layer
1-hidden layer fully connected network Input layer Hidden Output layer layer

Recurrent networks A network with feedback, where some of its inputs are connected to some of its outputs StudentRockStars.com (discrete time).
Recurrent network
Input layer
Output layer
StudentRockStars.com Multilayered Perceptron (Contd.)
Some important issues: How big should be my network ? No specific answer is known till date. The size of the network depends on the complexity of the problem at hand and the training accuracy which is desired. A good training accuracy does not always means a good network. If the number of free parameters of StudentRockStars.com the network is almost the same as the number of data points, the network tends to memorize the data and gives bad generalization. How many hidden layers to use ? It has been proved that a single hidden layer is sufficient to do any mapping task. But still experience shows that multiple hidden layers may be sometimes simplify learning.
Artificial Intelligence(AI) and Neural Network

The AI is to development of Algorithms that require
machines to perform cognitive tasks.

AI systemStudentRockStars.com must be capable of 3 things
1.Store knowledge. 2.Apply the knowledge store to slove problems. 3.Acquire new knowledge through experience.
Representation. Reasoning. Learning
Back-Propagated Delta Rule Networks (BP)

Inputs are put
through a Hidden Layer before the StudentRockStars.com Hidden Layer ... output layer All nodes connected ... between layers
H0 H1 Hm O0 O1 Oo Output 0 Output 1
Input 0
Input 1
...
Input n
...
Output o
How to Build Prior Information into Neural Network Design
Block diagram of an invariant feature-space type of system
Thank You

Introduction To Neural Networks: by Suneel

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To Neural Networks: by Suneel

Uploaded by

Copyright:

Available Formats

StudentRockStars.

Introduction to Neural Networks

What is a Neural Network?

Characteristics of Biological Neural Networks

Massively parallel computers are

What can a Neural Net do?

Human Brain vs Supercomputers

Man versus Machine (hardware)

reliability of elements low

Man versus Machine (information processing)

Von Neumann computer digital

maps a set of inputs to a set of outputs

The Network itself is composed

of an arbitrary number of nodes with an arbitrary topology

Organization of the Brain

The Biological Neuron

The human brain is made of about 100 billions of such neurons.

For about 50 years we have seen computer

Intel expects the first petaflop (1015 ops/second)

Artificial Neural Networks

Artificial Neural Networks

hidden layers (one shown)

Inherent massively parallel

Can deal with incomplete and/or noisy data

Still works when part of the net fails

Learning instead of programming

Hard orStudentRockStars.com impossible to train Difficult to assess internal operation

Discrete asymmetric transfer f(c, x) = ( x > c ? 1 : 0 ) Heaviside step-function:

Continuous asymmetric transfer

NEURAL NETWORK VIEWED AS DIRECTED GRAPHS

Multilayer feedforward network

Single layer feedforward Multilayer feedforward StudentRockStars.com Recurrent networks, etc.

Different Network Topologies

Single layer network

Different Network Topologies

Different Network Topologies

StudentRockStars.com Multilayered Perceptron (Contd.)

Artificial Intelligence(AI) and Neural Network

machines to perform cognitive tasks.

Representation. Reasoning. Learning

Back-Propagated Delta Rule Networks (BP)

How to Build Prior Information into Neural Network Design

Block diagram of an invariant feature-space type of system

You might also like