10.

Introduction to Data-Parallel architectures

• SIMD {Single Instruction Multiple Data}
• 10.1 Introduction
• 10.2 Connectivity
• 10.3 Alternative architecture

• e.g. add: (c1=a1+b1), (c2=a2+b2), (c3=a3+b3)

TECH
Computer Science
CH01
Introduction
• Single instruction, multiple data
• One instruction stream is broadcast to all processors
• Each processor, also called a processing element (or
PE), is usually simplistic and logically is essentially
an ALU;
– PEs do not store a copy of the program
nor have a program control unit.
• Individual processors can remain idle during
execution of segments of the program (based on a
data test).
• All active processor executes the same instruction
synchronously, but on different data
Cont…
• Technically, on a memory access, all active
processors must access the same location in their
local memory.
This requirement is sometimes relaxed a bit.
• The data items form an array (or vector) and an
instruction can act on the complete array in one cycle.
• Quinn also calls this architecture a processor array.
Some examples include
ILLIAC IV (1974) was the first SIMD computer
The STARAN and MPP (Dr. Batcher architect)
Connection Machine CM2, built by Thinking
Machines).
MasPar (for Massively Parallel) computers.
Data-parallel computation (bit parallel)
Scalar vs. SIMD
Ex:Multimedia Extension Architectures
• At the core of multimedia extensions
– SIMD parallelism
– Variable-sized data fields:
– Vector length = register width / type size
Application of Data-parallel Architectures:
One data entity processed by one PE
Multimedia / Scientific Applications
• Image
– Graphics : 3D games, movies
– Image recognition
– Video encoding/decoding : JPEG, MPEG4
• Sound
– Encoding/decoding: IP phone, MP3
– Speech recognition
– Digital signal processing: Cell phones
• Scientific applications
– Double precision Matrix-Matrix multiplication
(DGEMM)
– Y[] = a*X[] + Y[] (SAXPY)
Connectivity
• Important aspects of the design space of data parallel
computers is the connectivity.

• This connectivity is established between processing

elements(PEs).

• The inter connectivity methods are:

• Near neighbor, tree, pyramid and hypercube—widely
used data parallel connectivity.

• Bus ,crossbar, and multistage networks –used for

functional parallel design
Mapping Problem space into Architectural Space:
Data entity onto PE (1-to-1 mapping)
Near- Neighbor Connectivity

• It is arose in the context of spatial mapping coherent

data on SIMD systems.
• The data is spatially correlated.
• Ex: 8-connected near neighbor case
Near-neighbor connectivity (2-D: Mesh)
 Example of a 2-D Processor Interconnection
Network in a Processor Array

Each VLSI chip has 16 processing elements.

Each PE can simultaneously send a value to a neighbor.

PE =
processor
element

13
Trees and Graphs
• Near-Neighbour Connected system is specialized for
image Processing applications.
• Any general problem has a natural expression in the
form of a graph.
• The problems that are represented as hierarchical
structure like database searching, model matching and
expert system can be represented through a tree
connectivity.
• In this many simultaneous processes are executed at a
low level, then progressively fewer at higher level.
• The computations at this level depends on the results
of lower level.
Tree: 2-D hierarchy
The Pyramid
• The basic architectutre of the pyramid is a for
connected mesh.
• Connection between the layers is fixed and every
processor is connected to one element in the layer
above and four elements in the layer below.
• This approach is suitable for basic image processing
applications.
• Consider the data passing in an N ×N mesh.
• To pass an item of data from one corner of the array
to that diagonally opposite need 2N shift operations.
• If it forms a base level of pyramid ,the same operation
can be performed in 2log2N shifts.
Pyramid: 3-D hierarchy
Hypercube: 2^N nodes in N dimension
Hypercube: 4-D
Long and short-range connections
Data-parallel approaches
Principal characteristics of data-parallel systems