517 454, 517 441: Parallel and Distributed Computing

Apisake Hongwitayakorn apisake.cpsu@gmail.com

What is Parallel Computing?

Traditional Computing

Traditionally, software has been written for

serial computation.
Problem

instructions

Parallel Computing
In the simplest sense, it is the simultaneous
use of multiple computing resources to solve a computational problem.

Problem

The computer resources

The computational problem

Demonstrates characteristics such as the

ability to be:

Broken apart into discrete pieces of work

that can be solved simultaneously any moment of time

Execute multiple program instructions at Solved in less time with multiple compute
resources than with a single computer resource

Think about Natural World!!

Parallel computing attempts to emulate many complex, interrelated events
what has always been the state of affairs in the natural world happening at the same time, yet within a sequence

Grand Challenge Problems

Weather and Climate

Grand Challenge Problems

Biological, Human Genome

Grand Challenge Problems

Geological, Seismic Activity

Grand Challenge Problems

Design & Manufacturing

Grand Challenge Problems

Search for Extraterrestrial Intelligence

Grand Challenge Problems

Search for Extraterrestrial Intelligence

Commercial Applications The Driving Force

..need faster computers for processing of large amount of data in sophisticated ways

 parallel databases, data mining oil exploration web search engine, web-based business services computer-aided diagnosis in medicine management of national & multi-national corporations advanced graphics and virtual reality and a lot more...

Why Use Parallel Computing?

Primary Reasons
Save time - wall clock time Solve large problems Provide concurrency

Other Reasons...
Taking advantage of non-local resources Cost-savings Overcoming memory constraints

Limits to Serial Computing

Transmission speeds Limit to miniaturization Economic limitations

The Future
During the past 10 years, the trends
indicated by:

Parallelism is the future of computing

ever faster networks, distributed systems, multi-processor computer architectures

Who and What?

Whos Doing Parallel Computing?

Academic Industry Classied Research Government Vendor

Segments share for 11/2007

1% 3% 17% 2% 57%

20%

150

225

75

Aerospace

Benchmarking

CFD

Database

Economics

Environment

Geophysics

Information Service

Manufacturing

Media

Application Area share for 11/2007

Research

Software

Transportation

WWW

What are they using it for?

Digital Media

Gaming

Concepts & Terminology

von Neumann Architecture

Store-program concept - the CPU executes
a stored program that species a sequence of read & write operations on the memory
CPU

fetch
Memory

execute
(manipulate data as programmed)

Basic Design
Memory is used to store both program and
data instruction

Program instructions are coded data which

tell the computer to do something the program

Data is simply information to be used by A CPU gets instructions and/or data from
memory, decodes the instructions, and then sequentially perform them

Flynns Classical Taxonomy

Single Instruction Single Data Multiple Instruction

SISD

MISD

Multiple Data

SIMD

MIMD

Single Instruction, Single Data (SISD)

A serial computer Single instruction: one
instruction stream / clock cycle
load A load B C =A + B store C A=B*2 store A

time

Single data: one data

stream / clock cycle execution

Deterministic

Single Instruction, Multiple Data (SIMD)

prev instruction load A(1) load B(1) C(1)=A(1)*B(1) store C(1) next instruction prev instruction load A(2) load B(2) C(2)=A(2)*B(2) store C(2) next instruction prev instruction load A(n) load B(n) C(n)=A(n)*B(n) store C(n) next instruction

time

P1

P2

Pn

Multiple Instruction, Single Data (MISD)

prev instruction load A(1) C(1)=A(1)*1 store C(1) next instruction prev instruction load A(1) C(2)=A(1)*2 store C(2) next instruction prev instruction load A(1)

time

C(n)=A(1)*n store C(n) next instruction

P1

P2

Pn

Multiple Instruction, Multiple Data (MIMD)

prev instruction load A(1) load B(1) C(1)=A(1)*B(1) store C(1) next instruction prev instruction call funcD x=y^z sum=x^2 call sub1(i,j) next instruction prev instruction do 10 i=1,N alpha=w**3 zeta=C(i) 10 continue next instruction

time

P1

P2

Pn