Communication

Communication costs
costs in
in static
static
Interconnection
Interconnection networks
networks

Principal parameters
- startup time (ts)
- per-hop time (th)
- per-word transfer time (tw)

Routing techniques
- store-and-forward routing
- cut-through routing

1
 Communication
Communicationcosts
costsdepends
dependson
onrouting
routingstrategy
strategy

Store and forward routing - the message is sending

between different processors and each intermediate processor
store it in the local memory until received the whole message

t comm = t s + (mtw + th )l
Cut-through routing - the message is divided on parts
which are sending between processors without waiting for the
whole message

tcomm = ts + lth + mtw

2 2
 Basic
Basiccommunication
communicationoperations
operations

-Simple message transfer between two processors

-One-to-all broadcast

-All-to-all broadcast

-One-to-all personalized communication

-All-to-all personalized communication

- Circular shift

3
 One-to-all broadcast
M M M M
0 1 p-1 Single-node accumulation 0 1 p-1

Mp-1 Mp-1 Mp-1

All-to-all broadcast M1 M1 M1
M0 M1 Mp-1 M0 M0 M0
0 1 p-1 0 1 p-1
Multinode accumulation

Mp-1
One-to-all personalized
M1
M0 M0 M1 Mp-1
0 1 p-1 0 1 p-1
Single-node gather

M0,p-1 M1,p-1 Mp-1,p-1 Mp-1,0 Mp-1,1 Mp-1.p-1

All-to-all personalized
M0,1 M1,1 Mp-1,1 M1,0 M1,1 M1,p-1
M0,0 M1,0 Mp-1,0 M0,0 M0,1 M0,p-1
0 1 p-1 0 1 p-1
Multinode gather

4
 One-to-all broadcast - SF
3 4
7 6 5 4

2 4
0 1 2 3
1 2 3
a) in a ring with even number of procesors.
3 4
7 6 5

0 1 2 3
1 2 3
b) in a ring with odd number of procesors.

T = (t s + tw m ) p 
 2 
one _ to _ all _ b
5 5
 One-to-all broadcast - SF

12 13 14 15
4 4 4 4

8 9 10 11

4 4 4 4
4 5 6 7

3 3 3 3
1 2
0 1 2 3

2
in a mesh with wraparound

 p
Tone _ to _ all _ b = 2(t s + t w m ) 
 2 
6 6
 One-to-all broadcast - SF

(110) (111)
3
6 7

(010) (011)
2 3 2
3 3
4 5
2 1 (100) (101)

0 1
(000) 3 (001)

in a hypercube

T one _ to _ all _ b = (t s + twm ) log p

7 7
 One-to-all broadcast - SF

{ 1} procedure ONE_TO_ALL_BC(d,my_id,X);
{ 2} begin
{ 3} mask:=2d-1;
{ 4} for i:=d-1 downto 0 do
{ 5} begin
{ 6} mask:=mask XOR 2i;
{ 7} if (my_id AND mask)=0 then
{ 8} if (my_id AND 2i)=0 then
{ 9} begin
{10} msg_destination:=my_id XOR 2i;
{11} send X to msg_destination;
{12} endif
{13} else
{14} begin
{15} msg_source:=my_id XOR 2i;
{16} receive X from msg_source;
{17} endelse;
{18} endfor;
{19} end ONE_TO_ALL_BC

A code of one to all broadcast operation in hypercube

(processor with label 0 is broadcasting its message).
8 8
 One-to-all broadcast - CT

4 4 4 4
3 3
2
15 14 13 12 11 10 9 8

1
0 1 2 3 4 5 6 7

2
3 3
4 4 4
in a ring

T one − to − all − bc = t s log p + t w m log p + th (p − 1 )

9
 One-to-all broadcast - CT

12 13 14 15

4 4 4 4
8 9 10 11

3 3 3 3
4 5 6 7

4 4 4 4
2 2
0 1 2 3

in a mesh with wraparound

T one − to − all − bc = (t s + twm ) log p + 2 th ( p − 1 )

10 10
 One-to-all broadcast - CT

0 3 1 2 3 3 4 3 5 6 3 7

in a balanced binary tree

Tone −to − all −bc = (t s + t w m + t h (log p + 1)) log p

11 11
 All-to-all broadcast - SF
1(6) 1(5) 1(4)

communication
7 6 5 4

step
(7) (6) (5) (4)

1.
1(7) (0) (1) (2) (3) 1(3)
0 1 2 3
1(0) 1(1) 1(2)
2(5) 2(4) 2(3)

communication
7 6 5 4
(6,7) (5,6) (4,5) (3,4)

step
2(6) 2(2)

2.
(0,7) (0,1) (1,2) (2,3)
0 1 2 3
2(7) 2(0) 2(1)

7(0) 7(7) ... 7(6)

communication
7 6 5 4
(1..7) (0..6) (0..5,7) (0..4,6,7)

step
7(1)

7.
(0,2..7) (0,1,3..7) (0..2,4..7) (0..3,5..7) 7(5)
0 1 2 3
7(2) 7(3) 7(4)
12 12
 All-to-all broadcast - SF

{ 1} procedure ALL_TO_ALL_BC_RING(my_id,my_msg,p,result);
{ 2} begin
{ 3} left:=(my_id - 1) mod p;
{ 4} right:=(my_id + 1) mod p;
{ 5} result:=my_msg;
{ 6} msg:=result;
{ 7} for i:=1 to p-1 do
{ 8} begin
{ 9} send msg to right;
{10} receive msg from left;
{11} result:=result ∪ msg;
{12} endfor;
{13} end ALL_TO_ALL_BC_RING;

T all − to − all − bc = (t s + tw m )( p − 1 )

13 13
 All-to-all broadcast - SF

12 13 14 15
(12) (13) (14) (15)

8 9 10 11
(8) (9) (10) (11)

4 5 6 7
(4) (5) (6) (7)

0 1 2 3
(0) (1) (2) (3)

12 13 14 15
(12..15) (12..15) (12..15) (12..15)

8 9 10 11
(8..11) (8..11) (8..11) (8..11)

4 5 6 7
(4..7) (4..7) (4..7) (4..7)

0 1 2 3
(0..3) (0..3) (0..3) (0..3)

14 14
 All-to-all broadcast - SF
procedure ALL_TO_ALL_BC_MESH(my_id,my_msg,p,result);
begin
left:= {...};
right:=(...};
result:=my_msg;
msg:=result;
for i:=1 to √p-1 do
begin
send msg to right;
receive msg from left;
result:=result ∪ msg;
endfor;
left:= {...};
right:=(...};
msg:=result;
for i:=1 to √p-1 do
begin
send msg to right;
receive msg from left;
result:=result ∪ msg;
endfor;
end ALL_TO_ALL_BC_MESH;

T all − to − all − bc = 2ts ( )

p − 1 + tw m (p − 1 )
15 15
 All-to-all broadcast - SF
(6) (7) (6,7) (6,7)
6 7 6 7
(2) 2 3 (3) (2,3) 2 3 (2,3)
(4) (5)
4 5 4 5 (4,5)
(4,5)
(0) 0 1 (1) (0,1) 0 1 (0,1)
a) Initial distribution of messages b) Distribusion before the second step

(4..7) (4..7) (0..7) (0..7)

6 7 6 7
(0..3) 2 3 (0..3) (0..7) 2 3 (0..7)

4 5 (4..7) 4 5 (0..7)
(4..7) (0..7)
(0..3) 0 1 (0..3) (0..7) 0 1 (0..7)

c) Distribusion before the third step d) Final distribusion of messages

T all − to − all − bc = 2ts ( )

p − 1 + twm (p − 1 )
16
 All-to-all broadcast - SF

{ 1} procedure ALL_TO_ALL_BC_HCUBE(my_id,my_msg,d,result);
{ 2} begin
{ 3} result:=my_msg;
{ 4} for i:=1 to d-1 do
{ 5} begin
{ 6} partner:=my_id XOR 2i;
{ 7} send result to partner;
{ 8} receive msg from partner;
{ 9} result:=result ∪ msg;
{10} endfor;
{11} end ALL_TO_ALL_BC_HCUBE;

T all − to − all − bc = (t s + twm ) log p

17 17
 One-to-all personalized - SF
T one − to − all − pers = (t s + twm )( p − 1 )

communication
7 6 5 4

step
1.
0 1 2 3
1(7)

communication
7 6 5 4

step
2.
0 1 2 3
2(6) 2(7)

7(7) ...
7(6) 7(5)

communication
7 6 5 4

step
7.
7(4)
0 1 2 3
7(1) 7(2) 7(3)
18 18
 One-to-all personalized - SF

(12..15) 12 13 14 15

(8..11) 8 9 10 11

(4..7) 4 5 6 7

(0..3) 0 1 2 3

12 13 14 15
(12) (13) (14) (15)

8 9 10 11
(8) (9) (10) (11)

4 5 6 7
(4) (5) (6) (7)

0 1 2 3
(0) (1) (2) (3)

19
T one − to − all − pers = 2ts ( p − 1 + twm) (p − 1 ) 19
 All-to-all broadcast - SF

6 7 6 7
2 3 2 3

4 5 4 5
(4..7)
(0..7) 0 1 (0..3) 0 1
a) Initial distribution of messages b) Distribusion before the second step

(6,7) (6) (7)

6 7 6 7
(2,3) 2 3 (2) 2 3 (3)
(4) (5)
4 5 4 5
(4,5)
(0,1) 0 1 (0) 0 1 (1)
c) Distribusion before the third step d) Final distribusion of messages

T one − to − all − pers = t s log p + twm (p − 1 )

20
 All-to-all personalized - SF
(4,0) (3,0)
(4,1) (3,1)
(4,2) (3,2)
(4,3) (3,4)
5 (4,5) 4 (3,5) 3
(5,0) (2,0)
(5,1) 1.
(2,1)
(5,2) communication
(2,3)
(5,3) (2,4) step
(5,4) 0 1 2
(2,5)
(0,1) (1,0)
(0,2) (1,2)
(0,3) (1,3)
(0,4) (1,4)
(0,5) (1,5)

(3,0) (2,0)
(3,1) (2,1)
(3,2) (2,4)
5 (3,5) 4 (2,5) 3
(4,0) (1,0) 2.
(4,1) (1,3) communication
(4,2) (1,4)
(4,3) (1,5)
step
0 1 2
(5,1) (0,2)
(5,2) (0,3)
(5,3) (0,4)
(5,4) (0,5)
21 21
 All-to-all personalized - SF
(2,0) (1,0)
(2,1) (1,4)
5 (2,5) 4 (1,5) 3
(3,0) (0,3) 3.
(3,1) (0,4) communication
(3,2) (0,5) step
0 1 2
(4,1) (5,2)
(4,2) (5,3)
(4,3) (5,4)

(1,0) (0,4)
(1,5) (0,5)
5 4 3
(2,0) 4.
(5,3)
(2,1) communication
(5,4)
step
0 1 2
(3,1) (4,2)
(3,2) (4,3)

(0,5) (5,4)
5 4 3
(1,0) 5.
(4,3)
communication
0 1 2 step
22 (2,1) (3,2)
22
 One-to-all personalized - SF

6 7 8
(6,0),(6,3),(6,6) (7,0),(7,3),(7,6) (8,0),(8,3),(8,6)
(6,1),(6,4),(6,7) (7,1),(7,4),(7,7) (8,1),(8,4),(8,7)
(6,2),(6,5),(6,8) (7,2),(7,5),(7,8) (8,2),(8,5),(8,8)

3 4 5
(3,0),(3,3),(3,6) (4,0),(4,3),(4,6) (5,0),(5,3),(5,6)
(3,1),(3,4),(3,7) (4,1),(4,4),(4,7) (5,1),(5,4),(5,7)
(3,2),(3,5),(3,8) (4,2),(4,5),(4,8) (5,2),(5,5),(5,8)

0 1 2
(0,0),(0,3),(0,6) (1,0),(1,3),(1,6) (2,0),(2,3),(2,6)
(0,1),(0,4),(0,7) (1,1),(1,4),(1,7) (2,1),(2,4),(2,7)
(0,2),(0,5),(0,8) (1,2),(1,5),(1,8) (2,2),(2,5),(2,8)

23 23
 One-to-all personalized - SF

6 7 8
(6,0),(6,3),(6,6) (6,1),(6,4),(6,7) (6,2),(6,5),(6,8)
(7,0),(7,3),(7,6) (7,1),(7,4),(7,7) (7,2),(7,5),(7,8)
(8,0),(8,3),(8,6) (8,1),(8,4),(8,7) (8,2),(8,5),(8,8)

3 4 5
(3,0),(3,3),(3,6) (3,1),(3,4),(3,7) (3,2),(3,5),(3,8)
(4,0),(4,3),(4,6) (4,1),(4,4),(4,7) (4,2),(4,5),(4,8)
(5,0),(5,3),(5,6) (5,1),(5,4),(5,7) (5,2),(5,5),(5,8)

0 1 2
(0,0),(0,3),(0,6) (0,1),(0,4),(0,7) (0,2),(0,5),(0,8)
(1,0),(1,3),(1,6) (1,1),(1,4),(1,7) (1,2),(1,5),(1,8)
(2,0),(2,3),(2,6) (2,1),(2,4),(2,7) (2,2),(2,5),(2,8)

T all − to − all − pers = (2 t s + t w mp )( p − 1 )

24 24
 All-to-all broadcast - SF
(0,0),(0,1),(0,2),(0,3) (0,0),(0,1),(0,2),(0,3)
(0,4),(0,5),(0,6),(0,7) 6 7 (0,4),(0,5),(0,6),(0,7)
(0,0),(0,1),(0,2),(0,3) (0,0),(0,1),(0,2),(0,3)
(0,4),(0,5),(0,6),(0,7) 2 3 (0,4),(0,5),(0,6),(0,7)

(0,0),(0,1),(0,2),(0,3) (0,0),(0,1),(0,2),(0,3)
(0,4),(0,5),(0,6),(0,7) 4 5 (0,4),(0,5),(0,6),(0,7)
(0,0),(0,1),(0,2),(0,3) (0,0),(0,1),(0,2),(0,3)
(0,4),(0,5),(0,6),(0,7) 0 1 (0,4),(0,5),(0,6),(0,7)

a) Initial distribution of messages

(6,0),(6,2),(6,4),(6,6) (6,1),(6,3),(6,5),(6,7)
(7,0),(7,2),(7,4),(7,6) 6 7 (7,1),(7,3),(7,5),(7,7)
(2,0),(2,2),(2,4),(2,6)
2 (2,1),(2,3),(2,5),(2,7)
(3,0),(3,2),(3,4),(3,6) 3
(3,1),(3,3),(3,5),(3,7)

(4,0),(4,2),(4,4),(4,6)
(5,0),(5,2),(5,4),(5,6)
(0,0),(0,2),(0,4),(0,6)
(1,0),(1,2),(1,4),(1,6)
25
(4,1),(4,3),(4,5),(4,7)
4 5 (5,1),(5,3),(5,5),(5,7)
(0,1),(0,3),(0,5),(0,7)
0 1 (1,1),(1,3),(1,5),(1,7)
b) Distribusion before the second step
 All-to-all broadcast - SF
(4,2),(4,6),(5,2),(5,6) (4,3),(4,7),(5,3),(5,7)
(6,2),(6,6),(7,2),(7,6) 6 7 (6,3),(6,7),(7,3),(7,7)
(0,2),(0,6),(1,2),(1,6)
2 (0,3),(0,7),(1,3),(1,7)
(2,2),(2,6),(3,2),(3,6) 3
(2,3),(2,7),(3,3),(3,7)

(4,0),(4,4),(5,0),(5,4) (4,1),(4,5),(5,1),(5,5)
(6,0),(6,4),(7,0),(7,4) 4 5 (6,1),(6,5),(7,1),(7,5)
(0,0),(0,4),(1,0),(1,4)
(2,0),(2,4),(3,0),(3,4) 0 (0,1),(0,5),(1,1),(1,5)
1
(2,1),(2,5),(3,1),(3,5)
c) Distribution before the third step
(0,6),(1,6),(2,6),(3,6) (0,7),(1,7),(2,7),(3,7)
(4,6),(5,6),(6,6),(7,6) 6 7 (4,7),(5,7),(6,7),(7,7)
(0,2),(1,2),(2,2),(3,2)
2 (0,3),(1,3),(2,3),(3,3)
(4,2),(5,2),(6,2),(7,2) 3
(4,3),(5,3),(6,3),(7,3)

(0,4),(1,4),(2,4),(3,4) (0,5),(1,5),(2,5),(3,5)
(4,4),(5,4),(6,4),(7,4) 4 5 (4,5),(5,5),(6,5),(7,5)
(0,0),(1,0),(2,0),(3,0)
(4,0),(5,0),(6,0),(7,0) 0 (0,1),(1,1),(2,1),(3,1)
1
26 d) Final distribution of messages (4,1),(5,1),(6,1),(7,1)
 All-to-all personalized - CT
6 7 6 7
2 3 2 3

4 5 4 5
0 1 0 1
a) label XOR 1 b) label XOR 2

6 7 6 7
2 3 2 3

4 5 4 5
0 1 0 1
c) label XOR 3 d) label XOR 4

6 7 6 7
2 3 2 3

4 5 4 5
0 1 0 1
27 27
a) label XOR 5 b) label XOR 6
 All-to-all personalized - CT

6 7
0 -> 1 -> 3 -> 7
2 3 1 -> 0 -> 2 -> 6
2 -> 3 -> 1 -> 5
4 5 3 -> 2 -> 0 -> 4
4 -> 5 -> 7 -> 3
0 1 5 -> 4 -> 6 -> 2
6 -> 7 -> 5 -> 1
c) label XOR 7
7 -> 6 -> 4 -> 0

{ 1} procedure ALL_TO_ALL_PERSONAL(my_id,M,d,R);
{ 2} begin
{ 3} for i:=1 to 2d-1 do
{ 4} begin
{ 5} partner:=my_id XOR i;
{ 6} send Mpartner to partner;
{ 7} receive Rpartner from partner;
{ 8} endfor;
{ 9} end ALL_TO_ALL_PERSONAL;

28 28
 Circular Shift - SF

12 (12) 13 (13) 14 (14) 15 (15)

8 (8) 9 (9) 10 (10) 11 (11) 1.

communication
step
4 (4) 5 (5) 6 (6) 7 (7)

0 (0) 1 (1) 2 (2) 3 (3)

12 (15) 13 (12) 14 (13) 15 (14)

2.
8 (11) 9 (8) 10 (9) 11 (10)
communication
step
4 (7) 5 (4) 6 (5) 7 (6) (correction)

0 (3) 1 (0) 2 (1) 3 (2)

29 29
 Circular Shift - SF
12 (11) 13 (12) 14 (13) 15 (14)

8 (7) 9 (8) 10 (9) 11 (10)

3.
communication
4 5 (4) 6 (5) 7 (6) step
(3)

0 (15) 1 (0) 2 (1) 3 (2)

12 (7) 13 (8) 14 (9) 15 (10)

8 (3) 9 (4) 10 (5) 11 (6) Final distribution

of the data
4 (15) 5 (0) 6 (1) 7 (2)

0 (11) 1 (12) 2 (13) 3 (14)

30 30
 Circular Shift - SF
(4) (3)
4 5 (5) 4 5 (2)
(3) (0)
3 2 (2)
3 2 (1)

7 (7) 6 (6)
7 (4) 6 (5)
0 (0)
1 (1) 0 (7)
1 (6)
a) First communication step of the 4-shift b) Second communication step of the 4-shift

(0) (7)
4 5 (1) 4 5 (0)
(7) (6)
3 2 (6)
3 2 (5)

7 (3) 6 (2)
7 (2) 6 (1)
0 (4)
1 (5) 0 (3)
1 (4)

c) One communication step of the 1-shift d) Final distribusion after the 5-shift

31