IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-26, NO.
2, MARCH 1980
Communicafionr. Urbana, IL: Univ. of Illinois, 1949.
I121 T. Berger, Rate Disfortion Theoty. Englewood Cliffs, NJ: Pren-
tice-Hall, 1971.
I131 R. E. Blahut, “Computation of channel capacity and rate-distor-
tion functions,” IEEE Trans. Inform. Theory, vol. IT-18, pp.
460-473, July 1972.
[I41 W . A. Finamore, “Optimal encoding of continuous-amplitude
sources with constrained size output alphabet,” Ph.D. dissertation,
Dep. Elec. Comput. Eng., Univ. Wisconsin-Madison, Madison,
W I, May 1978.
iI51 A. J. Viterbi and J. K. Omura, “Trellis encoding of memoryless
discrete-time sources with a fidelity criterion,” IEEE Trans. In-
form. Theory, vol. IT-20, pp. 325-332, May 1974.
1161 C. R. Davis and M. E. Hellman, “On tree coding with a fidelity
criterion,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 373-378,
July 1975.
Towards a General Theory
IMRE CSISZAR and JANOS KGRNER
Absttzzct-A unified approach to muhitermlnal source codll problem
not lnvolvlng rate4Jistortlon theory is presented. It is shown that, for
determlnlng the achievable rate region, attention may be restricted to
source networks of a relatively simple structure. A product space char-
acterization of the achievable rate region pinpoints the mathematical
problem to be solved for getting a single letter characterization. The
complex&y of this problem depends on a structural condltiou, viz., the
number of encoders of a certain kind in the source network. Tbls approach
yields all the known slugle-letter cbaracterizatious of achievable rate
regions and a number of new ones for more complex networks. As a
digression, for a class of source networks including that of Slepiau and
Wolf, exponential error bounds are derived which are attainable by unlver-
sal codes. These huunds are tight lu a neighborhood of the boundary of the
achievable rate region.
I. INTRODUCTION
ANY COMMUNICATION situations can be
M m o d e led by a graph (or network) having informa-
tion sources, encoders, and destinations located at its
vertices. Each encoder operates on messages from the
sources connected to it, and the resulting codeword is
m a d e available (without noise) to all destinations to which
the encoder is connected. Each destination must be able
to reproduce accurately the messagesfrom certain speci-
Manuscript received January 19, 1979. This paper was presented at the
Czech-Hungarian-Soviet Seminar on Information Theory, Tsahkadsor,
Armenian SSR, USSR, September 1978, and at the IEEE International
Symposium on Information Theory, Grignano, Italy, June 25-29, 1979.
The authors are with the Mathematical Institute of the Hungarian
Academy of Sciences, H-1053 Budapest, Realtanoda u. 13-15, Hungary.
0018-9448/80/0300-0155$00.75 01980 IEEE
155
1’71 J. B. Anderson and S. Mohan, “A systematic analysis of cost for
sequential coding alaorithms,” Commun. Res. Lab.. Facultv of
En&, McMaster-Univ., Hamilton, ON, Canada, Tech. Rep.-No. .,
CRL-56, July 1978.
I181 J. B. Anderson, “Algorithms for tree coding with a fidelity criter-
ion,” Ph.D. dissertation, School of Elec. Eng., Cornell Univ.,
Ithaca, NY, May 1972.
1191 T. Berger, F. Jelinek, and J. K. Wolf, “Permutation codes for
sources,” IEEE Trans. Inform. Theoty, vol. IT-18, pp. 160-169,
Jan. 1972.
1201Y. Linde and R. M. Gray, “A fake process approach to data
compression,” IEEE Trans. Comm., vol. COM-26, pp. 840-847,
June, 1978.
Pll F. Jelinek and J. B. Anderson, “Instrumentable tree encoding of
information sources,” IEEE Trans. Inform. Theory, vol. IT-17, pp.
118-l 19. Jan. 1971.
of Source Networks
fied sources from the codewords it has received. A prob-
lem of primary interest is to determine the achievable rate
region, i.e., those rates of the individual encoders for
which the reproduction requirements can be met. These
m u ltiterminal source coding problems are substantially
more difficult than the standard textbook material ‘con-
cerning the encoding of a single source. An essential
feature is that the answer cannot be given in terms of the
individual sources only; rather, it strongly depends on the
joint statistics of the sources, which are assumed to be
known.
The first m a jor contribution to m u ltiterminal source
coding was the paper of Slepian and W o lf [l]. They
considered the case when each source is connected to a
separate encoder, and these encoders are connected to a
single destination which must be able to reproduce all the
sources with a small probability of error. They derived the
surprising result that the sum of the rates of the individual
encoders can be m a d e as small as the rate of a common
encoder connected with all sources, i.e., as small as the
joint entropy of these sources. M u ltiterminal source
coding theory has been further developed by Wyner and
Z iv [2], Ahlswede and Kiirner [3], Wyner [4], Cover [5],.
and quite a few others, cf. the surveys by Korner [6],
Berger [7], and G e lfand and Prelov 181.
Most of the presently known m u ltiterminal source
coding theorems have been obtained by rather similar
methods. This suggeststhat a general theory should exist
containing them as special cases.One aim of this paper is
156 IEEE TRANSACTIONS
to attempt such a theory, i.e., a unified approach to a
large class of multiterminal source coding problems. This
class will be specified by the definition of a source network
given in Section II. Informally we restrict ourselves to 1)
discrete memoryless sources, 2) graphs in which no edge
joins two sources or two encoders or two destinations, and
3) graphs in which the destinations are required to repro-
duce the messages of the specified sources with small
probability of error.
It should be noted that a few interesting results (cf.
Wyner-Ziv [9], Berger [7]) do not fit into this restricted
framework. While our definitions could be easily extended
to cover multiterminal rate-distortion problems, no pre-
sently known general methods yield single-letter char-
acterizations of achievable rate regions for the type of
problems we have excluded.
It will be shown in Section IV that from the point of
view of determining the achievable rate region attention
may be restricted to source networks of a relatively simple
structure called normal source networks. A subclass of
normal source networks will be studied in more detail in
Section III. For this subclass, exponential error bounds
will be obtained which are tight in a neighborhood of the
boundary of the achievable rate region. Further, these
bounds are universally attainable, i.e., they are attainable
by encoders and decoders not depending on the source
statistics. For the particular case of the network of Slepian
and Wolf [l], the results of Section III were obtained
jointly with K. Marton. A characterization of the achiev-
able rate region of arbitrary normal source networks is
given in Section V. As one would expect, this is a non-
computable “product space characterization.” Its main
relevance is to show which mathematical problem ought
to be solved in order to get a single-letter characterization.
Thus one is led to the entropy characterization problem of
Section VI. The approach of Sections V and VI gener-
alizes the one introduced by Ahlswede and Korner [3] for
solving the “side information” problem. Of course, a
complete solution of the entropy characterization problem
is a long way off. Still the special case already settled
yields the single-letter characterization of the achievable
rate region of quite a few source networks, including all
those for which such a characterization has been pub-
lished previously. This will be illustrated by the examples
of Section VII.
Much of the ideas of this paper were influenced by our
previous joint research with R. Ahlswede, P. Gacs, and K.
Marton. Also, a referee called our attention to a paper of
Han and Kobayashi [18] submitted to this TRANSACTIONS
about the same time as our paper. In [ 181source networks
similar to our normal source networks are considered and
an inner bound to the achievable rate region is derived.
The results of the two papers do not overlap in any
significant manner. In particular, while [18] also contains
a product space characterization of the achievable rate
region, our characterization in Section V is more explicit
and, unlike the one in [18], yields single-letter characteri-
zations in quite a few cases as shown in Section VII.
ON INFORMATION THEORY, VOL. IT-26, NO. 2, MARCH 1980
II. DEFINITIONS
We shall consider random variables (RV’s) X, taking
values in finite sets %a where a ranges over some finite
index set &?.For any subset C of @ we write
f& A & %, xe k vc7Le*
X,” resp. Xi will be a shorthand notation for a k-length
sequence of RV’s {Xa,i}$C1 resp. {XL,i}$=i.
Definition 1: A discrete memoryless multiple source
(DMMS) with generic variables X,, a E &?, is a sequence
lx@,i>E 1
of independent replicas of X, The sequence
ixa,i>E-l of independent replicas of X, is called the
component source of index a of this DMMS.
Let disjoint finite sets Ci?,% , t?, and a directed graph
with vertex set @ u ‘$J u C?be given such that no edge ends
in @, no edge starts in 6?, and there are no edges between
vertices in 3. The elements of Ci?,Ci3,and 6? will be called
input, intermediate, and output vertices, respectively. For
any vertex o we denote by S, the set of starting points of
edges leading to v. Further, for each output vertex c, let
5 c 6? denote the set of all input vertices from which
thcereis a path to c.
We shall regard the vertices in &?, %I, and 6? as loca-
tions of sources, encoders, and destinations, respectively,
the edges of the graph describing their connections. Then
YC gives the locations of those sources from which infor-
mation can be conveyed to the destination at c. We shall
assign to each output vertex c E C?a set qC c TC of input
vertices interpreted as the locations of the sources to be
reproduced for the destination at c.
Definition 2: A source network is defined by a graph as
above, a DMMS with generic variables X,, a E 8, and sets
‘QCc TC,,c E C?.The generic variables of the component
sources will also be called the generic variables of the
source network.
Source networks will often be given graphically. The
vertices of the graph will be represented by boxes. We
write the generic variables of the component sources into
the boxes representing the input vertices and put the
symbol X0 into the box of an output c whenever a E ‘%I=;
e.g., the source network of Slepian and Wolf [l] is given
by Fig. 1, and the zigzag source network of Kiirner and
Marton [lo] is given by Fig. 2.
Definition 3: A k-length block code for a source
network is a family of mappings
fb: %&+Gs(tb, bEa; vcpe:&, ‘&+J%i,, cEC
e
where “srt,, b E C?Iare arbitrary finite sets and %b k 5%:
if b E 6?. The mappings fb are called encoders and the ‘p,
are called decoders.
Performing the encoding and decoding operations pre-
scribed by a k-length block code on the k-length random
messageXi of the DMMS, we obtain at each,output c E C?
an RV
CSIS& AND KhUiER: GENERAL THEORY OF SOURCE NETWORKS
Fig. 1. SW sourcenetwork.
Fig. 2. Zigzag sourcenetwork.
For the destination c, an error occurs if and only if
J?kC#XGC. Thus the probability of error of this code at c
is
In addition to the error probabilities, the other m a in
indices of the performance of a code are the rates
Here, as usual, l]&l] stands for the cardinality of the range
off,, i.e., II&II p P&I.
4: A vector R= { Rb}bE3 with nonnegative
Definition
components is an c-achievable rate vector for a source
network if for every 6 > 0 and all sufficiently large k there
exist k-length block codes for this source network with
error probabilities
e, <E for every cEC?,
and rates
~wll <%+s foreverybE%.
Further, g is called an achievable rate vector if it is
e-achievable for every E> 0. The set of all these vectors is
the achievable rate region of the source network.
Notice that the achievable rate region of every source
network is a closed set by definition. Further, by a stan-
dard time-sharing argument, it is also a convex set. A
primary goal of m u ltiterminal source coding theory is to
determine (in a computable manner) the achievable rate
region of source networks. W e shall show that it is
sufficient to solve this problem for the following simpler
class of source networks.
Definition 5: A source network is called a normal
source network (NSN) if
9 there are no edges from inputs to outputs,
ii) )& ( = 1 %] and the edges from &? to 3 define a
one-to-one correspondencebetween input and inter-
m e d iate vertices,
iii) all the sets S,, c E c? are different, and
157
iv) for each pair of output vertices c’ and c”, the
inclusion $5, c SC,,implies OC,c gi),,,.
Intuitively SC,c SC,,means that all information availa-
ble at the output c’ is also available at c”. Thus the
m e a n ing of condition iv) is that a more informed destina-
tion has more demanding reproduction requirements.
Because of the one-to-one correspondence of ii) be-
tween &? and 9, for NSN’s we shall assume that the
component sources are also assigned to the intermediate
vertices b E ‘33, and the generic variables X, resp. alpha-
bets x0, a E @ , will be denoted also by X, resp. !?&,
bE93.
For NSN’s it will be significant whether or not there
exist vertices b E ?i3 such that some destination having
available information from the component source
assigned to b does not require the reproduction of this
source. For such a destination, the only role of informa-
tion stemming from the encoder at b is to help in repro-
ducing the messagesof some other component sources.
Definition 6: A vertex b E 93 of an arbitrary source
network is a helper if there exists a c E C?with b ES, such
that 5,~ qC.
For example the zigzag source network of F ig. 2 has
two helpers. However, by replacing ? in the second out-
put box by z?, the resulting NSN (which has the same
achievable rate region as the original source network) has
but one helper.
III. NORMAL SOURCENETWORKS W ITH No HELPER
NSN’s without helpers are characterized by the prop-
erty that at each output c E E! all component sources
assigned to the vertices b E S, have to be reproduced.
These source networks are uniquely determined by the
DMMS {X~,i}~~ and the sets S,c?2i3, CE&?. They are
the most direct generalizations of the network of F ig. 1
considered by Slepian and W o lf [l].
Theorem 1: The achievable rate region of an NSN
without helpers equals the set of those vectors R=
which satisfy the inequalities
{Rb)bEB
2
bEf2
Rb > H(XdXS,,!2) (1)
for every output c E L? and set C c S, which is disjoint
from every S, c S, (c’E (3).
Remark: This theorem contains the statement that the
above inequalities imply the validity of (1) for every
CCS,.
Proof: The necessity of the inequalities (1) for every
c E C?and C c S, is obvious. If these inequalities hold for
every c and C as in the Theorem then they also hold for
every c E &? and C c $5,. This can be shown recursively,
using the fact that for every e = C, u f$ with c, c SC,c s,,
k c 5, \S, we have
wwkc\~) ~~(X,*IXScYG)
+ wf~lX*c%)~
Hence it remains to show that the vectors R satisfying the
 158 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-26, NO. 2, MARCH 1980

inequalities (1) for every c E 6?, lZ c 5, are achievable rate
vectors. This can be done easily by the random partition
method of Cover [5]. We omit the details as we shall prove
a stronger result in Theorem 2 (cf. the remark following
that theorem).
We now derive universally attainable exponential error
bounds for NSN’s without helpers. These results will not
be needed in further sections. For the case I@I = 2 the
results below were obtained jointly with K. Marton. Par-
tial results for that case (with nonuniversal decoding) had
been derived earlier by Gallager [ 1l] and KoSelev [12].
Universally attainable error exponents in channel coding
have been given by Csiszar, Korner, and Marton [13], cf.
also Csiszar and Korner [14].
We need some notation. For any pair of Rys ? and Y
with values in the same finite set 9, let D( Y II Y) be the
informational divergence of their distributions, i.e.,
Pr{ ?=r}
D( YII P) A x Pr{ f=y}log
Pr{ Y=y} ’
YEY
For any k-length sequence YE qk let q,-(y) denote the yields a correct reproduction of the sequences Xb if b E
relative frequency of the symbol y in this sequence. The S\C but not if bEC.
distribution P7 A { PY_(y)}YEq is the type of J. The entropy
of the sequencejj is defined as the entropy of its type, i.e., possible types of sequences U E %$. For every P E 9, let
H(j) A H(Pj).
A minimum entropy (ME) decoder corresponding to a
mapping f: qk --+a is a mapping ‘p: %-+qk
=yiff(y)=mandH(y)<H(.?‘)
o(m) whenever f( j7) = m for f #j,
1 is arbitrary if there is no such YE ‘?Jk.
For NSN’s without helpers, we shall consider codes with
ME decoders. These codes are determined by the en-
coders fb: %~--+9?& b E %I, letting ‘p, be an ME decoder
corresponding to the Cartesian product fsc: g, +
XbE5, ?&, of the mappings fb, b E S,, for every output
CEt?.
Theorem 2: Given the set xb, b E ‘% , and any S > 0,
for sufficiently large k (depending only on the sizes I’%,]
and S) to every vector R = { Rb} bEb with nonnegative
where zs={??b}bES runs over all IS I-tuples of RV’s with
values in xb, b E 5, and Itl+ e max(t,O).
Remark: One immediately sees that the exponents
E,(R,XS ) are all positive whenever X, satisfies the condi-
tions (1)’with strict inequalities for every c E e, l? c 5,.
Thus the codes of Theorem 2, which are independent of
the source statistics, yield exponentially small error proba-
bilities whenever R is an inner point of the achievable rate
region of the source network with the particular X,.
Consider the family ‘% of of all 1%I-tuples of
Proof
mappingsf=ifb)bEB where fb: %~+-Gx, for some fixed
sets 9Rb satisfying
ForeveryfE%andsets0#CcSc% let G(S,C,f)be
the set of those 1%(-tuples U= { ?i?b} bEJ of k-length
sequences XbE 3Eg for which the ME decoder correspond-
ing to the Cartesian product of the mappings f,, bES,
Let $? be the set of those distributions on 96%which are
Tp denote the set of U of type P. Since clearly
19 I< (k+ l)‘%‘, (5)
such that it follows from Lemma A of the Appendix that for
sufficiently large k
.exp( k( b~~R,-H(P,)+H(P,“))) <l,
where P, is the marginal distribution of P on !?&
= x xb. This implies the existence of an f E 9 such
thabt?& every P E 9 and every 0# C c 5 c 91 we have
IqwLf)r--lET,I
Rb-H(-?,.#s,I?) ,
ITPI

components there exist mappings f, with domain %t and

where x,={X,,}b~% denotes a dummy RV with distribu-
$logllfbil <Rb+ 6, bE%, (2) tion P. We claim that this f = { fb} bEa satisfies the asser-
tion of the theorem. In fact, for a DMMS with generic
such that for every NSN without helpers with generic variables X,, b E %!I,all sequences in ?Jp are equiprobable.
variables X, taking values in the sets xb, b E ‘81, the code Since
, with ME decoders determined by the encoders fb has
probability of error Pr{ X&E Tp} < exp{ - kD(j?,@&}
e,<exp( -kE,(E,XsC)+k8), c~C?. (3) (cf., e.g., Dueck and Korner [15, Lemma 11)it follows that
Here for every 5 c $8
E,(E,X,) k mm Pr{X$E&(S,C,f)n9p} <exp -k D(g:ellX,)
i (
xs
+ min
e:Ozecs b~cRb-H(‘&%,,)
I1
+ ) + b;e % - H(XddS\C>
CSIS& AND KhiF3R: GENERAL THEORY OF SOURCE NETWORKS 159

whence by (5) { R;}G~G with

4, if be%,
R$ K, if bECB3,,
1 0, if b E Q2,
is achievable for $; and
iii) 5 has no more helpers than S .
As the ME decoder at c commits an error if and only if
GE u0#lszsc & (S,, C,f), the bound (3) for large k Lemma I: To-every source network S there exists a
follows from the last inequality and the well-known fact source network S without edges from inputs to outputs
that O(xsllX,) <d(x,llX,). and having the same set of helpers such that the achiev-
able rate region of S is a projection of that of $.
Rather surprisingly, the universally attainable error
bounds of Theorem 2 are asymptotically optimal in a Proof 5 is obtained from S as follows. Introduce a
neighborhood of the boundary of the achievable rate new intermediate vertex 6 and a new output vertex E for
region of any NSN without helpers, i.e., no essentially every input a E @ connected with some output vertex. Set
smaller error probabilities are attainable even if codes D,- k {a}, introduce a path a&?, and connect 6 with all
depending on the source statistics are allowed. This is seen those output vertices CEC? which are connected by an
from the following result. edge with a. F inally, delete all edges from a to output
vertices.
Theorem 3: For every 6 > 0 and sufficiently large k,
every k-length block code for an NSN with no helper Lemma 2: To every source network 5 without edges
satisfying from inputs to outputs there exists a source network S of
the same kind such that
i) the edges between the input and intermediate vert-
has error probabilities ices of 5 establish a one-to-one correspondence;
ii) the set 6 % of intermediate vertic_esof S may be
e,>exp( -kE,(E,X,C)-kS), cEC?. identified with a subset of the set ‘$Bof intermediate
Here for every s c ‘??I vertices of $ so that a vector R= { R,,}bE4 belongs
to the achievable rate region of S if and only if
E,(R,XS) A m-in~(~s/lX,) setting
*.s
where X,= {X,},,, runs over those IS I-tuples of RV’s for
which
+
m in x R, - H(zf&,) =O. the vector $= {&,},,G is an element of the achiev-
e: 0zecs bEe able rate region of s; and
This theorem can be proved by routine arguments. W e iii) with the identification of ii), S and $I have the same
o m it the proof. set of helpers.
Proof: From the point of view of an encoder resp.
IV. REDUCTIONOF SOURCENETWORKSTO encoder-decoder pair of a given source network, it is
NORMALONES natural to consider as a single source the collection of
those component sources which are encoded resp. both
In this section we reduce the problem of determining encoded and reproduced together. The idea of the proof is
the achievable rate region of an arbitrary source network to introduce a new DMMS with these sources as compo-
to the corresponding problem for an NSN. More exactly, nents.
we establish through a series of three lemmas the follow- Formally, for every fixed b E ‘% consider the nonvoid
ing reduction theorem. subsets of @ which are of the form S, n 9, for some edge
bc of 5. The collection consisting of these sets together
Theyem 4: To every source network S there exists an with S, will be denoted by A(b). Let the intermediate
NSN S with the following properties: vertices of 5 be the pairs (a’, b) fof every b E ??I and
i) for the set ‘% I and ‘% Iof intermediate vertices of S @ ’E A(b). Let the input vertices of S be in one-to-one
and S one has correspondence with its intermediate vertices; connect
each of them with the corresponding intermediate one.
Further, identifying the output vertices of 5 with those of
S , let 6= (a’, b) E ‘% be connected with c E C?if and only
ii) R= { Rb}bEs is an achievable rate vector for Ss if if b is connected with c in S and either 8’ = S, or
and only if for some constant K the vector R= @ ‘=S,n9c.
160 IEEE TRANSACTIONS ON INFORhfATION THEORY, VOL. IT-26, NO. 2, MARCH 1980

Finally, consider a DMMS with generic variables Y,-,

~E%I, where
:ry
Y,-= %,’ if b”=(@‘,b).
The definition of $ is completed by assigning the compo-
nent source with generic variable Y,- to the input vertex + L-l
connected with 6 and requiring at each output c the Fig. 3. “Two-help-one”sourcenetwork.
reproduction of this component source if and only if the
vertex b = (@‘,b) is connected with c and @’c 9=.
The source network S obtained in this way clearly with an a E qc. Then G, c S, and Definition 6 can be
satisfies the assertions of the lemma if we identify the rewritten as follows.
intermediate vertices (S,, b), b E ‘Zi3,with the original b. Definition 6* : A vertex b E 9 of an NSN is a helper if
Lemma 3: Given any source network, one can modify bES,\&, for some cE(?.
the sets Qc, c E (2, without changing the achievable rate Theorem 5: The achievable rate region of an NSN with
region and without increasing the set of helpers so that for set of helpers X equals the closure of the set of those
every pair of output vertices c’ and C” the inclusion vectors R= {Rb}bES to which for some n > 1 there exist
S, c S,. implies 6iJc,c 9cPPfor the modified qc. After this functions fb, b E X with domain ?$ (and arbitrary range)
modification, if there are several outputs with the same such that
S,, one can delete all but one of them without changing
the achievable rate region. %a ~~(f,(x,")) (6)
Proof: For a fixed pair of outputs with S, c SC,,the
addition of the component sources to be reproduced at c’
to those to be reproduced at c” does not change the
achievable rate region. Then proceed iteratively. The last
assertion is obvious.
+ ,,Fn, fwxx,“>) (7)
Lemmas l-3 prove that to every source network one
can easily construct an NSN (with, at most, the same for every c E e and C c G, disjoint from every &,, such
number of helpers) such that the achievable rate region of that S, c S, with c’#c.
the former can be immediately obtained from that of the Proof: To prove the converse part, let R’ = { RL}bE3
latter. In particular, in order to get a computable char- be an achievable rate vector. Then, by Definition 4, to
acterization of the achievable rate region of an arbitrary arbitrary small 6 > 0 and E> 0 there exists an n-length
source network it suffices to derive a computable char- block code (for sufficiently large n) such that the encoders
acterization of that of the corresponding NSN. It should fb, b E 9~ satisfy
be noted, however, that for determining the achievable
rate region of a particular source network, a complete +ogI~f,ll <R;+ 6, bE% (8)
solution of the corresponding NSN problem may not be
needed, as is obvious from the reduction steps above. and the error probabilities satisfy
ec <c, CEC?. (9
V. NORMALSOURCENETWORKS WITH ANY It suffices to show that setting Rb p R,‘+26 and properly
NIJMBER OF HELPERS choosing E, the encoders fb for the helpers b E X satisfy all
While for NSN’s without helpers the achievable rate the stated inequalities.
region is easy to characterize, for arbitrary NSN’s no Clearly (8) implies the inequality (6) for every b E x ;
single-letter characterization is known. Already the simple thus it is enough to show that
NSN of Fig. 3 represents a very hard problem, cf. Kiirner
and Marton [16]. We give next a (noncomputable) prod- x & 2 ~H(X~Ix~c,, {fb(Xbn))bE&“%) (lo)
bet?\%
uct space characterization of the achievable rate region of for every c E (? and l? c &, . On account of (8) and the fact
arbitrary NSN’s. The main significance of this result con- that $5,= G, u (S, n X) (cf. Definition 6*), we have
sists in reducing the problem of single-letter characteriza-
tions to a common mathematical problem of a rather
simple structure. As another application of independent
interest we shall show at the end of this section that the
so-called “optimistic” and “pessimistic” viewpoints in def-
ining the achievable rate region of a source network lead
to the same result.
For an arbitrary NSN, let us denote by G, the set of
those intermediate vertices b E %I which are connected
CSISZh AND K&NER: GENERAL THEORY OF SOURCE
Using (9) and Fano’s inequality the last term is lower
bounded by
(where h(e) stands for the binary entropy function), so
that (10) follows.
Turning to the direct part, it is enough to show that if R
satisfies the inequalities (6) and (7) of the theorem for
some n and functions fb, b E X, then E is an achievable
rate vector. To this end we construct, for the given NSF
S and the functions fb, bE X, an auxiliary NSN S
without helpers.
Write & P u .Ee&,. To every b E X n & introduce a
new intermediate vertex b+ and a new input vertex con-
nected with b +. Connect b+ with those c for which
b E G ,. Further, to every b E X introduce a new output
vertex and connect it with b. Define a new DMMS with
generic variables assigned to the intermediate vertices of
the m o d ified graph, setting
Y, p X,“, if bBX,
if bEX,
fb(Xt37
Y,+ 42 x;, if bEXn&.
Let $ be the NSN without helpers uniquely defined by
lx
thi assignment.
this
W e claim that if a vector 2,
R, with components ib,
b E 3, and %+, b E X n & isanachieva
bE%,andi?b+,bEXn& is an achievable rate vector for
$ then
R, g
$%, if b@Xn&,
;( ib + ib+), if bEXn&,
defines an achievable rate vector i k { Rb} bEb for S .
TO, see this, observe first that every I-length block code
for S gives rise to an In-length block code with at most
the same error probabilities for S ; further, the rates of
this code for S are related to the rates of the code for S
asin(ll).Infact,theencodersf?,,b_E%,and~~+,bEXf
G , of any I-length block code for S define m a p p ings Fb
resp. Fb+ with d o m a in Xk such that
I;,+(X’“) =fb+( Y;+).
&ix”) =fb( ybl)p
Consider the m a p p ings Fb, b 6 % n G , resp. the pairs
(Fb,Fb+), b EX n G , as encoders of an In-length block
code for S. Every decoder I$~ of the code for S which
corresponds to an output vertex c E e has a range defined
as a Cartesian product containing all the sets % F , b E F,,
as components. Thus one can define the corresponding
decoders of the code for S by projecting the value of @c
on % tc. Clearly the code obtained in this manner for the
source network S d_oeshave the announced properties.
This proves that if R” is an achievable rate vector for 5
and E is defined by (1 l), then to every e > 0, 6 > 0, and
sufficiently large integral m u ltiple k^= In of n there exists a
NETWORKS 161
i-length block code for S with rates
and error probabilities satisfying (9).
Now we show that tl+ implies the same result for every
large k, and therefore R is an achievable rate vector for S
as we have claimed. To this end, consider any k with
(I - 1) n <k <In and consider a code of block length i= In
for S with encoders fb, b E%, and decoders $$,, CE e,
satisfying (12) and (9). One can apply this code to k-
length sequencesby adding to the latter arbitrary suffixes
of-length i-k. Formally, for every L={Zb}bE9, ?$,E
%z- k, define k-length block encoders fb,i by
where Z,,Zbstands for the jUXtapO$tiOnO f .?band ?& Let 9$
be defined by deleting the last k- k symbols from every
component sequenceof the values of $. Denoting by e,,,
the error probabilities of the code with encoders fb,L,
b E $i3 and decoders ‘pc,c E e, (9) implies
It follows that for some ZE!?&” we have e.., ;<elel08 for
every c E e. This completes the proof of our ilaim con-
cerning (11).
Now the direct part of the theorem will follow if we
show that whenever E satisfies the inequality of the state-
ment of the theorem then
,
(11)
kb+ = nRb - H( fb(X:)),
if bEYC,
if b@X,
_ if b E X n & ,
(13)
defines an achievable rate vector d for 5. Notice that $ is
an NSN without helpers for which the set of intermediate
vertices connected with an output vertex c E e is S, u &:,
where
G: b {b+; bE&,nX},
Further, the outputs of 5 not belonging to e are in
one-to-one correspondence with the elements of X, each
being connecte$only with the corresponding b E x. Thus
by Theorem 1 R is an achievable rate vector for $, if
&, 2 H( y,,), bEX, (14)
and
x ‘b+ x kb+ >H(Y,Y,+IYS,\e,Y,:,,+) (15)
bEf? b+EfZ.+
for every C c S, \X, C+ c G : such that C u C+ is disjoint
from every S, u G : properly contained in S, u &: . Since
S,\X=&,\X and S,u&~cS,u&,+ if and only if S,
c S,, the inequalities (14) and (15) follow from (6), (7),
and (13) by the definition of the RV’s Y, and Y,+.
Notice that Definition 4 of the achievable rate region
represents a “pessimistic” viewpoint. Instead of postulat-
162
ing the existence of good codes for every sufficiently large
k, an optimist would already call an F an achievable rate
vector if to every E> 0 and 6 > 0 there exists a k-length
block code (for just one k) satisfying the inequalities of
Definition 4. This alternative definition is quite common
in the literature.
An equivalent way of formulating the “optimistic” defi-
nition is to say that E is an achievable rate vector if for
some sequence of positive integers k, there exist k,-length
block codes of rates converging to the components of R
and error probabilities converging to zero. The point is
that here k,, may be any sequence, for example, even
k,+,/k,+w is possible.
Corollary: For every source network the optimistic and
pessimistic definitions lead to the same achievable rate
region.
Proof: As in the reduction of the problem of de-
termining achievable rate regions to the special case of
NSN’s, one sees that it is enough to prove the theorem for
normal source networks. Thus it suffices to prove that
Theorem 5 holds also with the optimistic definition of the
achievable rate region. As the optimist’s rate region could
only be larger, the statement follows since the proof of the
converse part of Theorem 5 applies literally to the “opti-
m istic” case.
VI. THE ENTROPY CHARACTERIZATION PROBLEM
Because of Theorem 5, to give a single-letter characteri-
zation of the achievable rate region of an NSN it is
sufficient to characterize the range of certain vectors of
information quantities involving as many arbitrary func-
tions as there are helpers in this NSN.
In particular, in the case of one helper, it suffices to
determine the set of all possible vectors of form
Here n ranges over the positive integers and f ranges over
functions with domain Fb,, where b, is the helper. In fact
the information quantities figuring in (6) and (7) for
X = {b,} can be immediately expressed in terms of the
components of such a vector and the known joint entro-
pies of the generic variables X,, b E ‘33. Since the RV’s X,,
J c $i3 can be considered as the generic variables of a new
DMMS, it will be convenient to formulate the following.
One-Parameter Entropy Characterization Problem: For
an arbitrary D M M S with generic variables Y, taking
values in finite sets % ,, a E @ , give a single-letter char-
acterization of the set of all J&]-dimensional vectors of the
form
where n=1,2;-. and f ranges over functions with
domain 9:,, a, being some fixed element of 8.
IEEE TRANSACTIONS
ON INFORMATIONTHEORY,VOL. 1~26, NO. 2, MARCH 1980
Clearly, if this problem could be solved for every
DMMS, then Theorem 5 would yield a single-letter char-
acterization of the achievable rate region of every NSN
with one helper. More generally, the same is true for
NSN’s with any number of helpers provided that no
output is connected with two helpers and no helper is
connected with two outputs.
Similarly, one could formulate an r-parameter entropy
characterization problem, the solution of which would yield
a single-letter characterization of the achievable rate re-
gion of every NSN with r helpers. Unfortunately, even the
two-parameter entropy characterization problem is be-
yond the reach of our present methods. In particular, it
seems already very hard to give a computable characteri-
zation of the set of all three-dimensional vectors
f ~(X”lf(X”)), ; NY7 g( Y”)),
(
; H(Z”lf(X”M Y ”)))
for a D M M S with generic variables X, Y, Z. Such a
characterization would lead to that of the achievable rate
region of the “two-help-one” source network of Fig. 3; the
difficulties of the latter problem are discussed in Korner
and Marton [16], cf. also Csiszar and KSrner [14].
While we believe that entropy characterization prob-
lems are the mathematical core of source network theory,
it is not the aim of this paper to enter into them in detail.
We lim it ourselves to illustrating the power of our ap-
proach by showing (in the next section) that even the
presently available partial results on the one-parameter
entropy characterization problem yield the achievable rate
region of quite a few source networks. For this purpose we
now state without proof two results from Csiszar and
Kiirner [14] which were obtained jointly with K. Marton.
The first one is the solution of the one-parameter entropy
characterization problem for D M M S ’s with three compo-
nents. The second result, a slight generalization of the
first, is a partial result on the same problem for D M M S ’s
with four components.
Theorem 6: Given a D M M S with generic variables X,
Y, 2, the closure of the set of all vectors of the form
(+wlf(X”))>; w~nlf(x"))?~ H(z"lfW)),
(with arbitrary positive integer n and function f) consists
of the vectors (a, b, c) satisfying
max[I(X/\YlU),I(X/\ZIU)]+t<a<H(XIU)+t
b=H(Y(U)+t, c=H(Z(U)+t
for some RV U and nonnegative number t where
i) the range % of U satisfies 1% I < 1% I + 2,
ii) the RV’s U, X, YZ form a Markov chain in this
order, and
iii) 0 < t < m in[1( U/\ Y), I( U/\ Z)].
CSISZuiR AND KiiRNER: GENERAL THE?ORY OF SOURCE NETWORKS 163

Theorem 7: G iven a DMMS with generic variables X,

S, Y, Z such that X, S, YZ form a Markov chain in this
order, the closure of the set of all vectors of the form

(with arbitrary positive integer n and function f) consists

Fig. 4. Source network with two helpers.
of the vectors (a, b, c) satisfying
max[I(X/\YlU),I(Xr\ZIU)]+H(SIX)+t
<a<H(SIU)+t,
b=H(YlU)+t, c=H(ZIU)+t,
for some RV U and nonnegative number t where
i) the range % of U satisfies IG21 I < I% I + 3,
ii) the RV’s U, X, S, YZ form a Markov chain in this Fig. 5. NSN corresponding to zigzag source network.
order, and
iii) 0 < t < min[ I( U/\ Y), I( Ur\Z)].
Remark: Setting X= Z in Theorem 6 we see that the
closure of the set of all vectors of the form

consists of the vectors (a, b) satisfying

Fig. 6. Source network of Example 3.
b-H(YIX)<a<H(XjU)+t, b=H(Y(U)+t,
for some U and t as in Theorem 6. It is easy to show that and ii) of Theorem 6. The result of this example is due to
the set of these (a, b) is the same as the set of (a, b) with Sgarro [ 171.
b-H(YIX)<a<H(XJU), b=H(YlU). Example 2: Consider the source network of Fig. 2,
This special case of Theorem 6 has been obtained by interchanging for notational convenience the RV’s X and
Ahlswede and Korner [3]. Y. By Lemma 3, this source network has the same achiev-
able rate region as the NSN with one helper shown in Fig.
5. By Theorem 5, the achievable rate region is the closure
VII. EXAMPLES of the set of those triples (Rx, R,, R,) which for some n
and mappings f with domain % ” satisfy
Example 1: The NSN of Fig. 4 has two helpers. Still,
as no two helpers are connected with the same output and R, 2 fJ( Y),
each helper is connected but with one output, the achiev- Rx > ; H(X”I Y”,f(X”)) + + H(f(X”)),
able rate region of this source network can be determined
via a one-parameter entropy characterization problem. By
Theorem 5, the achievable rate region is the closure of the R, > $ H(Z”I f(X”)).
set of those triples (R,, R,, R,) which for some positive Noticing that
integer n and mappings f and h with domain %?’resp. %*
satisfy Hf(X”))+ NX”I Y”,f(X”))
= nH(X, Y) - H( Y”( f(X”)),
Rx> +(f(X")), R, 2 ; ff( Y”lf(x”)), it follows from Theorem 6 that (Rx, R,, R,) is achievable
if and only if
R, > +f( Y”lh(Z”)), R, > ; H(h(Z”)). R,>H(X,Y)-H(YIU)-t,
R,=(Y),
Writing H(f(X”)) = nH(X) - H(X”] f(X”)), by the remark
at the end of Section VI the first two conditions have the R,>H(ZIU)+t,
single letter form where U and t are the same as in Theorem 6. The
single-letter characterization of the achievable rate region
Rx > I(XA U), R,>H(YIU); of this source network has been given by Korner and
similarly, the last two conditions become Marton [lo].
Example 3: By Theorem 5, the achievable rate region
R, 2 ff( YI v>, R, >I(ZA V), of the NSN with one helper of Fig. 6 is the closure of the
where U and V are arbitrary RV’s satisfying conditions i) set of those triples (R,,R,,R,) which for some n and
164 IEEE TRANSACTIONS ON INFORMATIONTHEORY,VOL. IT-26,N0. 2, MARCH 1980

mapping f satisfy
R,=(YIX),

Rx 2 ; H(X”( Yn,f(X”)) + ; ff(f(X”)),

Rx+ R,> ;H(X’Y’lf(X”))+ $H(f(X”)),

R, > ; H(Z”l f(X”)).

These conditions can be equivalently written as

R, > H( YIX),
R, >H(XY) - $ H( Y”J f(T)),

Rx + R, > H(XY),
Fig. 7. Source network and corresponding NSN.

R, 2 $ H(Z”(f(X”)).

By Theorem 6, we get the following single-letter char-

acterization of the achievable rate region:
R,)H(YlX),
R,>H(X,Y)-H(YJU)-t,
Rx + R, > H(XY),
Fig. 8. Source network of Example 5.
R,>H(Z)U)+t,
where U and t are as in Theorem 6. Notice that Example see that (R,, R,, R,) is an achievable triple for the NSN if
3 contains Example 2 as a special case. and only if for some positive integer n and mapping f with
Example 4: Consider the source network of Fig. 7(a), domain %? it satisfies the inequalities
denoting the three intermediate vertices by 1, 2, and 3. By
the reduction steps of Lemmas 1-3, (R,,R,,R,) is an R, ) ; H( Y”lf(X”)),
achievable rate vector for this source network if and only
if (H( Y),R,,R,, R3) is an achievable rate vector for the
R, ) ; H(Z”lf(X”)),
NSN of Fig. 7(b). This NSN has one helper, viz., the
vertex 1. By Theorem 5 the achievable rate region is the
closure of the set of those vectors (R,, R,, R,, R3) which for Rx > ; H(X” I Y”Xn,f(X’)) + ; H( f(X”)).
some n and f satisfy Noticing that
R, ) ff( Y), fqX”I YnZn,f(X9) + Wf(X”))
R, ) ; H(X”I Y”,f(X”)) + i H(f(X”)), = nH(X, Y, Z) - H( YT” If(X”)),
we see that Theorem 7 applies. In fact, considering a
R,) $ W(X”If(X”)), DMMS with generic variables X, S, Y, Z such that S
equals the pair YZ, the Markov chain condition of that
R, > ; H(Z”(f(X”)).
theorem is fulfilled. Thus our previous inequalities yield
that (R,, R,, R,) is an achievable rate triple for our NSN
By the same observation as in the previous example, it if and only if
follows from Theorem 6 that (R,, R,, R,) is an achievable
rate vector of the source network of Fig. 7(a) if and only if R,>H(Y(U)+t,
R, >H(ZIU)+ t,
R,>H(X,Y)-H(YIU)-t,
R,>H(X,Y,Z)--H(YZIU)-t
R, > max[ 1(X/\ YI U),I(XAZI U)] + t,
for some U and t as in Theorem 7.
R,>H(ZIU)+t,

for some U and t as in Theorem 6. VIII. CONCLUSION

Example 5: Consider the source network of Fig. 8. By
Lemma 3, this source network has the same achieyable
rate
-IA region as the NSN obtained upon replacing X by
XYZ at the middle output vertex. Applying Theorem 5 we
We have attempted to develop a general theory of
source networks. We have shown that in order to
determine the achievable rate region of an arbitrary
CSISzliR AND KihER: GENERAL THEORY OF SOURCE NETWORKS
source network it is sufficient to solve this problem for a Now
corresponding normal source network. A product space
characterization of the achievable rate region of NSN’s
led us to the entropy characterization problem as the
mathematical core of source network theory. The achiev-
able rate region of an NSN is determined by a boundary
(from some direction) of the set involved in the corre-
sponding entropy characterization problem. At present,
only the one-parameter entropy characterization problem
can be tackled. Using recent results in this direction
jointly obtained with K. Marton, single-letter characteri-
zations of the achievable rate region of many source Substituting (Al) and (4), the assertion follows.
networks can be derived. This is illustrated by five exam-
ples, three of which are original. Our approach offers the
possibility of a systematic exploration of the power of
presently known methods. It appears that the difficulty of
a source network problem depends on the number of
helpers and their connections to the decoders in the
corresponding NSN. In particular, for the case of more
than one helper, essentially new ideas will be needed,
unless all the edges between helpers and outputs are
disjoint.
One aim of this paper was to convince the reader that
future research in multiterminal source coding theory
should be concentrated on the entropy characterization
problem, rather than on individual source networks.
APPENDIX
Lemma A: With the notation of the proof of Theorem 2, we
haveforeveryPE??,0#CcScQ
. exp b$ % - H(P,) + ff(f’s,e) + ;
Proof: For a fixed fE%, a 1%I-tuple tiC(%b
sequences XbE %$ belongs to J(S) lZ,jJ if and only if there exists
an 1s I-tuple i?= { yb}bES withjbE%t, bES such that
i) fb(jb)‘fb(fb) for every bE s ,
ii) jb#Eb for bee andjb=Zb for bES\e,
iii) H(5) < H(i&) where iis p { -iFb}bo$,.
In the last condition, the types of 3 and ti, are defined interpret-
ing them as k-length sequences of elements of !?‘&.
For every UE~~ denote by X(B) the set of those 3 for which
ii) and iii) hold. Then the type Q of a 6~3C(17) (which is a
distribution on CX& must satisfy H(Q) < H(P,), and the margi-
nal of Q on &,e must equal PS,e For every such Q we have by
symmetry
IX(u)nEibl=l~~lll~~p,,,l.
Hence (cf., e.g., Dueck and Korner [15, Lemma 11)
jfJC(a)nTQ,I<(k+l) ‘s’exp{k(H(Q)- ff(J’~d))-
Since X(Z) is the union of its intersections with less than
(k + 1)1%1different TQ for each of which H(Q) < H(P,), we have
lW4l G (k+ 1)2’3i”‘exp(k(H(P,)-H(P,,,))).
165
. some CE X(U) satisfies (i)} ]
y&Ji?T !{f.
< &J i~Fp I~~(Y)~{f:fb(~b)=fb(Xb)foTeve~b~e)l
P
REFERENCES
111 D. Slepian and J. K. Wolf, “Noiseless coding of correlated infor-
mation sources,”IEEE Trans. Inform Theory,, vol. IT-19, pp.
471-480. Reprinted in Key Papers in the Development of Informa-
tion Theory, D. Slepian, Ed. New York: IEEE Press, 1974.
PI A. D. Wyner and J. Ziv, “A theorem on the entropy of certain
binary sequences and applications I-II.” IEEE Trans. Inform.
Theory, vol. IT-19, pp. 769-778, Nov. 1973.
[31 R. Ahlswede and J. Khmer, “Source coding with side information
at the decoder and a converse for degraded broadcast channels,”
IEEE Trans. Inform. Theory, vol. IT-21, pp. 629-637, Nov. 1975.
[41 A. D. Wyner, “On source coding with side information at the
decoder,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 294-300,
May 1975.
[51 T. M. Cover, “A proof of the data compression theorem of Slepian
and Wolf for ergodic sources,” IEEE Trans. Inform. Theory, vol.
IT-21, pp. 226-228, Mar. 1975.
161 J. Khmer, “Some methods in multi-user communication-a tu-
torial survey,” Information Theory-New Trends and Open Prob-
lems (CISM Courses and Lectures No. 219), G. Longo, Ed.
Wien: Springer-Verlag, 1975, pp. 173-224.
source coding,” in The Information The-
171 T. Bergen “Multiterminal
ory Approach to Communications (CISM Courses and Lecture
Notes No. 229), G. Longo, Ed. Wien and New York: Springer-
Verlag, 1977, pp. 171-231.
PI S. I. Gelfand and V. V. Prelov, “Multi-user communication” (in
Russian), in Probability Theory, Mathematical Statistics and The-
. oretical Cybernetics, Vol. IS, P. V. Gamkrelidze, Ed., 1978, pp.
123-162.
191 A. D. Wyner and J. Ziv, “The rate-distortion function for source
of coding with side information at the decoder,” IEEE Trans. Inform.
Theory, vol. IT-22, pp. l-l 1, Jan. 1976.
WI J. Kiimer and K. Marton, “Images of a set via two channels and
their role in multi-user communication,” IEEE Trans. Inform
Theory, vol. IT-23, pp. 751-761, Nov. 1977.
1111R. G. Gallager, “Source coding with side information and univer-
sal coding,” (preprint), presented at the IEEE Int. Symp. on In-
form. Theoty, Ronneby, Sweden, June 21-24, 1976.
WI V. N. Koielev, “On a problem of separate coding of two depen-
dent sources” (in Russian), Probl. Peredach. Inform., vol. 13, no. 1,
pp. 26-32, 1977.
1131 I. Csisz&, J. Khmer, and K. Marton, “A new look at the error
exponent of a discrete memoryless channel” (preprint), presented
at the IEEE Int. Symp. on Inform. Theory., Ithaca, NY, Oct. 10-14,
1977.
1141 I. Csistir and J. Khmer, Information Theoty: Coding Theoremsfor
Discrete Memovless Systems. New York: Academic, to appear.
[I51 G. Dueck and J. Khmer, “Reliability function of a discrete mem-
oryless channel at rates above capacity,” IEEE Trans. Inform.
Theory, vol. IT-25, pp. 82-85, Jan. 1979.
WI J. Khmer and K. Marton, “How to encode the modulo 2 sum of
two binary sources,” IEEE Trans. Inform. Theory, vol. IT-25, pp.
219-221, Mar. 1979.
v71 A. Sgarro, “Source coding with side information at several de-
coders,” IEEE Trans. Inform. Theory, vol. IT-23, pp. 179-182,
Mar. 1977.
WI T. S. Han and K. Kobayashi, “A unified achievable rate region for
a general class of multiterminal source coding systems,” IEEE
(Al) Trans. Inform. Theory, vol. IT-26, no. 3, May 1980, to appear.