ENCS 6161

Probability and Stochastic Processes

Concordia University
Course Material 8

Sums of random variables

Let 1 , 2 , , be a sequence of random
variables
Sum:
= 1 + 2 + +
Mean: Regardless of statistical dependence of
these random variables, the expected value
of their sum is equal to the sum of their
expected values:
= 1 + 2 + +
= 1 + 2 + +
2

 Variance:
Consider

=+

= 2
= + + 2
= + 2
= [ 2 + 2
+
+
= + + 2(, )
In general, the covariance (, ) is not equal to
zero, and thus, the variance of a sum is not necessarily
equal to the sum of the individual variances.
3

 The above results can be generalized as:

= 1 + 2 + +

[ ] +
=1

[ , ]
=1 =1

If are independent or pair-wise uncorrelated

random variables, then,
, = 0 for
1 + 2 + +
= 1 + 2 + + [ ]
4

Sum of independent, identically distributed

(iid) random variables
=
= 2

= 1 + 2 + + =
Since the covariance of the pairs of independent
random variables are zero.

2 = 2

[ ] =
=1

=1
5

The pdf of a sum of independent

random variables
=+
Let the random variables and be independent
The characteristic function of is given by
=
= +
Since X and Y are independent, the functions of these
random variables, and , are also independent
random variables. Therefore,
=
=
Thus, the characteristic function of is the product of the
individual characteristic functions of and .
6

 ()) = ( )( )
Therefore,
= () ()
The above results can be generalized to the sum of
independent random variables as
= 1 + 2 + +
=
= 1+2++
= 1 2
= 1 2
Thus, the characteristic function of the sum of
independent random variables can be expressed as a
product of the characteristic functions of the individual
random variables.
7

Sum of independent Gaussian random

variables
= 1 + 2 + +
are independent Gaussian random variables
with means and variances, respectively, as
1 , 2 , , and 12 , 22 , , 2 .
=

2 /2 2

= 1, 2, ,

2
The characteristic functions of these individual Gaussian
random variables are given by

2 2
2 ,

= 1, 2, ,
8

2 2

2

=1

= exp{ 1 + 2 + +
2 12 + 22 + + 2 /2}
2 2

2

=
This is the characteristic function of a
Gaussian random variable with mean and
2
variance .

 Thus, is a Gaussian random

variable with mean
= 1 + 2 + +
and variance
2 = 12 + 22 + + 2

1
2

22

10

Characteristic function of sum of iid

random variables
If, in addition to being independent, the
random variables 1 , 2 , , are identically
distributed, then
=
=
= 1

for = 1, ,

11

Sum of iid exponential random

variables
The pdf of a single exponential random variable :
= ,
0, > 0; = 1, 2, ,
The characteristic function of :

=
,
= 1, 2, ,

Therefore, characteristic function of :

=

Thus, is an n-Erlang random variable, with its pdf given by
=

( )2
,
1 !

> 0

12

Sum of integer-valued random variables

Let be the sum of integer-valued random variables:
= 1 + 2 + +
Thus, is also an integer-valued random variable. In the
case of integer-valued random variables, it is more
convenient to work with the probability generating function:
=
The generating function of , the sum of independent
discrete random variables is given by
=
= 1+2++
= 1 2
= 1 2
where is the probability generating function of the ith
discrete random variable .
13

Sum of iid geometrically distributed

random variables
Let be the sum of iid geometrical random variables :
= 1 + 2 + +
The pmf of the random variables :
= , = 0,1, ; = 1, 2, ,
The probability generating of :

=
1
Therefore, the probability generating function of :

=
1

14

Sum of a random number of iid

random variables

=1

where is a random number and s are

independent identically distributed (iid) random
variables.
It is assumed that is independent of s . For
example, may be the number of students in a
class and may be height of the kth student.
15

 Using conditional expectation:

= |
Since

| = =

= ,
=1

we have
| =
Therefore,
=

16

Characteristic function of

=1

where is a random variable independent of the

iid random variables = 1, 2, , .
The characteristic function of is given by
=

17

 Using conditional expectation

= = |
we have
| = =

1 +2 ++

Since 1 , 2 , , are iid random variables.

| = =
=
Therefore,

| =

18

 Hence,

=
= =()
=
Thus, the characteristic function of is
obtained by evaluating the generating function of
at = ().

19

Example
The number of jobs submitted to a computer
in an hour is a geometric random variable with
parameter and the job execution times are
independent exponentially distributed random
1
variables with mean . Find the pdf for the sum

of execution times of the jobs submitted in an

hour.

20

The number of jobs submitted in an hour is a geometrically distributed random

variables:
= = ,
= 0, 1,
The generation function for is

=
1
A job execution time is exponentially distributed
= ,
0
Its characteristic function for is given by

=
Therefore, the characteristic function of is given by
=

21

=

+ 1
=

=+ 1

Therefore,
= 1

1
= 1 + 1

= + 1 e ,
0
1

22

 Interpretation:
1. With probability , there are no job arrivals, and
hence, the total execution time is zero.
2. With probability (1 ), there are one or more
arrivals and the execution time is an exponential
1
random variable with mean .

23

Sample mean
Let X be a random variable with mean: = .
Let 1 , 2 , , be independent repeated measurements of .
are iid random variables.
The sample mean of the sequence is used to estimate []:

1
=

=1

The sample mean itself is a random variable and thus, it exhibits

random variations.
We will compute the expected value and variance of in order to
assess the effectiveness of as an estimator of [].

24

Relative frequency and sample mean:

Consider a sequence of independent repetitions
of some random experiment, and let the
random variable be the indicator function for
the occurrence of event in the jth trial. The
total number of occurrences of in the first
trials is then
= 1 + 2 +

25

 The relative frequency of the event in the

first repetitions of the experiment is then

1
=

=1

The relative frequency estimator for the

probability of an event is a special case of a
sample mean.

26

 Properties of an estimator:
(i) It should give the correct value of the parameter
being estimated, i.e,, = .
(ii) It should not vary too much about the correct
value of the parameters, i.e., 2 is
small.

27

The expected value of the sample mean:

1
=

=1

1
=

[ ]
=1

Since = = for all .

1
=

=
=1

Thus, the expected value of the sample mean is

equal to = . It is because of this reason, the
sample mean is called an unbiased estimator.
28

The variance of the sample mean:

=
= 2

If = 1 + 2 + +

1
= 2 1 + 2 + +

Since are iid random variables.

2
1

= 2 2 =

Thus, the variance of the sample mean approaches zero

as the number of samples is increased

29

 Thus, the probability that the sample mean is close to the true mean
approaches 1 as n becomes very large. Using Chebysbev inequality:

[ ]
2
=
2
=

2
[ ] 2

2
1 < 2

2
< 1 2 = 1

Thus, for any choice2 of error and probability 1 , we can find a

value of as = 2 , such that is within of the true mean with

probability 1 or greater.
30

Example 1
A voltage of constant but unknown value is to be
measured. Each measurement is actually the
sum of the desired voltage and a noise voltage
of zero mean and standard deviation of 1.
= +
= 1,2,
Assume that the noise voltages are independent
random variables. How many measurements are
required so that the probability that is within
= 0.5 of the true mean is at least 0.99
31

 = 1 0.99 = 0.01
= 1
= 0.5
2
1
= 2=
= 400
2

(0.01) 0.5
Thus, if we were to repeat the measurements
400 times and compute the sample mean, on
the average, at least 99 times out of 100, the
resulting sample mean will be within 0.5 of
the true mean.

32

Weak law of large numbers

Let 1 , 2 , be a sequence of iid random
variables with mean = , then for > 0,
lim < = 1

The weak law of large numbers states that for a

large enough fixed value of , the sample mean
using samples will be close to the true mean
with high probability.
This law does not address the question about what
happens to the sample mean as a function of as
we make additional measurements.
33

 We make a series of independent

measurements of the same random variable.
Let 1 , 2 , be the resulting sequence of iid
random variables. Now, consider the sequence
of sample means that results from the above
measurements: 1 , 2 , , where is the
sample mean computed using 1 , 2 , , .

34

Strong law of large numbers

Let 1 , 2 , be a sequence of iid random
variables with finite mean = and finite
variance, then
lim = = 1

The strong law of large numbers states that

with probability 1, every sequence of sample
mean calculations will eventually approach
and stay close to = .
35

The central limit theorem

Let be the sum of n iid random variables,
1 , 2 , , , with = and = 2
= 1 + 2 + +
= =
= 2
Let be the zero mean, unit variance random variable
defined by

=

Then,

1
2 /2

lim =

2
36

 Thus, the cdf of as , approaches to

that of the zero-mean, unit-variance Gaussian
random variable, that is, as is a zero
mean, unit-variance Gaussian random variable.
According to this theorem, the summands
can have any distribution as long as they have
a finite mean and finite variance.

37