Ignacio Cascos Fernndez a Department of Statistics Universidad Carlos III de Madrid

Parameter estimation
Statistics 20112012
A random sample of size n is formed by n independent random variables X1 , X2 , . . . , Xn sampled from the same population. That is, X1 , X2 , . . . , Xn follow the distribution of X.

Statistics (estimators)

Our aim is to obtain information about the population parameters (mean, variance, proportion,. . . ) when only a sample is available. A statistic is any transformation (function) of the observations of a random sample. Consequently, it is a random variable, f (X1 , X2 , . . . , Xn ) . An estimator of a parameter is any statistic = f (X1 , X2 , . . . , Xn ) that provides us with approximate value of . The value that an estimator assumes in a real sample (when the random variables are substituted by real numbers) is called estimation.

1.1

Properties of estimators

Unbiased estimator. An estimator of a parameter is unbiased if is ex pected value is , that is, is unbiased if E[] = . The distance of the expectation of the estimator to the true value of the parameter E[] is called bias, bias[] = E[] .

Eciency. From several unbiased estimators of a parameter, we would choose the one with a smaller variance. The eciency of an estimator is the inverse to its variance, 1 . E[] = var[] We can compare two unbiased estimators by means of their relative eciency, which is given by RE[2 ; 1 ] = E[2 ] var[1 ] = . E[1 ] var[2 ]

The standard error of an estimator is its standard deviation, = var[] .

When the standard error depends of the true value of parameter , we can substitute by an estimation in order to obtain the estimated standard error . Mean Square Error. We can compare biased and unbiased estimators by means of the Mean Square Error given by MSE[] = E[( )2 ] = var[] + bias[]2 . Consistency. An estimator is consistent if the probability of it being arbitrarily close to the true value of the parameter converges to one as the sample size tends to innity. It is the minimal requirement for a good estimator.

2
2.1

Sampling distributions
Distribution of the sample mean

Given a random variable X with mean and known standard deviation , consider X1 , X2 , . . . , Xn a random sample from X of size n. That is,

X1 , X2 , . . . , Xn are n independent random variables with the distribution of X. The sample mean is given by X= 1 n
n

Xi
i=1

which is obviously a random variable. The sample mean is an unbiased estimator of the population mean , that is, E[X] = and its variance is var[X] = 2 /n . If X is normally distributed, then X also follows a normal distribution. Further, after the Central Limit Theorem (if n 30) the distribution of X is approximately N(, / n). Distribution of the sample proportion. The sample proportion is a particular case of sample mean. Let p denote the population proportion of individuals with a certain characteristic. We will take an individual at random from the population and consider the random variable X that assume value 1 if the individual has the characteristic and 0 otherwise. Obviously X B(1, p). Let us now take a random sample of X of size n, X1 , X2 , . . . , Xn , then 1 X= n
n

Xi = p
i=1

represents the quotient of the number of individuals from the sample with the characteristic divided by the sample size, that is, the sample proportion. Finally, if n 30, after the Central Limit Theorem, the distribution of p is approximately N(p, p(1 p)/n ).

2.2

Distribution of the sample variance

Consider a random sample X1 , X2 , . . . , Xn of a random variable X (dening the population distribution) with mean and variance . The sample variance given by n 1 2 (Xi X)2 S = n 1 i=1 is an unbiased estimator of the population variance 2 , that is E[S 2 ]. 3

2.3

Sampling distributions for normal populations

Distribution of the sample variance (normal population). Let X N(, ) and X1 , X2 , . . . , Xn be a random sample of X of size n, the distribution of the sample variance S 2 satises (n 1)S 2 2 n1 2 where 2 stands for the chi-square distribution with n 1 degrees of freen1 dom. Distribution of the sample mean with unknown variance (normal population). Let X N(, ) and X1 , X2 , . . . , Xn be a random sample of X of size n, if is unknown, we can estimate the variance through the sample variance, and the distribution of the sample mean X satises X S 2 /n tn1

where tn1 stands for the t distribution with n 1 degrees of freedom. Distribution of the quotient of sample variances (normal populations). Let X N(X , X ) and X1 , X2 , . . . , Xn be a random sample of X of size n, and let Y N(Y , Y ) and Y1 , Y2 , . . . , Yn be a random sample of Y of size m. The ratio of their sample variances satises
2 2 SX /X Fn1,m1 2 2 SY /Y

where Fn1,m1 stands for the F distribution with n 1 and m 1 degrees of freedom.

Maximum Likelihood Estimation

We start with a random sample X1 , X2 , . . . , Xn from a known distribution model depending on one (or several) parameters. Our aim is to approximate the true value (0 ) of parameter .

The Maximum Likelihood Estimator (MLE) is the value of that maximizes the likelihood function (joint density or probability function). That is, the value of that makes our sample more likely. Let us denote our observations by x = (x1 , x2 , . . . , xn ), that is, x is a vector in Rn . We obtain the MLE of the following way: 1. Likelihood function. If our distribution model is discrete,
n

l(|x) =
i=1

P (Xi = xi |) ,

meanwhile, if it is continuous,
n

l(|x) =
i=1

f (xi |) ,

where f (|) denotes the density mass function under the assumption that the value of the parameter is . That is, the likelihood function is the joint probability function for discrete models and the joint density function for continuous models, any of them is to be evaluated at the sample and considered as a function of the parameter. The target if to obtain the value of that maximizes l(|x). 2. Loglikelihood function. L(|x) = ln l(|x) 3. First derivative. Solve L(|x)/ and nd , candidate for MLE. We must still check that a maximum is attained at . 4. Second derivative. Check 2 L()/2 < 0 to conrm that is the MLE M L . of , call it Properties of MLEs. Given a distribution model with known range that does not depend on any parameter, the MLEs are: Asymptotically unbiased. E[M L ] n ; Asymptotically normal. M L N(, var[M L ]) ; Asymptotically of minimal variance. var[M L ] =
2 L(M L ) 2 1

Invariant under bijective transformations. If M L is the MLE of , then g(M L ) is the MLE of g() . 5

Appendix: Common sampling distributions

Pearsons Chi-square distribution, 2 . Given n independent standard n normal random variables X1 , X2 , . . . , Xn , the random variable
2 2 2 Y = X 1 + X2 + . . . + Xn

follows a Chi-square distribution with n degrees of freedom, denoted by Y 2 . n A Chi-square random variable only assumes positive values and its parameters are: E[Y ] = n ; var[Y ] = 2n. Students t distribution, tn . Given X and Y two independent random variables such that X follows a standard normal distribution and Y follows a Chi-square distribution with n degrees of freedom, the random variable Z= X Y /n

follows a t distribution with n degrees of freedom, denoted by Z tn . A random variable with distribution t can assume any real value. E[X] = 0 ; var[X] = n n2 if n 3.

For large enough values of n, the tn distribution is very similar to the standard normal. Fishers F distribution, Fn1 ,n2 . Given X and Y two independent random variables such that X follows a Chi-square distribution with n1 degrees of freedom and Y follows a Chi-square distribution with n2 degrees of freedom, the random variable X/n1 Z= Y /n2 follows a F distribution with n1 and n2 degrees of freedom, denoted by Z Fn1 ,n2 . A random variable with distribution F can only assume positive values.