Decorrelating Data for Mixture of Gaussians

As we discussed in our meeting, it is really easy to implement FL in

mixture of Gaussians if the data is deccorelated , i.e the covariance
matrix is diagonal. Here I describe how can we decorrelate data , so that
we can obtain a covariance matrix that will have only diagonal elements.
Again assume that we have two tables R and S. R has two features or
columns X1 and X2. Y has three columns/features Y1,Y2 and Y3.
In order to decorrelate, we need the Eigen decomposition of the original
covariance matrix . We assume that is positive definite, which is
valid in our case ,since we generate the data points from a multivariate
normal distribution.
1. We first centre the data so that mean of each random variable(each
feature , in our case 5 features 2 from R table and 3 from S table) is
zero. By centering, we mean that subtracting mean of all the values
of that feature from each feature pointso that, = E[Xi] = 0; where
Xi =X1,X2,Y1,Y2,Y3.
2. The covariance matrix can be expressed as follows:
1

1
2

1
2

= = 1 , where is a diagonal matrix with the

eigenvalues, i , of on the diagonal and is the matrix of
corresponding eigenvectors, ei . In addition, the columns of are
orthonormal so that: 1 = .
3. Now, let = . Then Y is a random vector with a decorrelated
multivariates Gaussian distribution with variances i on the
diagonal of its covariance matrix.
An example of how the transform works is presented below. For
the purpose of visualization in two dimensional space, I am

assuming there are two features now . The scatter plot of the datapoints are show in the figure 1-a. Clearly the variables are
correlated and we dont have a diagonal matrix. Now I apply the
above explained transform and then plotted Y , i.e. the new data
points after the transform. As can be seen from figure 1-b that the
data is now decorrelated. Clearly this still preserves the nature or
shape of the data(i.e Gaussian ).

Figure 1-a

Figure 1-b

In order to make things clearer with an example , consider the following

simple example with two features.
1. For the purpose of illustration, I will use a bivariate Gaussian
10 6
vector X N(, ) where =
.
6 5
2. Let S be the collection of data points obtained by taking n samples
from X. The rows of S are the data points . Thus there are 2
columns here one for each dimension(feature).
3. The contour is an ellipse that is centered at the origin and rotated
through an angle. Say that we do not know the parameters of the
distribution from which S was obtained so we estimate them.:
1
0.0753
=
=
=1

0.0568

1
9.9559 5.8718
=
( )( ) =
5.8718 4.8099
1
=1

4. Next, we find the eigenvectors of the estimated covariance matrix

by finding the orthonormal basis for the nullspace of matrix
. =
1.0078

5. The covariance matrix of Y is now:

, where a

13.3845
= 8.2991016 ~0.
6. Thus clearly the above covariance is diagonal and hence our data is
decorrelated.
Now as we have decorrelated data I can easily apply my mixture of
Gaussians equation