Vector quantization I.Introduction Vector quantization (VQ) is a lossy data compression method based on the principle of block coding.

It is a fixed-to-fixed length algorithm. A VQ is nothing more than an approximator. The idea is similar to that of ``rounding-off'' (say to the nearest integer). An example of a 1-dimensional VQ is shown below.

Fig. A 1- dimensional Vector quantization Here, every number less than -2 are approximated by -3. Every number between -2 and 0 are approximated by -1. Every number between 0 and 2 are approximated by +1. Every number greater than 2 are approximated by +3. Note that the approximate values are uniquely represented by 2 bits. This is a 1-dimensional, 2-bit VQ. It has a rate of 2 bits/dimension. An example of a 2-dimensional VQ is shown below:

Fig 2-dimensional Vector quantization Here, every pair of numbers falling in a particular region are approximated by a red star associated with that region. Note that there are 16 regions and 16 red stars -- each of which can be uniquely represented by 4 bits. Thus, this is a 2-dimensional, 4-bit VQ. Its rate is also 2 bits/dimension. In the above two examples, the red stars are called codevectors and the regions defined by the blue borders are called encoding regions. The set of all codevectors is called the codebook and the set of all encoding regions is called the partition of the space. Vector quantization From Wikipedia, the free encyclopedia Vector quantization is a classical quantization technique from signal processing which allows the modeling of probability density functions by the distribution of prototype vectors. It was originally used for data compression. It works by dividing a large set of points (vectors) into groups having approximately the same number of points closest to them. Each group is represented by its centroid point, as in k-means and some other clustering algorithms. The density matching property of vector quantization is powerful, especially for identifying the density of large and high-dimensioned data. Since data points are represented by the index of their closest centroid, commonly occurring data have low error, and rare data high error. This is why VQ is suitable for lossy data compression. It can also be used for lossy data correction and density estimation. Vector quantization is based on the competitive learning paradigm, so it is closely related to the self-organizing map model. Training A simple training algorithm for vector quantization is: 1. 2. 3. Pick a sample point at random Move the nearest quantization vector centroid towards this sample point, by a small fraction of the distance Repeat

A more sophisticated algorithm reduces the bias in the density matching estimation, and ensures that all points are used, by including an extra sensitivity parameter: 1. 2. 3. Increase each centroids sensitivity by a small amount Pick a sample point at random Find the quantization vector centroids with the smallest <distance-sensitivity>

1. 4. 2. Repeat

Move the chosen centroids toward the sample point by a small fraction of the distance Set the chosen centroids sensitivity to zero

It is desirable to use a cooling schedule to produce convergence: see Simulated annealing. The algorithm can be iteratively updated with 'live' data, rather than by picking random points from a data set, but this will introduce some bias if the data is temporally correlated over many samples. Applications Vector quantization is used for lossy data compression, lossy data correction and density estimation. Lossy data correction, or prediction, is used to recover data missing from some dimensions. It is done by finding the nearest group with the data dimensions available, then predicting the result based on the values for the missing dimensions, assuming that they will have the same value as the group's centroid. For density estimation, the area/volume that is closer to a particular centroid than to any other is inversely proportional to the density (due to the density matching property of the algorithm). Use in data compression Vector quantization, also called "block quantization" or "pattern matching quantization" is often used in lossy data compression. It works by encoding values from a multidimensional vector space into a finite set of values from a discrete subspace of lower dimension. A lower-space vector requires less storage space, so the data is compressed. Thanks to the density matching property of vector quantization, the compressed data have errors that are inversely proportional to their density. The transformation is usually done by projection or by using a codebook. In some cases, a codebook can be also used to entropy code the discrete value in the same step, by generating a prefix coded variable-length encoded value as its output. The set of discrete amplitude levels is quantized jointly rather than each sample being quantized separately. Consider a K-dimensional vector [x1,x2,...,xk] of amplitude levels. It is compressed by choosing the nearest matching vector from a set of N-dimensional vectors [y1,y2,...,yn]. All possible combinations of the N-dimensional vector [y1,y2,...,yn] form the codebook. Block Diagram: A simple vector quantizer is shown below Only the index of the codeword in the codebook is sent instead of the quantized values. This conserves space and achieves more compression. Twin vector quantization (VQF) is part of the MPEG-4 standard dealing with time domain weighted interleaved vector quantization.

References ^ "Vorbis I Specification". Xiph.org (2007-03-09). Retrieved on 2007-03-09.

III. Design Problem The VQ design problem can be stated as follows. Given a vector source with its statistic properties known, given a distortion measure, and given the number of codevectors, find a codebook (the set of all red stars) and a partition (the set of blue lines) which result in the smallest average distortion. We assume that there is a training sequence consisting of source vectors:

This training sequence can be obtained from some large database. For example, if the source is a speech signal, then the training sequence can be obtained by recording several long telephone conversations. is assumed to be sufficiently large so that all the statistical properties of the source are captured by the training sequence. We assume that the source vectors are Let -dimensional, e.g.,

be the number of codevectors and let -dimensional, e.g., and let ,

represents the codebook. Each codevector is Let

be the encoding region associated with codevector

denote the partition of the space. If the source vector then its approximation (denoted by ) is :

is in the encoding region

Assuming a squared-error distortion measure, the average distortion is given by:

where follows: Given and , find

. The design problem can be succinctly stated as and such that is minimized.

IV. Optimality Criteria

If and are a solution to the above minimization problem, then it must satisfied the following two criteria.

Nearest Neighbor Condition:

This condition says that the encoding region should consists of all vectors that are closer to than any of the other codevectors. For those vectors lying on the boundary (blue lines), any tie-breaking procedure will do.

Centroid Condition:

This condition says that the codevector

should be average of all those training

vectors that are in encoding region . In implementation, one should ensure that at least one training vector belongs to each encoding region (so that the denominator in the above equation is never 0).

V.LBG Design Algorithm

The LBG VQ design algorithm is an iterative algorithm which alternatively solves the above two optimality criteria. The algorithm requires an initial codebook . This initial codebook is obtained by the splitting method. In this method, an initial codevector is set as the average of the entire training sequence. This codevector is then split into two. The iterative algorithm is run with these two vectors as the initial codebook. The final two codevectors are splitted into four and the process is repeated until the desired number of codevectors is obtained. The algorithm is summarized below. LBG Design Algorithm 1. Given 2. Let . Fixed and to be a ``small'' number.

Calculate

3. Splitting: For

, set

Set

. . Set the iteration index , find the minimum value of .

4. Iteration: Let i. For

over all minimum. Set

. Let

be the index which achieves the

ii.

For

, update the codevector

iii. iv.

Set Calculate

v. vi.

If Set . For

, go back to Step (i). , set

as the final codevectors. 5. Repeat Steps 3 and 4 until the desired number of codevectors is obtained.

VI. Performance
The performance of VQ are typically given in terms of the signal-to-distortion ratio (SDR):

(in dB), where is the variance of the source and is the average squared-error distortion. The higher the SDR the better the performance. The following tables show the performance of the LBG-VQ for the memoryless Gaussian source and the first-order Gauss-Markov source with correlation coefficient 0.9. Comparisons are made with the optimal performance theoretically attainable, SDRopt, which is obtained by evaluating the rate-distortion function.

Rate (bits/dimension) 1 2 3 4 5 4.4 9.3 14.6 20.2 26.0 4.4 9.6 15.3 21.1 27.0 4.5 9.9 15.7 -------

SDR (in dB) 4.7 10.2 ---------4.8 10.3 ---------4.8 ------------4.9 ------------5.0 -------------

SDRopt 6.0 12.0 18.1 24.1 30.1

Memoryless Gaussian Source

Rate (bits/dimension) 1 2 3 4 5 4.4 9.3 14.6 20.2 26.0 7.8 13.6 19.0 24.8 30.7 9.4 15.0 20.6 -------

SDR (in dB) 10.2 15.8 ---------10.7 16.2 ---------11.0 ------------11.4 ------------11.6 -------------

SDRopt 13.2 19.3 25.3 31.3 37.3

First-Order Gauss-Markov Source with Correlation 0.9

VIII. References
1. 2. 3. 4. A. Gersho and R. M. Gray, Vector Quantization and Signal Compression. H. Abut, Vector Quantization. R. M. Gray, ``Vector Quantization,'' IEEE ASSP Magazine, pp. 4--29, April 1984. Y. Linde, A. Buzo, and R. M. Gray, ``An Algorithm for Vector Quantizer Design,'' IEEE Transactions on Communications, pp. 702--710, January 1980.

Vector quantization

Given a vector (x1, x2, x3, , xn), find the vector (y1, y2, y3, , yn) from the code-book which minimizes the MSE (mean square error): 1 n ( xi yi ) 2 n i =1 The Schematic of a Vector Quantizer

Vector Quantization encoding VQ was first proposed by Gray in 1984. First, construct codebook which is composed of codevector. For one vector being encoding, find the nearest vector in codebook. (determined by Euclidean distance) Replace the vector by the index in codebook. When decoding, find the vector corresponding by the index in codebook. Two important issue Code vectors in codebook need representative. It affect the quality of decompressed image a lot. So, how to build a good codebook is an important issue. Euclidean distance is time-consuming. How to fasten to search for the nearest vector is also an important issue.

LBG Algorithm
1. Determine the number of codewords, N, or the size of the codebook. 2. Select N codewords at random, and let that be the initial codebook . The initial codewords can be randomly chosen from the set of input vectors. 3. Using the Euclidean distance measure clusterize the vectors around each codeword. This is done by taking each input vector and finding the Euclidean distance between it and each codeword. The input vector belongs to the cluster of the codeword that yields the minimum distance. 4. Compute the new set of codewords . This is done by obtaining the average of each cluster. Add the component of each vector and divide by the number of vectors in the cluster.

where i is the component of each vector (x, y, z, ... directions), m is the number of vectors in the cluster. 4. Repeat steps 2 and 3 until the either the codewords don't change or the change in the codewords is small.

Codebook Training: K-means Algorithm

1. Determine the number of codewords, N, or the size of the codebook 2. Select N codewords at random, and let that be the initial codebook The initial codewords can be randomly chosen from the set of input vectors. 3. Using the Euclidean distance measure to cluster the vectors around each codeword This is done by taking each input vector and finding the Euclidean distance between it and each codeword The input vector belongs to the cluster of the codeword that yields the minimum distance

4. Compute the new set of codewords This is done by obtaining the average of each cluster Add the component of each vector and divide by the number of vectors in the cluster 5. Repeat steps 2 and 3 until either the codewords don't change or the change in the codewords is small

Codebook Training: Binary Splitting Algorithm 1. Design 1-vector codebook (no iteration) 2. Double codebook size by splitting each codeword y n according to the rule:
y+ n = y n (1 + ) y n = y n (1 )

where 1nN, and is a splitting parameter (0.01 0.05) 1. Use K-means algorithm to get best set of centroids 2. Repeat (2)-(3) until desired codebook size is obtained Cell split method
algorithm 1. Divide image into blocks. Choose a block (k-dimension) X=(x1, x1,,x1) as initial vector. 2. Spit X vector into two vector Y=(y1, y1,,y1) and Z=(z1, z1,,z1) yi=xi- ,zi=xi+ 3. Y and Z are centroids. For all blocks, find the nearest centroid. Re-compute the centroid of blocks and get new centroid Y and Z. 4. Recursively, do Y and Z. Repeat 2 ,3 step. Until find enough number of codevector.

Advantages: The bit rate of the VQ scheme is low (log2Nc /k), where N and k denote the codebook size and vector dimension, respectively. VQ has a simple decoding structure. Disadvantages: The reconstructed image quality highly depends on the codebook performance. The VQ scheme requires a lot of computation cost in codebook design process and VQ encoding process.