Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

Lossless Data Compression Using Neural Networks

FLORIN ALEXA1, VASILE GUI1, CATALIN CALEANU2, CORINA BOTOCA1 1 Communication Department, 2Applied Electronics Department Politehnica University of Timioara Bd. V. Prvan no.2 Timioara ROMANIA

Abstract: - This paper deals with the predictive compression of images using neural networks (NN). The idea is to use of the backpropagation algorithm in order to compute the predicted pixels. The results validation is performed by comparison with linear prediction compression used in JPEG algorithm. Key-Words: - lossless image compression, neural networks, prediction, backpropagation algorithm

1 Introduction
In the recent years the need for image compression has steadily grown. More complex applications have increased the storage capacity requirements to a point where the cost of providing required resources becomes prohibitive. For example a typical digital satellite image requires more than 48 Mbits of storage space. Therefore, the compression of a data file prior to storage (or reducing the bit rate) and saving data in a smaller memory space will economize memory and transmission capacity. Data compression techniques can be classified in two categories, as shown in Fig. 1. One category, known as lossless compression technique (or reversible coding), enables an exact replica of original data to be reproduced after reconstruction. The other group, known as lossy compression techniques (or nonreversible coding), allows slight differences between the original and the reconstructed data. This type of technique allows a greater degree of compression. These techniques are useful in applications such as broadcast television, videoconference and facsimile transmission. In fact, many lossy encoding techniques are capable of reproducing recognizable monochrome images from data that have been compressed by a factor more than 30:1 and images that are virtually indistinguishable from the originals at a compression factor from 10:1 to 20:1. This type of techniques is however suitable for untypical applications in some scientific and medical images [1], [2]. A traditional approach to reduce the large amount of data would be to discard some images redundancy and introduce some noise after reconstruction. In this case, advanced technologies

to reduce the size of the storage device and improve the efficiency of transmission are needed. There are a number of useful methods to implement advanced data compression (see Fig.1). For example variable length coding is the simplest approach to lossless image compression which reduces only coding redundancy. Coding redundancy normally is present in any natural binary encoding of the gray levels in an image. The most popular technique for removing coding redundancy is Huffman coding. When individually coding the symbols of an information source, Huffman coding yields the smallest possible number of code symbols per source symbol [3].

2 Lossless Compression
A measure of average information per source output, denoted H(A), is defined as

H ( A) = p j log p j
j =1

[bit / pixel ]

(1) and is called the entropy of the source (or zero-th order entropy), where p j is the probability of symbol j. It defines the average amount of information obtained by observing a single source output. The source entropy measure gives us the average amount of information per symbol. The information from a source can be represented in fewer bits per symbol; in that case the maximum number of bits/symbol is n and it cannot be less than H. If that is true, it is said that the source contains statistical redundancy [3], [4]. Lossless compression is a class of compression algorithms that allows the exact original data to be

ISSN: 1790-5117

128

ISBN: 978-960-474-035-2

Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

reconstructed from the compressed data. Lossless compression is used when it is important that the original and the decompressed data are identical, or

when no assumption can be made on whether certain deviation is uncritical.

Data compression

Reversible (lossless) coding

Nonrevesible (lossy) coding

String coding

Statistical coding

Wave form coding

Analysis synthesis

RunLength coding

LZW coding

Huffman coding

Comma coding

Time domain coding

Frequency domain coding

Vocoders

APC

DPCM

ADM

Transform coding

Sub-band coding

LPC

ADPCM

DCT

SubNyquist sampling

RPELPC

Fig. 1. Data compression techniques In image processing lossless compression is based on the idea that since not all the gray levels in an image occur equally often, by assigning shorter binary code words to the more frequently occurring gray levels, one can achieve compression over the case when all the levels are represented by equal length code words. Lossless compression does not generally achieve as low bit rates as lossy compression due to the strict restriction of zero information loss. In order to avoid the possibility of destroying information that may be useful to future users, the data must be stored in a format that allows exact reconstruction of the original data images. The lossless predictive coding is based on eliminating the pixel redundancies of closely spaced pixels by extracting and coding only the new information in each pixel. The new information of a pixel is defined as the difference between the actual and predicted value of that pixel. Fig. 2 shows the basic components of a lossless predictive coding system [5].
Input image

fn

en

Symbol encoder

fn
Predictor Nearest integer

Symbol decoder Compressed image

en fn

' fn

Reconstructed image

e
Predictor

Fig. 2. Lossless predictive coding model: encoder and decoder

ISSN: 1790-5117

129

ISBN: 978-960-474-035-2

Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

The system consists of an encoder and a decoder, each containing an identical predictor. As each successive pixel of the input image, denoted f n is introduced to the encoder, the predictor generates the anticipated value of that pixel, based on a number of past inputs. The output of the predictor is then rounded to the nearest integer, denoted f n and used to form the prediction error: en = f n f n (2) which is coded using a variable-length code by the symbol encoder to generate the next element of the compressed data stream. The decoder reconstructs en from the received variable-length code words and performs the inverse operation: f =e + f (3)
n n n

The three neighboring samples must be already predicted samples. Anyone of the predictors shown in the Table 1 can be used to estimate the sample located at X. Predictors 1 to 7 are JPEGs recommended and predictor 8 and 9 are two commonly used [7], [8]. Table 1 1 2 3 4 5 6 7 8 9 A C B A+C-B A+(C-B)/2 C+(A-B)/2 (A+C)/2 0,75A+0,75C-0,75B 0,66A+0,66C-0,33B

Various local, global and adaptive methods can be used to generate f n . In this paper neural networks are used to predict the next pixels of images. The image obtained through prediction has a significantly less entropy less than the entropy of the original image. Because a lot of interpixels redundancy is removed by prediction and difference process, the probability density function of the prediction error is highly peaked at zero and characterized by a relatively small variance. In general, an estimate of the maximum compression of any lossless predictive coding approach may be obtained by dividing the average number of bits used to represent each pixel in the original image by a first order estimate of the entropy of the prediction data error [6], [7]. A lossless image compression algorithm can be based on a linear prediction and arithmetic coding. In the compression process, the predictor combines up to three neighboring samples at A, B, and C shown in Fig.3 in order to produce a prediction of the sample value at the position labeled by X.

3 Neural Networks Predictor Implementation

The most important advantages of using NN in prediction applications are [9]: their ability of learning from examples only their capacity of mapping hidden and nonlinear input-output dependencies even when the training set is corrupted with noise. The proposed predictor model is a feed-forward neural network with 4 inputs, 1 hidden layer with 3 neurons and 1 output, as shown in Fig.4.
Bh A B C H1 H2 H3 F Bo

D B C X A
Scan Directions

Previous Line Current line

Fig. 4. Predictor model Each input is connected to every neuron of the hidden layer; likewise, each neuron of the hidden layer is connected to every neuron of the output layer [10]. A, B, C, D are the inputs of the predictor. Bh and Bo are the bias of hidden and output layer,

Fig. 3. Neighboring samples around the sample to be predicted

ISSN: 1790-5117

130

ISBN: 978-960-474-035-2

Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

respectively. The output of neuron F is equal to the current pixel prediction X and has a value between 0 and 1. That is because the input value of the gray level of the image was normalized by division to 256. The backpropagation algorithm with momentum was used to train the weights of the NN [9], [10]. The learning rate was chosen to 0.25 and the momentum 0.1 All of test images have a spatial resolution of 256 256 pixels, represented by 8 bits binary words, corresponding to one of 256 gray levels, with 0 corresponding to black and 255 corresponding to white. This requires 64 kbytes for each image. We have used different types of images in order to make a comparison between the linear prediction and the neural network prediction. The images used are presented in Fig. 5.

Table 2
Image name Neural Linear Neural Zero-order network prediction network entropy entropy Entropy entropy (original) (A) (4 inputs) (10 inputs)

CTH CY SAT

7,61 7,80 7,61

5,74 6,02 6,09

6,18 6,48 6.75

5,76 6,35 6.52

We have tested different NNs structures and observed that, the 4-3-1 NN structure performed better even than the 10-3-1 NN. The best results were obtained in the case of using two hidden layers (network with structure 10:7:3:1) but they are not significantly better compared with results obtained in the case of using the initial structure (4:3:1 from Fig. 4). Table 3
Configur ation 4,3,1 4,5,1 4,3,2,1 10,3,1 10,7,3,1 Learning rate Momen tum Entropy Compress ion 81,20 % 6,18 82,43 % 6,27 82,20 % 6,25 81,72 % 6,22 81,03 % 6,16

0.4 0.2 0.4 0.4 0.2

0.1 0.1 0.7 0.7 0.1

Comparing the results in terms of the mean square error, obviously the obtained value of mean square error in the case of the neural network prediction is smaller than in linear prediction.

4 Conclusion
The entropy of prediction error images produced by neural networks is slightly higher than that produced by linear prediction. So, in terms of the entropy value, linear prediction is better than the neural network one. The neural networks minimize the mean squared error, thus the neural networks prediction, leads to better results. We expect that using an advanced lossless compression technique the performance of neural network predictor would be improved [11].

Fig. 5. Test image a) cathedral (CTH), b) cyclone (CY), c) satellite image (SAT) We have used different numbers of iterations to train NN predictor. In our experiments, after 80 iterations the weights were constant and the results were similar. The results, for each case, are presented in the Table 2 and Table 3. Comparing the results of the neural network with the linear prediction, one can observe that the image entropy is worst than in the case of the linear prediction.

References: [1] S. Cho, D. Kim, and W.A. Pearlman, Lossless compression of volumetric medical images with improvement 3-D SPIHT algorithm, Journal of Digital Imaging, vol.17, no.1, 2004, pp.57-63 [2] S. Winkler, A. Sharma, and D. McNally, Perceptual video quality and blockiness metrics

ISSN: 1790-5117

131

ISBN: 978-960-474-035-2

Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

for multimedia streaming applications, in Proc. International Symposium on Wireless Personal Multimedia Communications, (Aalborg, Denmark), Sept. 912, 2001, pp. 547552,. [3] Z. Bojkovic, C. Toma , V. Gui , R. Vasiu, Advanced Topics in Digital Image Compression, Politehnica Publishing House, Timioara 1997 [4] R.J. Clark, Digital Compression of Still Images and Video, Academic Press, 1995 [5] C. K. Wallace, The JPEG still picture compression standard, Communications of the ACM, 34(4), 1991, pp.31-44, [6] H.G..Mussman, Predictive Image Coding, in Image Transmission Techniques, W.K.Pratt editor, Advances in Electronics and Electron Physics, Supplement 12, 1979, Academic Press, Orlando, FL, pp.73-112 [7] D. Memon Nasir, Wu Xiaolin, V. Sippy and G. Miller, Interband coding extension of the new

lossless JPEG standard, Proc. SPIE Int. Soc. Opt. Eng., vol. 3024, no. 47, January 1997, pp.47-58 [8] M. Weinberger, G. Seroussi and G. Sapiro, The LOCO-I lossless image compression algorithm: Principles and standardization into JPEG-LS, IEEE Trans. Image Processing, vol. 9, no. 8, Aug. 2000, pp. 13091324 [9] S. Haykin , Neural Networks, a comprehensive foundation, Macmillan College Publishing Company, 1994 [10] C. Cleanu, Facial recognition using committee of neural networks, Proceedings of the 5th Seminar on Neural Network Applications in Electrical Engineering NEUREL 2000, Belgrad 25-27 sept. 2000, pp. 97-100 [11] G. Langdon and C. Haidinyak, Context dependent distribution shaping and parameterization for lossless image compression, Application of Digital Image Processing XVII, SPIE 1994, pp.62-70

ISSN: 1790-5117

132

ISBN: 978-960-474-035-2