Synthesis of Denoising Wavelet Neural Networks
Mohamed Zine EL ABIDINE SKHIRI, Mohamed CHTOUROU
Research Unit on Intelligent Control, Design & Optimization of Complex Systems (ICOS)
Ecole Nationale d’Ingénieurs de Sfax ( ENIS ), BP.W,3038, Sfax, Tunisia
Abstract: This paper investigates the use of a wavelet denoising unit based on the wavelet mutiresolution analysis on wavelet networks instead of neural networks on which previously reported works have been performed. This full wavelet compound will certainly provide the possibility of building up two denoising analysis models. The most straight forward model is setup by placing the wavelet denoising unit ahead of the network input layer. In other words, the inputs data embedded in a white Gaussian noise of a wavelet network are firstly denoised then fed to the network. An alternative model is also investigated in which the denoising unit will be placed at the output level of the net since a wavelet network is may be considered as a first step smoothing unit. In this analysis version, the noisy signal is fed to the wavelet network and the corresponding output is then applied to the denoising unit.
Keywords: wavelet neural networks, wavelet denoising unit, soft thresholding, hard thresholding.
Skhiri Mohamed Zine El Abidine: received the B.S and M.S. degrees in Electrical Engineering, from Syracuse University, Syracuse New York, USA. Currently, he is a teaching assistant in the Department of Electrical Engineering at the Institute of Technology in Sousse, Tunisia. He is registered as a PhD student on the National School of Engineers of Sfax, Tunisia, and his current research includes wavelet network systems.
Chtourou Mohamed: received the B.S, M.S., and Ph.D. degrees in electrical engineering, from National School of Engineers of Sfax-Tunisia, Institut National des Sciences Appliquées de Toulouse-France and Institut National Polytechnique de Toulouse-France, respectively. He is currently a professor in the Department of Electrical Engineering of National School of Engineers of Sfax-Tunisia. His current research interests include neural and fuzzy systems, intelligent and adaptive control. He is author and co-author of more than ten papers in international journals.
CITE THIS PAPER AS:
Mohamed Zine EL ABIDINE SKHIRI, Mohamed CHTOUROU, Synthesis of Denoising Wavelet Neural Networks, Studies in Informatics and Control, ISSN 1220-1766, vol. 17 (4), pp. 453-464, 2008.
Since their appearance wavelets have shown the ability to solve different kind of problems in different domains including numerical analysis, signal processing and mathematical modelling. Looking back to the 80’s, Morlet and Grossmann had the idea to use a dilated and a translated function (t) called a wavelet which is well localized in time and frequency . Meyer, Mallat and others had the possibility to seriously contribute to this early work until the paper of Ingrid Daubechies in 1988 that brought the mathematical concept to signal processing, statistics and numerical analysis. However, the most powerful and advanced feature of wavelets is the multiresolution analysis, by which a signal can be decomposed into different time-scale approximations and details. In other words, a signal is broken down into two different types of signals. In one hand, signals that carry the approximations of the original signal, and on the other, signals that carry the details. It is likely to be considered as the “Mathematical Microscope”. This characteristic will certainly help reducing any presented noise since the signal is totally decomposed to the finest detail. Indeed, wavelet thresholding techniques have proved to be the most powerful denoising tool since the work of Donoho and Jhonstone [5-8]. Therefore, it should be absolutely useful in every domain that deals with noisy data including neural and wavelet networks.
During the last decades neural networks ( NN ) have not given up showing their ability and success in solving system processing problems, approximating functions etc. On the other hand, the appearance of the wavelet theory has made it even more successful and powerful by creating analogous networks called the wavelet neural networks in which wavelets are used as the activation functions of the hidden neurons in the conventional neural network. Wavelet networks are traced back to the work of Daugman  in which Gabor wavelets were used for image compression. They are introduced as a special feedforward neural network, and they became more popular after the work of Pati , Zhang , and Szu. They have been applied to many different areas and applications such as nonlinear functional approximation and nonparametric estimation , system identification and control tasks, modelling and classification. The generated wavelets used in networks are the dilated and the translated versions of a mother wavelet which could be in a continuous or a discrete form.
In this paper, the most wavelet advantages are exploited including the wavelet networks as well as the wavelet multiresolution analysis for denoising. Basically, two possible denoising process models are introduced. In the first model, the inputs data of any static or dynamic wavelet network are firstly denoised based on the wavelet multiresolution analysis. Then, they are fed to a wavelet network having as activation functions some dilated and translated versions of a certain mother wavelet. In the second model, however, the denoising unit is added at the network output level. In other words, the network takes the noisy signal as the input and the corresponding output is then denoised. As far as the adjusted wavelet network parameters are concerned, it should be noticed that their initialization process is somehow delicate, and their number is important. Unlike neural networks in which the weights could be initialized randomly, wavelet networks require a greater care in choosing the initial values of the parameters, especially, the dilations and the translations.
This paper is organized as follows: In section II, the related works are presented followed by the proposed denoising process models in sectionIII. Afterwards, the multiresolution analysis and the wavelet networks are introduced in section IV and V respectively. The simulation results, however, are discussed in section VI. Finally, the last section will conclude the paper.
Noise reduction has been and certainly will be the most treated issue on signals based applications including wavelet networks data. However, the most powerful tool to carry out this task turns out to be the denoising based on wavelets. Therefore, it would very convenient to process the noisy inputs or outputs data of any static or dynamic wavelet networks by a wavelet based denoising unit. The position of this type of unit with respect to the wavelet network provides the possibility to generate two different denoising models. One model includes the denoising unit at the network input level, and the second one at the output level. It is clear though that the model with the denoising unit added at the input level has generated more satisfactory results than the second model. However, when the denoising unit is placed at the output level, the whole denoising process is may be considered to be carried out twice. The first is totally related to the wavelet network which undergoes throughout the training process some kind of signal smoothing at the output. In fact, the performance of this process is quite interesting since it preserves at least the main shape of the free noise data. In other words, the wavelet network could be considered as a first step denoising unit. The second process, however, is carried out on the generated network output using the based wavelet denoising unit. Placed whether at the input or at the output of the network, the performance of the denoising unit depends on the type of the used wavelet and the number of the decomposition levels. As far as the wavelet networks are concerned different networks structures could be processed with a special care that should be taken towards the network parameters initialization phase. Unlike neural networks, wavelet networks require a great care in choosing the adjusted parameters especially the dilations and the translations.
- BURKE, B., The mathematical Microscope: waves, wavelets and beyond, A Positron Named Priscilla, Scientific Discovery at the Frontier, chapter 7, 1994, pp 196-235.
- DAUBECHIE, I, The wavelet transform, time-frequency localization and signal analysis, IEEE Trans. Informat. Theory Vol. 36, 1990.
- DAUGMAN, J. G., Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression, IEEE Trans.Acoust Speech, Signal Process vol. 36, no. 7, 1988, pp. 1169-1179.
- DONOHO, D.L., Nonlinear wavelet methods for recovering signals, images, and densities from indirect and noisy data, Proceedings of symposia in applied Mathematics, vol. 47, 1993, pp. 173-205.
- DONOHO, D.L., Denoising by soft-thresholding, IEEE Trans. Inform. Theory, vol.41, no.3, 1995, pp. 613-627.
- DONOHO, D.L. and I.M. JOHNSTONE, Ideal spatial adaptation via wavelet shrinkage, Biometrika, vol. 81, no. 3, pp. 425-455.
- DONOHO, D.L. and I.M. JOHNSTONE, Adapting to unknown smoothness via wavelet shrinkage, J. Amer: statist. Assoc, vol. 90, no. 432, 1995, pp. 1200-1224.
- DONOHO, D.L., I.M. JOHNSTONE, KERKYACHARIAN PICARD D., Wavelet shrinkage: Asymptotia?, J. Royal statist. Soc., vol. 51, no. 2, 1995, pp. 301-337.
- JHONSTONE, M., B.W. SILVERMAN, Wavelet threshold estimators for data with correlated noises, Journal of the Royal Statistical Society, Vol. 59, 1997, pp. 319 351.
- LOTRIC, U., DOBNIKAR A., Wavelet based smoothing in time series prediction with neural networks. In: V. Kurkova, N. C. Steele, R. Neruda, M. Karny (eds.): Artificial Neural Nets and Genetic Algorithms: Proceedings of the ICANNGA Conference in Prague, Czech Republic, Springer, 2001, pp. 43-46.
- LOTRIC, U., Wavelet Based Denoising Integrated into Multilayered Perceptron, Neurocomputing, vol. 62, 2004, pp. 179-196.
- LOTRIC, U., A. DOBNIKAR, Neural Networks with Wavelet Based Denoising Layer for Time Series Prediction, Neural Computing and Applications, vol. 14, no. 1, 2005, pp.11-17.
- MALLAT, S., A theory of multiresolution signal decomposition: the wavelet representation, IEEE Trans on pattern recognition and Machine Intelligence, vol. 11, 1969, pp. 674-693.
- MEYER, Y., Wavelets: algorithms and applications, Philadelphia, SIAM, 1993.
- PATI, Y.C. and P.S. KRISHNAPARASAD, Analysis and synthesis of feedforward neural networks using discrete affine Wavelet transforms, IEEE Trans. Neural Networks, vol. 4, no. 1, pp. 73-85.
- SZU, H., B. TELFER, and S. KADAMBE, Neural network adaptive wavelets for signal representation and classification, Optical Engineering, 31:1907-1961, 1992.
- ZHANG, Q.H. and A. BENVENISTE, Wavelet networks, IEEE trans. Neural Networks, vol. 3, no. 6, 1992, pp. 889-898.
- ZHANG, Q.H., Wavelet networks: the radial structure and an efficient initialization procedure, Technical Report of Linköping University, LiTH-ISY-I-1423. 1992.
- ZHANG, Q., Regressor Selection and Wavelet Networks Construction, Technical Report of Linköping University, no. 1967. 1993.
- ZHANG, Q., Using Wavelet Network in Nonparametric Estimation, IEEE Trans On Neural Networks, vol. 8, no. 2, 1997.
- ZHANG, S. and E. SALARI, Image denoising using a neural network based non-linear filter in wavelet domain, Acoustics, Speech, and Signal Processing, 2005. Proceedings. (ICASSP ) IEEE International Conference, Vol. 2, 2005, pp. 989 – 992.
- ZHANG, XIAO-PING, Thresholding Neural Network for adaptive noise reduction, IEEE Trans. Neural Networks, vol. 12, no. 3, 2001.