PDF

Abstract:
In this paper we study image recognition tasks in which the images are described by high dimensional feature vectors extracted with deep convolutional neural networks and principal component analysis. In particular, we focus on the problem of high computational complexity of a statistical approach with non-parametric estimates of probability density implemented by the probabilistic neural network. We propose a novel statistical classification method based on the density estimators with orthogonal expansions using trigonometric series. It is shown that this approach makes it possible to overcome the drawbacks of the probabilistic neural network caused by the memory-based approach of instance-based learning. Our experimental study with Caltech-101 and CASIA WebFace datasets demonstrates that the proposed approach reduces the error rate by 1–5 % and increases the computational speed by 1.5 – 6 times when compared to the original probabilistic neural network for small samples of reference images.

Keywords:
statistical pattern recognition, image processing, deep convolutional neural networks, probabilistic neural network, orthogonal series estimates, unconstrained face identification.

Citation:
Savchenko AV. Trigonometric series in orthogonal expansions for density estimates of deep image features. Computer Optics 2018; 42(1): 149-158. DOI: 10.18287/2412-6179-2018-42-1-149-158.

References:

1. Prince SJD. Computer vision: Models, learning, and inference. Cambridge: Cambridge University Press; 2012. ISBN: 978-1-107-01179-1.
2. Goodfellow I, Bengio Y, Courville A. Deep learning. Cambridge, London: The MIT Press; 2016. ISBN: 9780262035613.
3. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. Proc CVPR 2015: 1-9. DOI: 10.1109/CVPR.2015.7298594.
4. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
5. Savchenko AV. Probabilistic neural network with homogeneity testing in recognition of discrete patterns set. Neural Networks 2013; 46: 227-241. DOI: 10.1016/j.neunet.2013.06.003.
6. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Proc NIPS'12 2012; 1: 1097-1105.
7. Rassadin A, Gruzdev A, Savchenko A. Group-level emotion recognition using transfer learning from face identification. Proc ICMI 2017: 544-548. DOI: 10.1145/3136755.3143007.
8. Sharif Razavian A, Azizpour H, Sullivan J, Carlsson S. CNN features off-the-shelf: an astounding baseline for recognition. Proc CVPRW'14 2014: 806-813. DOI: 10.1109/CVPRW.2014.131.
9. Savchenko AV. Maximum-likelihood approximate nearest neighbor method in real-time image recognition. Pattern Recognition, 2017; 61: 459-469.  DOI: 10.1016/j.patcog.2016.08.015.
10. Savchenko AV. Deep neural networks and maximum likelihood search for approximate nearest neighbor in video-based image recognition. Opt Mem Neural Networks 2017; 26(2): 129-136. DOI: 10.3103/S1060992X17020102.
11. Raudys SJ, Jain AK. Small sample size effects in statistical pattern recognition: Recommendations for practitioners. IEEE Transactions on Pattern Analysis and Machine Intelligence 1991, 13(3): 252-264. DOI: 10.1109/34.75512.
12. Savchenko AV. Maximum-likelihood dissimilarities in image recognition with deep neural networks [In Russian]. Computer Optics 2017, 41(3): 422-430. DOI: 10.18287/2412-6179-2017-41-3-422-430.
13. Pan SJ, Yang Q. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 2010, 22(10): 1345-1359. DOI: 10.1109/TKDE.2009.191.
14. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Li FF. ImageNet large scale visual recognition challenge. International Journal of Computer Vision 2015; 115(3): 211-252. DOI: 10.1007/s11263-015-0816-y.
15. Theodoridis S, Koutroumbas C, (eds). Pattern recognition. 4th ed. Burlington, San Diego, London: Elsevier Inc; 2009. ISBN: 978-1-59749-272-0.
16. Webb AR. Statistical pattern recognition. 2nd ed. Chichester, England: John Wiley & Sons, Ltd; 2002. ISBN: 978-0-470-84513-4.
17. Savchenko AV. Image recognition on the basis of probabilistic neural network with homogeneity testing [in Russian]. Computer optics 2013; 37(2): 254-262.
18. Specht DF. Probabilistic neural networks. Neural Networks 1990; 3(1): 109-118. DOI: 10.1016/0893-6080(90)90049-Q.
19. Kusy M, Kluska J. Probabilistic neural network structure reduction for medical data classification. Proc ICAISC 2013: 118-129. DOI: 10.1007/978-3-642-38658-9_11.
20. Savchenko AV. Pattern classification with the probabilistic neural networks based on orthogonal series kernel. Proc ISNN 2016: 505-512. DOI: 10.1007/978-3-319-40663-3_58.
21. Cencov NN. Statistical decision rules and optimal inference. Providence, RI: American Mathematical Society; 2000. ISBN: 978-0-8218-1347-8.
22. Devroye L, Gyorfi L. Nonparametric density estimation: The L1 view. Wiley; 1985. ISBN: 978-0-4718-1646-1.
23. Efromovich S. Nonparametric curve estimation: Methods, theory, and applications. New York: Springer; 1999. ISBN: 978-0-387-98740-8.
24. Greblicki W, Pawlak M. Classification using the Fourier series estimate of multivariate density functions. IEEE Transactions on Systems, Man, and Cybernetics 1981; 11(10): 726-730. DOI: 10.1109/TSMC.1981.4308594.
25. Rutkowski L. Sequential pattern recognition procedures derived from multiple Fourier series. Pattern Recognition Letters 1988; 8(4): 213-216. DOI: 10.1016/0167-8655(88)90027-X.
26. Fei-Fei L, Fergus R, Perona P. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence 2006; 28(4): 594-611. DOI: 10.1109/TPAMI.2006.79.
27. Yi D, Lei Z, Liao S, LiSZ. Learning face representation from scratch. arXiv preprint arXiv:1411.7923, 2014.
28. Wasikowski M, Chen X. Combating the small sample class imbalance problem using feature selection. IEEE Transactions on Knowledge and Data Engineering 2010; 22(10): 1388-1400. DOI: 10.1109/TKDE.2009.187.
29. Zherdev DA, Kazanskiy NL, Fursov VA. Object recognition by the radar signatures of electromagnetic field scattering on base of support subspaces method [In Russian]. Computer Optics 2014; 38(3): 503-510.
30. Savchenko VV. The principle of the information-divergence minimum in the problem of spectral analysis of the random time series under the condition of small observation samples. Radiophysics and Quantum Electronics 2015; 58 (5): 373-379. DOI: 10.1007/s11141-015-9611-4.
31. Orlov AI. Development of the methods of mathematical research (2006-2015) [In Russian]. Industrial Laboratory. Diagnostics of Materials 2017; 83(1-I): 78-86.
32. Shatskikh SYa, Melkumova LE. Normality assumption in statistical data analysis. CEUR Workshop Proceedings 2016; 1638: 763-768. DOI: 10.18287/1613-0073-2016-1638-763-768.
33. Lapko AV, Lapko VA, Chentsov SV. Nonparametric models of pattern recognition under conditions of small samples. Optoelectronics, Instrumentation and Data Processing 1999; 6: 83-90.
34. Franti P, Kaukoranta T, Shen DF, Chang KS. Fast and memory efficient implementation of the exact PNN. IEEE Transactions on Image Processing 2000; 9(5): 773-777. DOI: 10.1109/83.841516
35. Savchenko AV. On improvement of computing efficiency of the probabilistic neural network in pattern recognition problem based on the orthogonal series estimates of class densities [In Russian]. Information Systems and Technologies 2015; 4: 28-38.
36. Rutkowski L. Adaptive probabilistic neural networks for pattern classification in time-varying environment. IEEE Transactions on Neural Networks 2004; 15(4): 811-827. DOI: 10.1109/TNN.2004.828757.
37. Duda P, Zurada JM. On the Cesaro orthogonal series-type kernel probabilistic neural networks handling non-stationary noise. Proceedings of International Conference on Parallel Processing and Applied Mathematics (LNCS) 2012; 7203(I): 435-442. DOI: 10.1007/978-3-642-31464-3_44.
38. Schwartz SC. Estimation of probability density by an orthogonal series. The Annals of Mathematical Statistics 1967: 1261-1265.
39. Efromovich S. Orthogonal series density estimation. Wiley Interdisciplinary Reviews: Computational Statistics 2010, 2(4): 467-476. DOI: 10.1002/wics.97.
40. Fichtenholz GM. Course of differential and integral calculus: Volume 3 [In Russian]. Moscow, “Fizmatlit” Publisher; 2001.
41. Zorich VA. Mathematical analysis: Part 2 [In Russian]. Moscow, “Nauka” Publisher; 1984.
42. Hall P. On trigonometric series estimates of densities. Annals of Statistics 1981; 9(3): 683-685.
43. Novosyolov AA. Parametrization of models of control systems [In Russian]. Herald of Siberian State Aerospace University 2010; 5: 52-56.
44. Hart JD. On the choice of a truncation point in Fourier series density estimation. Journal of Statistical Computation and Simulation 1985; 21(2): 95-116. DOI: 10.1080/00949658508810808.
45. Video-based face recognition software. Source: <https://git­hub.com/HSE-asavchenko/HSE_FaceRec/tree/master/src/re­cognition_testing>.
46. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Darrell T. Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM International Conference on Multimedia 2014: 675-678. DOI: 10.1145/2647868.2654889.
47. Wu X, He R, Sun Z., Tan T. A light CNN for deep face representation with noisy labels. arXiv preprint arXiv:1511.02683 2017.
48. Savchenko AV. Fast multi-class recognition of piecewise regular objects based on sequential three-way decisions and granular computing. Knowledge-Based Systems 2016; 91: 250-260. DOI: 10.1016/j.knosys.2015.09.021.
49. Savchenko AV. Sequential three-way decisions in efficient classification of piecewise stationary speech signals. International Joint Conference on Rough Sets (IJCRS 2017); II: 264-277. DOI: 10.1007/978-3-319-60840-2_19.

---

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20