Probability Model Based on Cluster Analysis to Classify Sequences of Observations for Small Training Sets
Abstract
The problem of recognizing patterns, when there are few training data available, is particularly relevant and arises in cases when collection of training data is expensive or essentially impossible. The work proposes a new probability model MC&CL (Markov Chain and Clusters) based on a combination of markov chain and algorithm of clustering (self-organizing map of Kohonen, k-means method), to solve a problem of classifying sequences of observations, when the amount of training dataset is low. An original experimental comparison is made between the developed model (MC&CL) and a number of the other popular models to classify sequences: HMM (Hidden Markov Model), HCRF (Hidden Conditional Random Fields),LSTM (Long Short-Term Memory), kNN+DTW (k-Nearest Neighbors algorithm + Dynamic Time Warping algorithm). A comparison is made using synthetic random sequences, generated from the hidden markov model, with noise added to training specimens. The best accuracy of classifying the suggested model is shown, as compared to those under review, when the amount of training data is low.References
J. R. Rohlicek, W. Russell, S. Roukod, and H. Gish. Continuous hidden markov model for speaker independent word spotting, International Conference on Audio, Speech and Signal Processing, vol. 1, pp. 627–630, 1989
J. G. Wilpon, L.R. Rabiner, and C. Lee. Automatic recognition of keywords in unconstrained speech using hidden Markov models,IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 38, no. 11, pp. 1870–1878, 1990.
B. H. Williams, M. Toussaint, A. J. Storkey. A primitive based generative model to infer timing information in unpartitioned handwriting data, IJCAI, vol. 2, pp. 1119–1124, 2007.
M. Elmezain, A. Al-Hamadi, and M. Bernd. A Hidden Markov Model Based Isolated and Meaningful Hand Gesture Recognition, International Journal of Electrical & Electronics Engineering, vol. 3, iss. 4, pp. 156–163, 2009.
S. Wang, A. Quattoni, L. P. Morency, D. Demirdjian, and T. Darrell. Hidden Conditional Random Fields for Gesture Recognition,In Conference on Computer Vision and Pattern Recognition (CVPR),2006.
V. Chandola, A. Banerjee, and V. Kumar. Anomaly Detection: A Survey: Technical Report, Minneapolis, Department of Computer Science and Engineering University of Minnesota, 2007.
E. Khalastchi, G. A. Kaminka, M. Kalech, and R. Lin. Online anomaly detection in unmanned vehicles, In The 10th International Conference on Autonomous Agents and Multiagent Systems, vol. 1, pp. 115–122, 2011.
S. Hansen. Fault Diagnosis and Fault Handling for Autonomous Aircraft, Ph.D. dissertation, Technical University of Denmark, Department of Electrical Engineering, Denmark, 2012.
E. Keogh. UCR Time Series Classification Archive, URL: http://www.cs.ucr.edu/∼eamonn/time series data/
A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, and J. Schmidhuber. Novel Connectionist System for Improved Unconstrained Handwriting Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence. vol.31, no. 5, 2009.
A. Graves, Abdel-rahman Mohamed, and G. Hinton. Speech Recognition with Deep Recurrent Neural Networks, Acoustics, Speech and Signal Processing (ICASSP) IEEE International Conference, pp. 6645–6649, 2013
D. Koller, and N. Friedman. Probabilistic Graphical Models, Massachusetts, MIT Press, 2009.
A. B. Merkov. Recognition of patterns: Introduction to methods of statistical learning, Moscow, Editorial URSS, 2011.
Z. Taushanov, and A. Berchtold, A Direct Local Search Method and its Application to a Markovian Model, SOIC (Statistics,Optimization and Information Computing: An International Journal), vol. 5, no. 1, pp. 19–34, 2017.
Sutton, and A. McCallum. An Introduction to Conditional Random Fields for Relational Learning, Massachusetts, MIT Press, 2006.
L.R. Rabiner. Hidden Markov models and their use in selected applications when recognizing speech: review, TIIER, ch. 77(2), pp. 86–120, 1989.
R.V. Andreao, B. Dorizzi, and J. Boudy. ECG signal analysis through hidden Markov models, Biomedical Engineering, IEEE Transactions, vol. 53, iss. 8, pp. 1541–1549, 2006.
A. Gunawardana, M. Mahajan, A. Acero, and J. C. Platt. Hidden Conditional Random Fields for Phone Classification, Interspeech,pp. 1117–1120, 2005.
Quattoni, S. Wang, L. P. Morency, M. Collins, and T. Darrell. Hidden-state Conditional Random Fields, IEEE PAMI, 2007.
S. Wang, A. Quattoni, L. P. Morency, D. Demirdjian, and T. Darrell. Hidden Conditional Random Fields for Gesture Recognition,In Conference on Computer Vision and Pattern Recognition (CVPR),2006.
I.N. Palamar, and S.S.Yulin. Generative probabilistic graphical model based on self-organizing map, Proceedings of SPIIRAN,Saint-Petersburg, no. 2, pp. 227–247, 2014.
N. Palamar, and S. S. Yulin. Probabilistic Graphical Model Based on Growing Neural Gas for Long Time Series Classification,Modern Applied Science, Canada (Toronto), vol.9, no 2, pp. 109–116, 2015.
V.P. Vapnik. Recovery of empirical data dependencies, Moscow, Nauka, 1979.
A. Ng, and M. Jordan. On Discriminative vs. Generative Classifiers: A comparison of logistic regression and Naive Bayes, In Advances in Neural Information Processing Systems 14, pp. 841–848, 2002.
J.-H. Xue, and D.M. Titterington. Comment on ¡¡discriminative vs. generative classifiers: a comparison of logistic regression and naive Bayes¿¿, Neural Processing Letters, vol. 28, iss. 3, pp. 169–187,2008.
P. Liang, and M. I. Jordan. An asymptotic analysis of generative, discriminative, and pseudo-likelihood estimators, In Proceedings of the 25th International Conference on Machine Learning (ICML), 2008.
GitHub. Probabilistic Modeling Toolkit for Matlab/Octave, [Online resource], 2010. URL: https://github.com/probml/pmtk3.
SourceForge. HCRF library (including CRF and LDCRF) [Online resource], 2011. URL: https://sourceforge.net/projects/hcrf/.
S. Julin. Bitbucket. PhD Codesource [Online resource], 2015. URL: https://bitbucket.org/sjulin/phd enterprisecode.
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
