Framework for the Classification of Facial Emotions Using Soft Computing
Techniques

Sourav      Maity; Karan      Veer

Abstract

Background: Facial emotion recognition (FER) technology is enumerated as a productive interface in several operations, which has been specifically focused on as a substitute communication path among a user and an appliance for human computer interface in the previous decade. The efficiency of the facial identification model straightaway relies on the capability of classification methods. In addition, an appropriate swap between recognition efficiency and computational cost is reckoned as the most important factor for planning such models.

Methods: The objective of this paper was to classify the facial emotion electromyogram (EMG) signals by means of a neural network algorithm (NN), support vector machine (SVM) algorithm, and Naive-Bayes algorithm. This research work was directed towards the correlation among the classification accuracies by applying distinct feature extraction procedures on fEMGs. At first, eight participants (six male and two female) were recruited for data recording. Four electrodes were placed on each participant's face for capturing facial gestures (happy, angry, sad, and fear) and two electrodes were placed on the wrist for grounding purposes. Data were recorded by using BIOPAC MP150. After this, the signals were filtered using a band-pass filter and segmentation techniques for enhanced processing. After that, the time-domain and frequency-domain feature extraction procedures were carried out. Time domain and frequency domain features were applied to recorded signals. In this research, we used LabVIEW and MATLAB to produce a set of characteristics from fEMG signals for four emotional conditions, such as anger, sad, fear, and happy. After the feature extraction process, the extracted features were aligned into respective emotions by applying classifiers. The extracted features were further trained and classified by applying the SVM classifier, neural network classifier, and Naive Bayes classifier in MATLAB 2020.

Results: The SVM classifier and neural network classifier generated an accuracy of 93.80% and 96.90%, respectively, whereas the Naive Bayes classifier generated an accuracy of 90.60%.

Conclusion: Facial emotion recognition (FER) is foresighted as a progressive or futuristic model, which has attracted the attention of researchers in several areas of learning due to its higher prospects in distinct applications. Acknowledgment of the emotions through biomedical signals produced from movements of facial muscles is lately presented using an explicit and authentic route.

Keywords: SVM, NN, RMS, MAV, IEMG, power spectral density.

[1]
Zecca M, Micera S, Carrozza MC, Dario P. Control of multifunctional prosthetic hands by processing the electromyographic signal. Crit Rev Biomed Eng  2002; 30(4-6): 459-85.
 [http://dx.doi.org/10.1615/CritRevBiomedEng.v30.i456.80] [PMID: 12739757]

[2]
Veer K. ‘Interpretation of surface electromyograms to characterize arm movement’. Instrum Sci Technol  2014; 42(5): 513-21.
 [http://dx.doi.org/10.1080/10739149.2014.913178]

[3]
Doulah ABMSU, Fattah SA. Wavelet domain feature extraction scheme based on dominant motor unit action potential of EMG signal for neuromuscular disease classification Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics AISTATS 2015, IEEE Trans Biomed Eng, deeply supervised nets. 8 

[4]
Veer K. A technique for classification and decomposition of muscle signal for control of myoelectric prostheses based on wavelet statistical classifier. Measurement  2015; 60: 283-91.
 [http://dx.doi.org/10.1016/j.measurement.2014.10.023]

[5]
Dwyer R. Detection of non-Gaussian signals by frequency domain kurtosis estimation International Conference on Acoustics, Speech, and Signal Processing (ICASSP’83).  8: 607-10.
 [http://dx.doi.org/10.1109/ICASSP.1983.1172264]

[6]
Maity S, Veer K. A generalized review of human-computer interaction using electromyogram signals. Recent Pat Eng 2022.
 [http://dx.doi.org/10.2174/1872212116666220518122621]

[7]
Hamedi M, Salleh S, Ting C, Astaraki M, Noor A. Robust facial expression recognition for MuCI: A comprehensive neuromuscular signal analysis In: IEEE Transactions on Affective Computing.   2018; 9: pp. (1)102-15.
 [http://dx.doi.org/10.1109/TAFFC.2016.2569098]

[8]
Hamedi M, Salleh S-H, Noor A. Facial neuromuscular signal classification by means of least square support vector machine for MuCI’. Applied Soft Computing.  Elsevier 2015; 30: pp. (C)83-93.
 [http://dx.doi.org/10.1016/j.asoc.2015.01.034]

[9]
Karan V. Wavelet transform-based classification of electromyogram signals using an ANOVA technique. Neurophysiology  2015; 47(4): 302-9.
 [http://dx.doi.org/10.1007/s11062-015-9537-7]

[10]
Huang C-N, Chen C-H, Chung H-Y. The review of applications and measurements in facial electromyography. J Med Biol Eng  2004; 25(1): 15-20.

[11]
Jerritta S, Murugappan M, Wan K, Yaacob S, Wan K, Yaacob S. Emotion recognition from facial EMG signals using higher order statistics and principal component analysis. Z red fruit go n constitute X UE look  2014; 37(3): 385-94.
 [http://dx.doi.org/10.1080/02533839.2013.799946]

[12]
Jiang M, Rahmani A, Westerlund T, Liljeberg P, Tenhunen H.  Facial expression recognition with sEMG method International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing. 981-8. UK
 [http://dx.doi.org/10.1109/CIT/IUCC/DASC/PICOM.2015.148]

[13]
Veer K, Sharma T. A novel feature extraction for robust EMG pattern recognition. J Med Eng Technol  2016; 40(4): 149-54.
 [http://dx.doi.org/10.3109/03091902.2016.1153739] [PMID: 27004618]

[14]
Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE. A survey of deep neural network architectures and their applications. Neurocomputing  2017; 234: 11-26.http://bura.brunel.ac.uk/handle/2438/14221
 [http://dx.doi.org/10.1016/j.neucom.2016.12.038]

[15]
Mahyar H, Salleh S-H. Surface electromyography-based facial expression recognition in bi-polar configuration Journal of Computer Science.  Science Publications 2011; 7: pp. (9)1407-5.

[16]
Veer K. Development of sensor system with measurement of surface electromyogram signal for clinical use. Optik  2016; 127(1): 352-6.
 [http://dx.doi.org/10.1016/j.ijleo.2015.10.072]

[17]
Cheong JH, Jolly E, Xie T, Byrne S, Kenney M, Chang LJ. Py-feat: a Python facial expression analysis toolbox.  Affective Science 2023; pp. 1-16.

[18]
Otte E, Habel U, Schulte-Rüther M, Konrad K, Koch I. Interference in simultaneously perceiving and producing facial expressions—Evidence from electromyography. Neuropsychologia  2011; 49(1): 124-30.
 [http://dx.doi.org/10.1016/j.neuropsychologia.2010.11.005] [PMID: 21075127]

[19]
Mithbavkar SA, Shah MS.  Recognition of emotion through facial expressions using EMG signal 2019 International Conference on Nascent Technologies in Engineering (ICNTE). Navi Mumbai, India. 2019; pp. 1-6.
 [http://dx.doi.org/10.1109/ICNTE44896.2019.8945843]

[20]
Perusquia-Hernandez M, Hirokawa M, Suzuki K.  Spontaneous and posed smile recognition based on spatial and temporal patterns of facial EMG 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). San Antonio, TX, USA. 2015; pp. 537-41.

[21]
Murugan PR, Varghese SM. EMG signal classification using ANN and ANFIS for neuro-muscular disorders. Int J Biomed Eng Technol  2014; 16(2): 156-68.
 [http://dx.doi.org/10.1504/IJBET.2014.065657]

[22]
Reaz MBI, Hussain MS, Mohd-Yasin F. Techniques of EMG signal analysis detection, processing, classification and applications Biol Proced  2006; 8(1): 11-35. Published online 23 March
 [http://dx.doi.org/10.1251/bpo115]

[23]
Ren S, He K, Girshick R, Sun J. Faster-CNN: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell  2017; 39(6): 1137-49.
 [http://dx.doi.org/10.1109/TPAMI.2016.2577031] [PMID: 27295650]

[24]
Rezazadeh IM, Wan X, Wang R.  Toward affective handsfree human machine interface approach in virtual environments-based equipment operation training 9th International Conference on Construction Applications of Virtual Reality.

[25]
Ryan A, Kohn JF, Lucey S, Saragih J, Lucey P.  Automated facial expression recognition system Proceedings of the IEEE International Carnahan Conference on Security Technology.

[26]
Sharma T, Veer K. Wavelet analysis-based evaluation of electromyogram signal using human machine cooperation. Int J Med Eng Inform  2016; 8(4): 317-28.
 [http://dx.doi.org/10.1504/IJMEI.2016.079358]

[27]
Sinha R, Parsons OA.  Multivariate response patterning of fear Conference on Cognition and Emotion. Vol. 10: 173-98.
 [http://dx.doi.org/10.1080/026999396380321]

[28]
Smolka J, Zurek S, Lukasik E, Skublewska-Paszkowska M.  Heart rate estimationusing an EMG system integrated with a motion capture system International Conference on Electromagnetic Devices and Processes in Environment Protection with Seminar Applications of Superconductors (ELMECO & AoS), Lublin. 1-4.

[29]
Song C, Chen C, Li Y, Wu X.  Deep reinforcement learning apply in electromyography data classification 2018 IEEE International Conference on Cyborg and Bionic Systems (CBS).
 [http://dx.doi.org/10.1109/CBS.2018.8612213]

[30]
Thulkar D, Hamde ST.  Facial electromyography for characterization of emotionsusing LabVIEW IEEE International Conference on Industrial Instrumentation and Control (IClC).

[31]
Zhang X.  Wavelet transform theory and its application in EMG signal processing Seventh International Conference on Fuzzy Systems and Knowledge Discovery.
 [http://dx.doi.org/10.1109/FSKD.2010.5569532]

[32]
Gunes H, Piccardi M, Jan T. Face and body gesture recognition for a vision-based multimodal analyzer Proceedings of Area Workshop on Visual Information Processing.  Sydney, Australia. 2003; 36: pp. 19-28.

[33]
Arjunan SP, Kumar DK.  Recognition of facial movements and hand gestures using surface Electromyogram (sEMG) for HCI based applications Proceedings of 9th Biennial Conference of the Australian Pattern Recognition Society on Digital Image Computing Techniquesand Applications. 1-6.Dec. 3-5, 2008;
 [http://dx.doi.org/10.1109/DICTA.2007.4426768]

[34]
Firoozabadi SMP, Oskoei MA, Hu H.  A human-computer interface based on forehead multi-channel bio-signals to control a virtual wheelchair Proceedings of 14th ICBME. 272-7.Feb. 13-14, 2008;

[35]
Ang LBP, Belen EF, Bernardo RA, Boongaling ER, Briones GH, Coronel JB. Facial expression recognition through pattern analysis of facial muscle movements utilizing electromyogram sensors. Proceedings of the TENCON IEEE Region 10 Conference. Chiang Mai, Thailand. 2004. pp. 3: 600-3. IEEE 0-7803-8560-8.
 [http://dx.doi.org/10.1109/TENCON.2004.1414843]

[36]
Gibert G, Pruzinec M, Schultz T, Stevens K.  Enhancement of human computer interaction with facial electromyographic sensors Proceeding of the 21st Annual Conference of the Australian Computer Human Interaction Special Interest Group on Design Open 247 OzCHI. 1-4.
 [http://dx.doi.org/10.1145/1738826.1738914]

[37]
Hamedi M, Rezazadeh IM, Firoozabadi SMP.  Facial gesture recognition using Two-channel biosensors configuration and fuzzy classifier: A pilot study Proceeding of the International Conference on Electrical, Control and Computer Engineering. Pahang, Malaysia. Jun. 21-22, 2011;

[38]
Turabzadeh S, Meng H, Swash R, Pleva M, Juhar J. Facial expression emotion detection for real-time embedded systems. Technologies  2018; 6(1): 17.
 [http://dx.doi.org/10.3390/technologies6010017]

[39]
Huang Y, Chen F, Lv S, Wang X. Facial expression recognition: A survey. Symmetry  2019; 11(10): 1189.
 [http://dx.doi.org/10.3390/sym11101189]

[40]
Weyers P, Mühlberger A, Hefele C, Pauli P. Electromyographic responses to static and dynamic avatar emotional facial expressions. Psychophysiology  2006; 43(5): 450-3.
 [http://dx.doi.org/10.1111/j.1469-8986.2006.00451.x] [PMID: 16965606]

[41]
Zahak M. Signal Acquisition Using Surface EMG and Circuit Design Considerations for Robotic Prosthesis. Computational Intelligence in Electromyography Analysis–A Perspective on Current Applications and Future Challenges. London, UK: InTech 2012.
 [http://dx.doi.org/10.5772/52556]

[42]
Krose B, van der Smagt P. An Introduction to Neural Network. Amsterdam, Netherlands: University of Amsterdam 1996.

[43]
Wagner J, Kim J, Andre E. From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification. Proceedings of the 2005 IEEE International Conference on Multimedia and Expo.  Amsterdam, the Netherlands. 2005; pp. 940-3.
 [http://dx.doi.org/10.1109/ICME.2005.1521579]

[44]
Wioleta S. Using physiological signals for emotion recognition. Proceedings of the 2013 6th International Conference on Human System Interactions (HSI).  Gdansk, Poland. 2013; pp. 556-61.
 [http://dx.doi.org/10.1109/HSI.2013.6577880]

[45]
Girardi D, Lanubile F, Novielli N. Emotion detection using noninvasive low cost sensors. Proceedings of the 2017 Seventh International Conference on A_ective Computing and Intelligent Interaction (ACII). San Antonio, TX, USA.   2017; p. 125-30.
 [http://dx.doi.org/10.1109/ACII.2017.8273589]

[46]
Martínez-Rodrigo A, Zangróniz R, Pastor JM, Latorre JM, Fernández-Caballero A. Emotion detection in ageing adults from physiological sensors. Proceedings of the Advances in Intelligent Systems and Computing.  Volume 376: 253-61.
 [http://dx.doi.org/10.1007/978-3-319-19695-4_26]

[47]
Nakasone A, Prendinger H, Ishizuka M.  ProComp infiniti biosignal encoder. Proceedings of the 5th International Workshop on Biosignal Interpretation. Tokyo, Janpan. 2005; pp. 219-22.

[48]
Ahsan MR, Ibrahimy MI, Khalifa OO. EMG signal classification for human computer interaction: A review. Eur J Sci Res  2009; 33(3): 480-501.

[49]
Micera S, Sabatini AM, Dario P, Rossi B. A hybrid approach to EMG pattern analysis for classification of arm movements using statistical and fuzzy techniques. Med Eng Phys  1999; 21(5): 303-11.
 [http://dx.doi.org/10.1016/S1350-4533(99)00055-7] [PMID: 10576421]

[50]
Boxtel A. Facial EMG as a tool for inferring a_ective states. Proc Meas Behav  2010; 2010: 104-8.

[51]
Ferreira C. Designing neural networks using gene expression programming Applied soft computing technologies: The challenge of complexity.  Springer-Verlag Berlin Heidelberg 2006; pp. 517-35.

[52]
Novák V, Perfilieva I, Mockor J. Mathematical principles of fuzzy logic. Springer Science & Business Media 2012.

[53]
Albertos P, Sala A. Fuzzy logic controllers. VIII International Congress of Automatic Control. 

[54]
Poole DL, Mackworth AK. Artificial Intelligence: foundations of computational agents. USA: Cambridge University Press 2010.
 [http://dx.doi.org/10.1017/CBO9780511794797]

[55]
Mitchell TM. Machine Learning. USA: McGraw-Hill Education 1997.

[56]
Raschka S. Linear discriminant analysis - bit by bit 2014.

[57]
Phinyomark A, Hu H, Phukpattaranont P, Limsakul C. Application of linear discriminant analysis in dimensionality reduction for hand motion classification. Meas Sci Rev  2012; 12(3): 82-9.
 [http://dx.doi.org/10.2478/v10048-012-0015-8]

[58]
Buckley JJ, Hayashi Y. Fuzzy neural networks: A survey. Fuzzy Sets Syst  1994; 66(1): 1-13.
 [http://dx.doi.org/10.1016/0165-0114(94)90297-6]

[59]
Lotte F, Congedo M, Lécuyer A, Lamarche F, Arnaldi B. A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng  2007; 4(2): R1-R13.
 [http://dx.doi.org/10.1088/1741-2560/4/2/R01] [PMID: 17409472]

[60]
Cesqui B, Tropea P, Micera S, Krebs H. EMG-based pattern recognition approach in post stroke robot-aided rehabilitation: A feasibility study. J Neuroeng Rehabil  2013; 10(1): 75.
 [http://dx.doi.org/10.1186/1743-0003-10-75] [PMID: 23855907]

[61]
Zhang ZG, Liu HT, Chan SC, Luk KDK, Hu Y. Time-dependent power spectral density estimation of surface electromyography during isometric muscle contraction: Methods and comparisons. J Electromyogr Kinesiol  2010; 20(1): 89-101.
 [http://dx.doi.org/10.1016/j.jelekin.2008.09.007] [PMID: 19027325]

[62]
Spiewak C, Islam M, Zaman M A U, Rahman MH. A comprehensive study on EMG feature extraction and classifiers. Open Access J Biomed Eng Biosci  2018; 1(1): 1-10.
 [http://dx.doi.org/10.32474/OAJBEB.2018.01.000104]

[63]
Changmok C, Micera S, Carpaneto J. Jung Kim. Development and quantitative performance evaluation of a noninvasive EMG computer interface. IEEE Trans Biomed Eng  2009; 56(1): 188-91.
 [http://dx.doi.org/10.1109/TBME.2008.2005950] [PMID: 19224732]

[64]
Han JS, Song WK, Kim JS, Bang WC, Heyoung L, Zeungnam B. New EMG pattern recognition based on soft computing techniques and its application to control of a rehabilitation robotic arm. Proc of 6th International Conference on Soft Computing (IIZUKA2000). 

Cite As

Current Signal Transduction Therapy

Framework for the Classification of Facial Emotions Using Soft Computing Techniques

Abstract