Emotion Recognition

Humans use different components of interactions to recognize emotions: voice, facial expressions, gestures, and text. The same is true for computers. They use different inputs, such as speech, images, videos, text, and physiological sensors, to recognize emotions. Also, we can combine processing results from different input channels in order to have a more precise picture of a given human’s state; this is called multimodal emotion recognition. So, the first step is to collect a dataset (or datasets in the case of the multimodal inputs) with examples of different emotions and label them. In general, there are six basic cross-cultural emotions corresponding to Ekman’s model: anger, disgust, fear, happiness, sadness, surprise. Quite often, data scientists add 12 compound emotions (based on basic emotions, e.g. angrily surprised, sadly disgusted, etc.), additional emotions (appall, hate, and awe), and plus neutral.

Then we train a model to classify emotions from different samples. Nowadays, the most efficient models are based on neural networks such as recurrent neural networks (RNN) or convolutional neural networks (CNN). For multimodal emotion recognition, there is also a fusion bloc that combines results from different channels and produces a final decision. The accuracy of such models varies from 60% to 95% depending on the database being used, the architecture of the neural network, and the signal type/types.

Emotion Recognition

Emotion Recognition

Let’s have a look at different types of recognition.

• Speech Emotion Recognition (SER). SER involves analyzing vocal behavior as an affect marker that focuses on the nonverbal aspects of speech. For example, anger often produces changes in respiration and an increase in muscle tension. Consequently, the vocal folds and vocal tract shape have additional vibration, and the acoustic characteristics of speech change. Speech signals are a good source for effective computing. This technology provides a more economical option (as only a microphone is needed) when compared to using biological signals (which may require an electrocardiogram) or facial recognition (which requires a camera). Furthermore, this technology has a wide range of applications. These include emotional hearing aids for people with autism; detection of an angry caller at an automated call center to transfer to a human; and presentation style adjustment of a computerized e-learning tutor if the student is bored.
• Facial Emotion Recognition (FER). FER is based on visual information: images and video. However, FER can be conducted using multiple sensors. For example, Microsoft Kinect 3D modeling is used. It has an infrared emitter and two cameras, so it is able to measure depth. Such a system solves two central problems of image recognition: proper lighting and pose behavior. There are several FER applications worth mentioning. Cars with AI can alert their driver when they are feeling drowsy. FER can be also used for marketing research (e.g., observation of a user’s reaction while interacting with a specific brand or product). One more usage of FER, as well as SER, involves ATM security: if a person withdrawing money is scared, the ATM will not dispense cash, as a safety measure.
• Multimodal Emotion recognition. There are different types of multimodal emotion recognition. Some of them combine recognition from speech and text data. Another use involves images and voice data. There are also more exotic systems that combine images, speech and text with physiological sensors (e.g., electrocardiography, electroencephalography). Multimodal systems give a wider picture of the human emotional state. For example, these systems be used in healthcare for determining patients' feelings and comfort level regarding their treatment.