Emotion Detection Ai | AI Final Year Project Report

Complete Technical Paper Content — Emotion Detection Ai

1. Executive Abstract, Problem Formulation & Background

Multimodal learning is a type of deep learning that integrates and processes multiple types of data, referred to as modalities, such as text, audio, images, or video. This integration allows for a more holistic understanding of complex data, improving model performance in tasks like visual question answering, cross-modal retrieval, text-to-image generation, aesthetic ranking, and image captioning. Multimodal learning was proposed in 2011 at the beginning of the deep learning period. Large multimodal models, such as Google Gemini and GPT-4o, have become increasingly popular since 2023, enabling increased versatility and a broader understanding of real-world phenomena.

Models such as CLIP (Contrastive Language–Image Pretraining) learn joint representations of images and text by optimizing contrastive objectives, allowing the model to match images with their corresponding textual descriptions.

Cross-modal retrieval: cross-modal retrieval allows users to search for data across different modalities (e.g., retrieving images based on text descriptions), improving multimedia search engines and content recommendation systems. Classification and missing data retrieval: multimodal Deep Boltzmann Machines outperform traditional models like support vector machines and latent Dirichlet allocation in classification tasks and can predict missing data in multimodal datasets, such as images and text. Healthcare diagnostics: multimodal models integrate medical imaging, genomic data, and patient records to improve diagnostic accuracy and early disease detection, especially in cancer screening. Content generation: models like DALL·E generate images from textual descriptions, benefiting creative industries, while cross-modal retrieval enables dynamic multimedia searches. Robotics and human-computer interaction: multimodal learning improves interaction in robotics and AI by integrating sensory inputs like speech, vision, and touch, aiding autonomous systems and human-computer interaction. Emotion recognition: combining visual, audio, and text data, multimodal systems enhance sentiment analysis and emotion recognition, applied in customer service, social media, and marketing. Code search: multimodal representations combining source code structure (e.g., abstract syntax trees) with natural language descriptions improve retrieval of relevant code snippets for a given query. Source code has also been rendered as a two-dimensional pixel image, for instance by converting a method's control-flow graph into a black-and-white image processed with a convolutional neural network to search for malwares.

2. Related Work, Literature Survey & Algorithmic Foundations

Affective computing is the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. It is an interdisciplinary field spanning computer science, psychology, and cognitive science. While some core ideas in the field may be traced as far back as to early philosophical inquiries into emotion, the modern idea originated with Rosalind Picard's 1995 paper entitled "Affective Computing" and her 1997 book of the same name published by MIT Press. One motivation for researching affective computing is the ability to give machines emotional intelligence, including simulating empathy. The goal is that a machine should interpret the emotional state of humans and adapt its behavior to those emotions, responding appropriately. Emotional AI is used to detect and recognize users' emotional states, and thereby offer real-time emotional support, such as generating personalized affective responses and interactions to regulate users emotions.

LDC – Classification happens based on the value obtained from the linear combination of the feature values, which are usually provided in the form of vector features. k-NN – Classification happens by locating the object in the feature space, and comparing it with the k nearest neighbors (training examples). The majority vote decides on the classification. GMM – A probabilistic model used for representing the existence of subpopulations within the overall population. Each sub-population is described using the mixture distribution, which allows for classification of observations into the sub-populations. SVM – A type of (usually binary) linear classifier which decides in which of the two (or more) possible classes, each input may fall into. ANN – is a mathematical model, inspired by biological neural networks, that can better grasp possible non-linearities of the feature space. Decision tree algorithms – work based on following a decision tree in which leaves represent the classification outcome, and branches represent the conjunction of subsequent features that lead to the classification. HMMs – a statistical Markov model in which the states and state transitions are not directly available to observation. Instead, the series of outputs dependent on the states are visible. In the case of affect recognition, the outputs represent the sequence of speech feature vectors, which allow the deduction of states' sequences through which the model progressed. The states can consist of various intermediate steps in the expression of an emotion, and each of them has a probability distribution over the possible output vectors. The states' sequences allow us to predict the affective state which we are trying to classify, and this is one of the most commonly used techniques within the area of speech affect detection. It has been proven that with enough acoustic evidence available, the emotional state of a person can be classified by a set of majority voting classifiers. The proposed set of classifiers is based on three main classifiers: kNN, C4.5 and SVM-RBF Kernel. This set achieves better performance than each basic classifier taken separately. It is compared with two other sets of classifiers: one-against-all (OAA) multiclass SVM with Hybrid kernels and the set of classifiers which consists of the following two basic classifiers: C5.0 and Neural Network. The proposed variant achieves better performance than the other two sets of classifiers.

Frequency characteristics Accent shape – affected by the rate of change of the fundamental frequency. Average pitch – describes how high/low the speaker speaks relative to normal speech. Contour slope – describes the tendency of the frequency change over time; it can be rising, falling or level. Final lowering – the amount by which the frequency falls at the end of an utterance. Pitch range – measures the spread between the maximum and minimum frequency of an utterance. Time-related features: Speech rate – describes the rate of words or syllables uttered over a unit of time Stress frequency – measures the rate of occurrences of pitch-accented utterances Voice quality parameters and energy descriptors: Breathiness – measures the aspiration noise in speech Brilliance – describes the dominance of high or low frequencies in the speech Loudness – measures the amplitude of the speech waveform, translates to the energy of an utterance Pause Discontinuity – describes the transitions between sound and silence Pitch Discontinuity – describes the transitions of the fundamental frequency.

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

Creation of an emotion database is a difficult and time-consuming task. However, database creation is an essential step in the formation of a system that will recognize human emotions. Most publicly available emotion databases include posed facial expressions only. In posed expression databases, the participants are asked to display different basic emotional expressions. In the alternative spontaneous expression databases, the expressions are natural. Spontaneous emotion elicitation requires significant effort in the selection of proper stimuli which can lead to a rich display of intended emotions. Secondly, the process requires emotions to be tagged by trained individuals manually, which makes the databases highly reliable. Since perception of expressions and their intensity is subjective in nature, the annotation by experts is essential for the purpose of validation. Researchers work with three types of databases: a database of peak expression images only, a database of image sequences portraying an emotion from neutral to its peak, and video clips with emotional annotations. Many facial expression databases have been created and made public for expression recognition purposes. Two widely used databases are CK+ and JAFFE.

During the late 1960's, Paul Ekman proposed the idea that facial expressions of emotion are not culturally determined, but universal. He discovered this by doing cross-cultural research in Papua, New Guinea on the Fore Tribesmen. Thus, he suggested that they are biological in origin and can, therefore, be safely and correctly categorized. He therefore officially put forth six basic emotions, in 1972:

Anger Disgust Fear Happiness Sadness Surprise However, in the 1990s, Ekman expanded his list of basic emotions, including a range of positive and negative emotions, not all of which are encoded in facial muscles. The new emotions are:

4. Neural Network Backbone, Layer Specifications & Loss Formulations

Amusement Contempt Contentment Embarrassment Excitement Guilt Pride in achievement Relief Satisfaction Sensory pleasure Shame

A system has been conceived by psychologists in order to formally categorize the physical expression of emotions on faces. The central concept of the Facial Action Coding System, or FACS, as created by Paul Ekman and Wallace V. Friesen in 1978 based on earlier work by Carl-Herman Hjortsjö. Action units (AU) are, basically, a contraction or a relaxation of one or more muscles. Psychologists have proposed the following classification of six basic emotions according to their action units ("+" here means "and"):

The fact that posed expressions are not natural, and therefore algorithms trained on these may not apply to natural expressions. The lack of rotational movement freedom. Affect detection works very well with frontal use, but upon rotating the head more than 20 degrees, "[there have] been problems". Facial expressions do not always correspond to an underlying emotion that matches them (e.g. they can be posed or faked, or a person can feel emotions but maintain a "poker face"). FACS did not include dynamics, which can help disambiguate (e.g. smiles of genuine happiness tend to have different dynamics than "try to look happy" smiles.) The FACS combinations do not correspond in a 1:1 way with the emotions that the psychologists originally proposed (note that this lack of a 1:1 mapping also occurs in speech recognition with homophones and homonyms and many other sources of ambiguity, and may be mitigated by bringing in other channels of information). Accuracy of recognition is improved by adding context; however, adding context and other modalities increases computational cost and complexity.

5. Training Protocol, Convergence Behavior & Experimental Evaluation

Gestures can be used as a means of detecting the particular emotional state of a user, especially when used in conjunction with speech and face recognition. Gestures could be simple reflexive responses, like lifting one's shoulders when one does not know the answer to a question, or they could be complex and meaningful, as when communicating with sign language. Without making use of any object or surrounding environment, we can wave our hands, clap or beckon. On the other hand, when using objects, we can point at them, move, touch, or handle them. A computer should be able to recognize these gestures, analyze the context, and respond in a meaningful way in order to be used for Human–Computer Interaction. Gestures can be detected using many methods. There are two main approaches towards gesture recognition: 3D model based and appearance-based. The foremost method makes use of 3D information of key elements of the body parts in order to obtain several important parameters, like palm position or joint angles. On the other hand, appearance-based systems use images or videos to for direct interpretation. Hand gestures have been a common focus of body gesture detection methods.

Facial electromyography is a technique used to measure the electrical activity of the facial muscles by amplifying the tiny electrical impulses that are generated by muscle fibers when they contract. The face expresses a great deal of emotion. However, there are two main facial muscle groups that are usually studied to detect emotion: The corrugator supercilii muscle, also known as the 'frowning' muscle, draws the brow down into a frown. Therefore, it is the best test for negative, unpleasant emotional responses.↵The zygomaticus major muscle is responsible for pulling the corners of the mouth back when one smiles, and therefore is the muscle used to test for a positive emotional response.

Galvanic skin response (GSR) is an outdated term for a more general phenomenon known as Electrodermal activity, or EDA. EDA is a general phenomena whereby the skin's electrical properties change. The skin is innervated by the sympathetic nervous system, so measuring its resistance or conductance provides a way to quantify small changes in the sympathetic branch of the autonomic nervous system. As sweat glands are activated, even before the skin feels sweaty, the level of the EDA can be captured (usually using conductance) and used to discern small changes in autonomic arousal. The more aroused a subject is, the greater the skin conductance tends to be. Skin conductance is often measured using two small silver-silver chloride electrodes placed somewhere on the skin and applying a small voltage between them. To maximize comfort and reduce irritation the electrodes can be placed on the wrist, legs, or feet, which leaves the hands fully free for daily activity.