Affective computing

Last updated
Electronic devices such as robots are increasingly able to recognise and mimic human emotion Sophia at the AI for Good Global Summit 2018 (27254369347) (cropped).jpg
Electronic devices such as robots are increasingly able to recognise and mimic human emotion

Affective computing is the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. It is an interdisciplinary field spanning computer science, psychology, and cognitive science. [1] While some core ideas in the field may be traced as far back as to early philosophical inquiries into emotion, [2] the more modern branch of computer science originated with Rosalind Picard's 1995 paper [3] on affective computing and her book Affective Computing [4] published by MIT Press. [5] [6] One of the motivations for the research is the ability to give machines emotional intelligence, including to simulate empathy. The machine should interpret the emotional state of humans and adapt its behavior to them, giving an appropriate response to those emotions.

Contents

Areas

Detecting and recognizing emotional information

Detecting emotional information usually begins with passive sensors that capture data about the user's physical state or behavior without interpreting the input. The data gathered is analogous to the cues humans use to perceive emotions in others. For example, a video camera might capture facial expressions, body posture, and gestures, while a microphone might capture speech. Other sensors detect emotional cues by directly measuring physiological data, such as skin temperature and galvanic resistance. [7]

Recognizing emotional information requires the extraction of meaningful patterns from the gathered data. This is done using machine learning techniques that process different modalities, such as speech recognition, natural language processing, or facial expression detection. The goal of most of these techniques is to produce labels that would match the labels a human perceiver would give in the same situation: For example, if a person makes a facial expression furrowing their brow, then the computer vision system might be taught to label their face as appearing "confused" or as "concentrating" or "slightly negative" (as opposed to positive, which it might say if they were smiling in a happy-appearing way). These labels may or may not correspond to what the person is actually feeling.

Emotion in machines

Another area within affective computing is the design of computational devices proposed to exhibit either innate emotional capabilities or that are capable of convincingly simulating emotions. A more practical approach, based on current technological capabilities, is the simulation of emotions in conversational agents in order to enrich and facilitate interactivity between human and machine. [8]

Marvin Minsky, one of the pioneering computer scientists in artificial intelligence, relates emotions to the broader issues of machine intelligence stating in The Emotion Machine that emotion is "not especially different from the processes that we call 'thinking.'" [9] The innovative approach "digital humans" or virtual humans includes an attempt to give these programs, which simulate humans, the emotional dimension as well, including reactions in accordance with the reaction that a real person would react in a certain emotionally stimulating situation as well as facial expressions and gestures. [10]

Emotion in machines often refers to emotion in computational, often AI-based, systems. As a result, the terms 'emotional AI' and 'emotion AI' are being used. [11]

Technologies

In psychology, cognitive science, and in neuroscience, there have been two main approaches for describing how humans perceive and classify emotion: continuous or categorical. The continuous approach tends to use dimensions such as negative vs. positive, calm vs. aroused.

The categorical approach tends to use discrete classes such as happy, sad, angry, fearful, surprise, disgust. Different kinds of machine learning regression and classification models can be used for having machines produce continuous or discrete labels. Sometimes models are also built that allow combinations across the categories, e.g. a happy-surprised face or a fearful-surprised face. [12]

The following sections consider many of the kinds of input data used for the task of emotion recognition.

Emotional speech

Various changes in the autonomic nervous system can indirectly alter a person's speech, and affective technologies can leverage this information to recognize emotion. For example, speech produced in a state of fear, anger, or joy becomes fast, loud, and precisely enunciated, with a higher and wider range in pitch, whereas emotions such as tiredness, boredom, or sadness tend to generate slow, low-pitched, and slurred speech. [13] Some emotions have been found to be more easily computationally identified, such as anger [14] or approval. [15]

Emotional speech processing technologies recognize the user's emotional state using computational analysis of speech features. Vocal parameters and prosodic features such as pitch variables and speech rate can be analyzed through pattern recognition techniques. [14] [16]

Speech analysis is an effective method of identifying affective state, having an average reported accuracy of 70 to 80% in research from 2003 and 2006. [17] [18] These systems tend to outperform average human accuracy (approximately 60% [14] ) but are less accurate than systems which employ other modalities for emotion detection, such as physiological states or facial expressions. [19] However, since many speech characteristics are independent of semantics or culture, this technique is considered to be a promising route for further research. [20]

Algorithms

The process of speech/text affect detection requires the creation of a reliable database, knowledge base, or vector space model, [21] broad enough to fit every need for its application, as well as the selection of a successful classifier which will allow for quick and accurate emotion identification.

As of 2010, the most frequently used classifiers were linear discriminant classifiers (LDC), k-nearest neighbor (k-NN), Gaussian mixture model (GMM), support vector machines (SVM), artificial neural networks (ANN), decision tree algorithms and hidden Markov models (HMMs). [22] Various studies showed that choosing the appropriate classifier can significantly enhance the overall performance of the system. [19] The list below gives a brief description of each algorithm:

  • LDC – Classification happens based on the value obtained from the linear combination of the feature values, which are usually provided in the form of vector features.
  • k-NN – Classification happens by locating the object in the feature space, and comparing it with the k nearest neighbors (training examples). The majority vote decides on the classification.
  • GMM – is a probabilistic model used for representing the existence of subpopulations within the overall population. Each sub-population is described using the mixture distribution, which allows for classification of observations into the sub-populations. [23]
  • SVM – is a type of (usually binary) linear classifier which decides in which of the two (or more) possible classes, each input may fall into.
  • ANN – is a mathematical model, inspired by biological neural networks, that can better grasp possible non-linearities of the feature space.
  • Decision tree algorithms – work based on following a decision tree in which leaves represent the classification outcome, and branches represent the conjunction of subsequent features that lead to the classification.
  • HMMs – a statistical Markov model in which the states and state transitions are not directly available to observation. Instead, the series of outputs dependent on the states are visible. In the case of affect recognition, the outputs represent the sequence of speech feature vectors, which allow the deduction of states' sequences through which the model progressed. The states can consist of various intermediate steps in the expression of an emotion, and each of them has a probability distribution over the possible output vectors. The states' sequences allow us to predict the affective state which we are trying to classify, and this is one of the most commonly used techniques within the area of speech affect detection.

It is proved that having enough acoustic evidence available the emotional state of a person can be classified by a set of majority voting classifiers. The proposed set of classifiers is based on three main classifiers: kNN, C4.5 and SVM-RBF Kernel. This set achieves better performance than each basic classifier taken separately. It is compared with two other sets of classifiers: one-against-all (OAA) multiclass SVM with Hybrid kernels and the set of classifiers which consists of the following two basic classifiers: C5.0 and Neural Network. The proposed variant achieves better performance than the other two sets of classifiers. [24]

Databases

The vast majority of present systems are data-dependent. This creates one of the biggest challenges in detecting emotions based on speech, as it implicates choosing an appropriate database used to train the classifier. Most of the currently possessed data was obtained from actors and is thus a representation of archetypal emotions. Those so-called acted databases are usually based on the Basic Emotions theory (by Paul Ekman), which assumes the existence of six basic emotions (anger, fear, disgust, surprise, joy, sadness), the others simply being a mix of the former ones. [25] Nevertheless, these still offer high audio quality and balanced classes (although often too few), which contribute to high success rates in recognizing emotions.

However, for real life application, naturalistic data is preferred. A naturalistic database can be produced by observation and analysis of subjects in their natural context. Ultimately, such database should allow the system to recognize emotions based on their context as well as work out the goals and outcomes of the interaction. The nature of this type of data allows for authentic real life implementation, due to the fact it describes states naturally occurring during the human–computer interaction (HCI).

Despite the numerous advantages which naturalistic data has over acted data, it is difficult to obtain and usually has low emotional intensity. Moreover, data obtained in a natural context has lower signal quality, due to surroundings noise and distance of the subjects from the microphone. The first attempt to produce such database was the FAU Aibo Emotion Corpus for CEICES (Combining Efforts for Improving Automatic Classification of Emotional User States), which was developed based on a realistic context of children (age 10–13) playing with Sony's Aibo robot pet. [26] [27] Likewise, producing one standard database for all emotional research would provide a method of evaluating and comparing different affect recognition systems.

Speech descriptors

The complexity of the affect recognition process increases with the number of classes (affects) and speech descriptors used within the classifier. It is, therefore, crucial to select only the most relevant features in order to assure the ability of the model to successfully identify emotions, as well as increasing the performance, which is particularly significant to real-time detection. The range of possible choices is vast, with some studies mentioning the use of over 200 distinct features. [22] It is crucial to identify those that are redundant and undesirable in order to optimize the system and increase the success rate of correct emotion detection. The most common speech characteristics are categorized into the following groups. [26] [27]

  1. Frequency characteristics [28]
    • Accent shape – affected by the rate of change of the fundamental frequency.
    • Average pitch – description of how high/low the speaker speaks relative to the normal speech.
    • Contour slope – describes the tendency of the frequency change over time, it can be rising, falling or level.
    • Final lowering – the amount by which the frequency falls at the end of an utterance.
    • Pitch range – measures the spread between the maximum and minimum frequency of an utterance.
  2. Time-related features:
    • Speech rate – describes the rate of words or syllables uttered over a unit of time
    • Stress frequency – measures the rate of occurrences of pitch accented utterances
  3. Voice quality parameters and energy descriptors:
    • Breathiness – measures the aspiration noise in speech
    • Brilliance – describes the dominance of high or low frequencies In the speech
    • Loudness – measures the amplitude of the speech waveform, translates to the energy of an utterance
    • Pause Discontinuity – describes the transitions between sound and silence
    • Pitch Discontinuity – describes the transitions of the fundamental frequency.

Facial affect detection

The detection and processing of facial expression are achieved through various methods such as optical flow, hidden Markov models, neural network processing or active appearance models. More than one modality can be combined or fused (multimodal recognition, e.g. facial expressions and speech prosody, [29] facial expressions and hand gestures, [30] or facial expressions with speech and text for multimodal data and metadata analysis) to provide a more robust estimation of the subject's emotional state.

Facial expression databases

Creation of an emotion database is a difficult and time-consuming task. However, database creation is an essential step in the creation of a system that will recognize human emotions. Most of the publicly available emotion databases include posed facial expressions only. In posed expression databases, the participants are asked to display different basic emotional expressions, while in spontaneous expression database, the expressions are natural. Spontaneous emotion elicitation requires significant effort in the selection of proper stimuli which can lead to a rich display of intended emotions. Secondly, the process involves tagging of emotions by trained individuals manually which makes the databases highly reliable. Since perception of expressions and their intensity is subjective in nature, the annotation by experts is essential for the purpose of validation.

Researchers work with three types of databases, such as a database of peak expression images only, a database of image sequences portraying an emotion from neutral to its peak, and video clips with emotional annotations. Many facial expression databases have been created and made public for expression recognition purpose. Two of the widely used databases are CK+ and JAFFE.

Emotion classification

By doing cross-cultural research in Papua, New Guinea, on the Fore Tribesmen, at the end of the 1960s, Paul Ekman proposed the idea that facial expressions of emotion are not culturally determined, but universal. Thus, he suggested that they are biological in origin and can, therefore, be safely and correctly categorized. [25] He therefore officially put forth six basic emotions, in 1972: [31]

However, in the 1990s Ekman expanded his list of basic emotions, including a range of positive and negative emotions not all of which are encoded in facial muscles. [32] The newly included emotions are:

  1. Amusement
  2. Contempt
  3. Contentment
  4. Embarrassment
  5. Excitement
  6. Guilt
  7. Pride in achievement
  8. Relief
  9. Satisfaction
  10. Sensory pleasure
  11. Shame

Facial Action Coding System

A system has been conceived by psychologists in order to formally categorize the physical expression of emotions on faces. The central concept of the Facial Action Coding System, or FACS, as created by Paul Ekman and Wallace V. Friesen in 1978 based on earlier work by Carl-Herman Hjortsjö [33] are action units (AU). They are, basically, a contraction or a relaxation of one or more muscles. Psychologists have proposed the following classification of six basic emotions, according to their action units ("+" here mean "and"):

EmotionAction units
Happiness6+12
Sadness1+4+15
Surprise1+2+5B+26
Fear1+2+4+5+20+26
Anger4+5+7+23
Disgust9+15+16
ContemptR12A+R14A

Challenges in facial detection

As with every computational practice, in affect detection by facial processing, some obstacles need to be surpassed, in order to fully unlock the hidden potential of the overall algorithm or method employed. In the early days of almost every kind of AI-based detection (speech recognition, face recognition, affect recognition), the accuracy of modeling and tracking has been an issue. As hardware evolves, as more data are collected and as new discoveries are made and new practices introduced, this lack of accuracy fades, leaving behind noise issues. However, methods for noise removal exist including neighborhood averaging, linear Gaussian smoothing, median filtering, [34] or newer methods such as the Bacterial Foraging Optimization Algorithm. [35] [36]

Other challenges include

  • The fact that posed expressions, as used by most subjects of the various studies, are not natural, and therefore algorithms trained on these may not apply to natural expressions.
  • The lack of rotational movement freedom. Affect detection works very well with frontal use, but upon rotating the head more than 20 degrees, "there've been problems". [37]
  • Facial expressions do not always correspond to an underlying emotion that matches them (e.g. they can be posed or faked, or a person can feel emotions but maintain a "poker face").
  • FACS did not include dynamics, while dynamics can help disambiguate (e.g. smiles of genuine happiness tend to have different dynamics than "try to look happy" smiles.)
  • The FACS combinations do not correspond in a 1:1 way with the emotions that the psychologists originally proposed (note that this lack of a 1:1 mapping also occurs in speech recognition with homophones and homonyms and many other sources of ambiguity, and may be mitigated by bringing in other channels of information).
  • Accuracy of recognition is improved by adding context; however, adding context and other modalities increases computational cost and complexity

Body gesture

Gestures could be efficiently used as a means of detecting a particular emotional state of the user, especially when used in conjunction with speech and face recognition. Depending on the specific action, gestures could be simple reflexive responses, like lifting your shoulders when you don't know the answer to a question, or they could be complex and meaningful as when communicating with sign language. Without making use of any object or surrounding environment, we can wave our hands, clap or beckon. On the other hand, when using objects, we can point at them, move, touch or handle these. A computer should be able to recognize these, analyze the context and respond in a meaningful way, in order to be efficiently used for Human–Computer Interaction.

There are many proposed methods [38] to detect the body gesture. Some literature differentiates 2 different approaches in gesture recognition: a 3D model based and an appearance-based. [39] The foremost method makes use of 3D information of key elements of the body parts in order to obtain several important parameters, like palm position or joint angles. On the other hand, appearance-based systems use images or videos to for direct interpretation. Hand gestures have been a common focus of body gesture detection methods. [39]

Physiological monitoring

This could be used to detect a user's affective state by monitoring and analyzing their physiological signs. These signs range from changes in heart rate and skin conductance to minute contractions of the facial muscles and changes in facial blood flow. This area is gaining momentum and we are now seeing real products that implement the techniques. The four main physiological signs that are usually analyzed are blood volume pulse, galvanic skin response, facial electromyography, and facial color patterns.

Blood volume pulse

Overview

A subject's blood volume pulse (BVP) can be measured by a process called photoplethysmography, which produces a graph indicating blood flow through the extremities. [40] The peaks of the waves indicate a cardiac cycle where the heart has pumped blood to the extremities. If the subject experiences fear or is startled, their heart usually 'jumps' and beats quickly for some time, causing the amplitude of the cardiac cycle to increase. This can clearly be seen on a photoplethysmograph when the distance between the trough and the peak of the wave has decreased. As the subject calms down, and as the body's inner core expands, allowing more blood to flow back to the extremities, the cycle will return to normal.

Methodology

Infra-red light is shone on the skin by special sensor hardware, and the amount of light reflected is measured. The amount of reflected and transmitted light correlates to the BVP as light is absorbed by hemoglobin which is found richly in the bloodstream.

Disadvantages

It can be cumbersome to ensure that the sensor shining an infra-red light and monitoring the reflected light is always pointing at the same extremity, especially seeing as subjects often stretch and readjust their position while using a computer. There are other factors that can affect one's blood volume pulse. As it is a measure of blood flow through the extremities, if the subject feels hot, or particularly cold, then their body may allow more, or less, blood to flow to the extremities, all of this regardless of the subject's emotional state.

The corrugator supercilii muscle and zygomaticus major muscle are the 2 main muscles used for measuring the electrical activity, in facial electromyography. Em-face-2.png
The corrugator supercilii muscle and zygomaticus major muscle are the 2 main muscles used for measuring the electrical activity, in facial electromyography.

Facial electromyography

Facial electromyography is a technique used to measure the electrical activity of the facial muscles by amplifying the tiny electrical impulses that are generated by muscle fibers when they contract. [41] The face expresses a great deal of emotion, however, there are two main facial muscle groups that are usually studied to detect emotion: The corrugator supercilii muscle, also known as the 'frowning' muscle, draws the brow down into a frown, and therefore is the best test for negative, unpleasant emotional response.↵The zygomaticus major muscle is responsible for pulling the corners of the mouth back when you smile, and therefore is the muscle used to test for a positive emotional response.

Here we can see a plot of skin resistance measured using GSR and time whilst the subject played a video game. There are several peaks that are clear in the graph, which suggests that GSR is a good method of differentiating between an aroused and a non-aroused state. For example, at the start of the game where there is usually not much exciting game play, there is a high level of resistance recorded, which suggests a low level of conductivity and therefore less arousal. This is in clear contrast with the sudden trough where the player is killed as one is usually very stressed and tense as their character is killed in the game. Gsrplot.svg
Here we can see a plot of skin resistance measured using GSR and time whilst the subject played a video game. There are several peaks that are clear in the graph, which suggests that GSR is a good method of differentiating between an aroused and a non-aroused state. For example, at the start of the game where there is usually not much exciting game play, there is a high level of resistance recorded, which suggests a low level of conductivity and therefore less arousal. This is in clear contrast with the sudden trough where the player is killed as one is usually very stressed and tense as their character is killed in the game.

Galvanic skin response

Galvanic skin response (GSR) is an outdated term for a more general phenomenon known as [Electrodermal Activity] or EDA. EDA is a general phenomena whereby the skin's electrical properties change. The skin is innervated by the [sympathetic nervous system], so measuring its resistance or conductance provides a way to quantify small changes in the sympathetic branch of the autonomic nervous system. As the sweat glands are activated, even before the skin feels sweaty, the level of the EDA can be captured (usually using conductance) and used to discern small changes in autonomic arousal. The more aroused a subject is, the greater the skin conductance tends to be. [40]

Skin conductance is often measured using two small silver-silver chloride electrodes placed somewhere on the skin and applying a small voltage between them. To maximize comfort and reduce irritation the electrodes can be placed on the wrist, legs, or feet, which leaves the hands fully free for daily activity.

Facial color

Overview

The surface of the human face is innervated with a large network of blood vessels. Blood flow variations in these vessels yield visible color changes on the face. Whether or not facial emotions activate facial muscles, variations in blood flow, blood pressure, glucose levels, and other changes occur. Also, the facial color signal is independent from that provided by facial muscle movements. [42]

Methodology

Approaches are based on facial color changes. Delaunay triangulation is used to create the triangular local areas. Some of these triangles which define the interior of the mouth and eyes (sclera and iris) are removed. Use the left triangular areas’ pixels to create feature vectors. [42] It shows that converting the pixel color of the standard RGB color space to a color space such as oRGB color space [43] or LMS channels perform better when dealing with faces. [44] So, map the above vector onto the better color space and decompose into red-green and yellow-blue channels. Then use deep learning methods to find equivalent emotions.

Visual aesthetics

Aesthetics, in the world of art and photography, refers to the principles of the nature and appreciation of beauty. Judging beauty and other aesthetic qualities is a highly subjective task. Computer scientists at Penn State treat the challenge of automatically inferring the aesthetic quality of pictures using their visual content as a machine learning problem, with a peer-rated on-line photo sharing website as a data source. [45] They extract certain visual features based on the intuition that they can discriminate between aesthetically pleasing and displeasing images.

Potential applications

Education

Affection influences learners' learning state. Using affective computing technology, computers can judge the learners' affection and learning state by recognizing their facial expressions. In education, the teacher can use the analysis result to understand the student's learning and accepting ability, and then formulate reasonable teaching plans. At the same time, they can pay attention to students' inner feelings, which is helpful to students' psychological health. Especially in distance education, due to the separation of time and space, there is no emotional incentive between teachers and students for two-way communication. Without the atmosphere brought by traditional classroom learning, students are easily bored, and affect the learning effect. Applying affective computing in distance education system can effectively improve this situation. [46]

Transportation

The applications of sensory computing may contribute to improving road safety. For example, a car can monitor the emotion of all occupants and engage in additional safety measures, such as alerting other vehicles if it detects the driver to be angry. [47] In addition, affective computing systems for monitoring the driver's stress may allow various interventions such as driver assistance systems adjusted according to the stress level [48] and minimal and direct interventions to change the emotional state of the driver. [49]

Healthcare

Social robots, as well as a growing number of robots used in health care benefit from emotional awareness because they can better judge users' and patient's emotional states and alter their actions/programming appropriately. This is especially important in those countries with growing aging populations and/or a lack of younger workers to address their needs. [50]

Affective computing is also being applied to the development of communicative technologies for use by people with autism. [51] The affective component of a text is also increasingly gaining attention, particularly its role in the so-called emotional or emotive Internet. [52]

Video games

Affective video games can access their players' emotional states through biofeedback devices. [53] A particularly simple form of biofeedback is available through gamepads that measure the pressure with which a button is pressed: this has been shown to correlate strongly with the players' level of arousal; [54] at the other end of the scale are brain–computer interfaces. [55] [56] Affective games have been used in medical research to support the emotional development of autistic children. [57]

Psychomotor training

Training methods of psychomotor operations such as steering and maneuvering are used in various fields such as aviation, transportation and medicine. Integrating affective computing capabilities in this type of training systems, in accordance with the adaptive automation approach, has been found to be effective in improving the quality of training and shortening the required training duration. [58]

Other applications

Affective computing has potential applications in human–computer interaction, such as affective mirrors allowing the user to see how he or she performs; emotion monitoring agents sending a warning before one sends an angry email; or even music players selecting tracks based on mood. [59]

One idea put forth by the Romanian researcher Dr. Nicu Sebe in an interview is the analysis of a person's face while they are using a certain product (he mentioned ice cream as an example). [60] Companies would then be able to use such analysis to infer whether their product will or will not be well received by the respective market.

One could also use affective state recognition in order to judge the impact of a TV advertisement through a real-time video recording of that person and through the subsequent study of his or her facial expression. Averaging the results obtained on a large group of subjects, one can tell whether that commercial (or movie) has the desired effect and what the elements which interest the watcher most are.

Cognitivist vs. interactional approaches

Within the field of human–computer interaction, Rosalind Picard's cognitivist or "information model" concept of emotion has been criticized by and contrasted with the "post-cognitivist" or "interactional" pragmatist approach taken by Kirsten Boehner and others which views emotion as inherently social. [61]

Picard's focus is human–computer interaction, and her goal for affective computing is to "give computers the ability to recognize, express, and in some cases, 'have' emotions". [4] In contrast, the interactional approach seeks to help "people to understand and experience their own emotions" [62] and to improve computer-mediated interpersonal communication. It does not necessarily seek to map emotion into an objective mathematical model for machine interpretation, but rather let humans make sense of each other's emotional expressions in open-ended ways that might be ambiguous, subjective, and sensitive to context. [62] : 284 [ example needed ]

Picard's critics describe her concept of emotion as "objective, internal, private, and mechanistic". They say it reduces emotion to a discrete psychological signal occurring inside the body that can be measured and which is an input to cognition, undercutting the complexity of emotional experience. [62] : 280  [62] : 278 

The interactional approach asserts that though emotion has biophysical aspects, it is "culturally grounded, dynamically experienced, and to some degree constructed in action and interaction". [62] : 276  Put another way, it considers "emotion as a social and cultural product experienced through our interactions". [63] [62] [64]

See also

Related Research Articles

Computer vision tasks include methods for acquiring, processing, analyzing and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the forms of decisions. Understanding in this context means the transformation of visual images into descriptions of the world that make sense to thought processes and can elicit appropriate action. This image understanding can be seen as the disentangling of symbolic information from image data using models constructed with the aid of geometry, physics, statistics, and learning theory.

Facial expression is the motion and positioning of the muscles beneath the skin of the face. These movements convey the emotional state of an individual to observers and are a form of nonverbal communication. They are a primary means of conveying social information between humans, but they also occur in most other mammals and some other animal species.

<span class="mw-page-title-main">Kismet (robot)</span>

Kismet is a robot head which was made in the 1990s at Massachusetts Institute of Technology (MIT) by Dr. Cynthia Breazeal as an experiment in affective computing; a machine that can recognize and simulate emotions. The name Kismet comes from a Turkish word meaning "fate" or sometimes "luck".

<span class="mw-page-title-main">Face detection</span> Identification of human faces in images

Face detection is a computer technology being used in a variety of applications that identifies human faces in digital images. Face detection also refers to the psychological process by which humans locate and attend to faces in a visual scene.

<span class="mw-page-title-main">Gesture recognition</span> Topic in computer science and language technology

Gesture recognition is an area of research and development in computer science and language technology concerned with the recognition and interpretation of human gestures. A subdiscipline of computer vision, it employs mathematical algorithms to interpret gestures.

<span class="mw-page-title-main">Facial Action Coding System</span> System of classifying human facial movements

The Facial Action Coding System (FACS) is a system to taxonomize human facial movements by their appearance on the face, based on a system originally developed by a Swedish anatomist named Carl-Herman Hjortsjö. It was later adopted by Paul Ekman and Wallace V. Friesen, and published in 1978. Ekman, Friesen, and Joseph C. Hager published a significant update to FACS in 2002. Movements of individual facial muscles are encoded by the FACS from slight different instant changes in facial appearance. It has proven useful to psychologists and to animators.

Multimodal interaction provides the user with multiple modes of interacting with a system. A multimodal interface provides several distinct tools for input and output of data.

Affective science is the scientific study of emotion or affect. This includes the study of emotion elicitation, emotional experience and the recognition of emotions in others. Of particular relevance are the nature of feeling, mood, emotionally-driven behaviour, decision-making, attention and self-regulation, as well as the underlying physiology and neuroscience of the emotions.

Implicit data collection is used in human–computer interaction to gather data about the user in an implicit, non-invasive way.

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

<span class="mw-page-title-main">Human–computer interaction</span> Academic discipline studying the relationship between computer systems and their users

Human–computer interaction (HCI) is research in the design and the use of computer technology, which focuses on the interfaces between people (users) and computers. HCI researchers observe the ways humans interact with computers and design technologies that allow humans to interact with computers in novel ways. A device that allows interaction between human being and a computer is known as a "Human-computer Interface (HCI)".

Affective haptics is an area of research which focuses on the study and design of devices and systems that can elicit, enhance, or influence the emotional state of a human by means of sense of touch. The research field is originated with the Dzmitry Tsetserukou and Alena Neviarouskaya papers on affective haptics and real-time communication system with rich emotional and haptic channels. Driven by the motivation to enhance social interactivity and emotionally immersive experience of users of real-time messaging, virtual, augmented realities, the idea of reinforcing (intensifying) own feelings and reproducing (simulating) the emotions felt by the partner was proposed. Four basic haptic (tactile) channels governing our emotions can be distinguished:

  1. physiological changes
  2. physical stimulation
  3. social touch
  4. emotional haptic design.

The normal way that a computer functions manually is through a person that controls the computer. An individual generates computer actions with the use of either a computer mouse or keyboard. However the latest technology and computer innovation might allow a computer to not only detect body language but also respond to it. Modern devices are being experimented with, that may potentially allow that computer related device to respond to and understand an individual's hand gesture, specific movement or facial expression.

Emotions in virtual communication are expressed and understood in a variety of different ways from those in face-to-face interactions. Virtual communication continues to evolve as technological advances emerge that give way to new possibilities in computer-mediated communication (CMC). The lack of typical auditory and visual cues associated with human emotion gives rise to alternative forms of emotional expression that are cohesive with many different virtual environments. Some environments provide only space for text based communication, where emotions can only be expressed using words. More newly developed forms of expression provide users the opportunity to portray their emotions using images.

Emotion recognition is the process of identifying human emotion. People vary widely in their accuracy at recognizing the emotions of others. Use of technology to help people with emotion recognition is a relatively nascent research area. Generally, the technology works best if it uses multiple modalities in context. To date, the most work has been conducted on automating the recognition of facial expressions from video, spoken expressions from audio, written expressions from text, and physiology as measured by wearables.

Artificial empathy or computational empathy is the development of AI systems—such as companion robots or virtual agents—that can detect emotions and respond to them in an empathic way.

Multimodal sentiment analysis is a technology for traditional text-based sentiment analysis, which includes modalities such as audio and visual data. It can be bimodal, which includes different combinations of two modalities, or trimodal, which incorporates three modalities. With the extensive amount of social media data available online in different forms such as videos and images, the conventional text-based sentiment analysis has evolved into more complex models of multimodal sentiment analysis, which can be applied in the development of virtual assistants, analysis of YouTube movie reviews, analysis of news videos, and emotion recognition such as depression monitoring, among others.

Automated Pain Recognition (APR) is a method for objectively measuring pain and at the same time represents an interdisciplinary research area that comprises elements of medicine, psychology, psychobiology, and computer science. The focus is on computer-aided objective recognition of pain, implemented on the basis of machine learning.

Hatice Gunes is a Turkish computer scientist who is Professor of Affective Intelligence & Robotics at the University of Cambridge. Gunes leads the Affective Intelligence & Robotics Lab. Her research considers human robot interactions and the development of sophisticated technologies with emotional intelligence.

References

Citations

  1. ↑ Tao, Jianhua; Tieniu Tan (2005). "Affective Computing: A Review". Affective Computing and Intelligent Interaction. Vol.  LNCS 3784. Springer. pp. 981–995. doi:10.1007/11573548.
  2. ↑ James, William (1884). "What Is Emotion". Mind. 9 (34): 188–205. doi:10.1093/mind/os-IX.34.188. Cited by Tao and Tan.
  3. ↑ "Affective Computing" MIT Technical Report #321 (Abstract), 1995
  4. 1 2 Picard, Rosalind (1997). Affective Computing. Cambridge, MA: MIT Press. p. 1.
  5. ↑ Kleine-Cosack, Christian (October 2006). "Recognition and Simulation of Emotions" (PDF). Archived from the original (PDF) on May 28, 2008. Retrieved May 13, 2008. The introduction of emotion to computer science was done by Pickard (sic) who created the field of affective computing.
  6. ↑ Diamond, David (December 2003). "The Love Machine; Building computers that care". Wired. Archived from the original on 18 May 2008. Retrieved May 13, 2008. Rosalind Picard, a genial MIT professor, is the field's godmother; her 1997 book, Affective Computing, triggered an explosion of interest in the emotional side of computers and their users.
  7. ↑ Garay, Nestor; Idoia Cearreta; Juan Miguel López; Inmaculada Fajardo (April 2006). "Assistive Technology and Affective Mediation" (PDF). Human Technology . 2 (1): 55–83. doi: 10.17011/ht/urn.2006159 . Archived (PDF) from the original on 28 May 2008. Retrieved 2008-05-12.
  8. ↑ Heise, David (2004). "Enculturating agents with expressive role behavior". In Sabine Payr; Trappl, Robert (eds.). Agent Culture: Human-Agent Interaction in a Mutlicultural World. Lawrence Erlbaum Associates. pp. 127–142.
  9. ↑ Restak, Richard (2006-12-17). "Mind Over Matter". The Washington Post. Retrieved 2008-05-13.
  10. ↑ Loveys, Kate; Sagar, Mark; Broadbent, Elizabeth (2020-07-22). "The Effect of Multimodal Emotional Expression on Responses to a Digital Human during a Self-Disclosure Conversation: a Computational Analysis of User Language". Journal of Medical Systems. 44 (9): 143. doi:10.1007/s10916-020-01624-4. ISSN   0148-5598. PMID   32700060. S2CID   220717084.
  11. ↑ Ho, Manh-Tung (29 March 2023). "An analytical framework for studying attitude towards emotional AI: The three-pronged approach". MethodsX . 10 (102149). doi:10.1016/j.mex.2023.102149. PMC   10113835 . PMID   37091958.
  12. ↑ Aleix, and Shichuan Du, Martinez (2012). "A model of the perception of facial expressions of emotion by humans: Research overview and perspectives" (PDF). The Journal of Machine Learning Research. 13 (1): 1589–1608.
  13. ↑ Breazeal, Cynthia; Aryananda, Lijin (2002). "Recognition of Affective Communicative Intent in Robot-Directed Speech" (PDF). Autonomous Robots. 12 (1). Springer: 83–104. doi:10.1023/a:1013215010749. ISSN   0929-5593. S2CID   459892.
  14. 1 2 3 Dellaert, F., Polizin, t., and Waibel, A., Recognizing Emotion in Speech", In Proc. Of ICSLP 1996, Philadelphia, PA, pp.1970–1973, 1996
  15. ↑ Roy, D.; Pentland, A. (1996-10-01). "Automatic spoken affect classification and analysis". Proceedings of the Second International Conference on Automatic Face and Gesture Recognition. pp. 363–367. doi:10.1109/AFGR.1996.557292. ISBN   978-0-8186-7713-7. S2CID   23157273.
  16. ↑ Lee, C.M.; Narayanan, S.; Pieraccini, R., Recognition of Negative Emotion in the Human Speech Signals, Workshop on Auto. Speech Recognition and Understanding, Dec 2001
  17. ↑ Neiberg, D; Elenius, K; Laskowski, K (2006). "Emotion recognition in spontaneous speech using GMMs" (PDF). Proceedings of Interspeech. doi:10.21437/Interspeech.2006-277. S2CID   5790745.
  18. ↑ Yacoub, Sherif; Simske, Steve; Lin, Xiaofan; Burns, John (2003). "Recognition of Emotions in Interactive Voice Response Systems". Proceedings of Eurospeech: 729–732. CiteSeerX   10.1.1.420.8158 . doi:10.21437/Eurospeech.2003-307. S2CID   11671944.
  19. 1 2 Hudlicka 2003 , p. 24
  20. ↑ Hudlicka 2003 , p. 25
  21. ↑ Charles Osgood; William May; Murray Miron (1975). Cross-Cultural Universals of Affective Meaning . Univ. of Illinois Press. ISBN   978-94-007-5069-2.
  22. 1 2 Scherer, Bänziger & Roesch 2010 , p. 241
  23. ↑ "Gaussian Mixture Model". Connexions – Sharing Knowledge and Building Communities. Retrieved 10 March 2011.
  24. ↑ S.E. Khoruzhnikov; et al. (2014). "Extended speech emotion recognition and prediction". Scientific and Technical Journal of Information Technologies, Mechanics and Optics. 14 (6): 137.
  25. 1 2 Ekman, P. & Friesen, W. V (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1, 49–98.
  26. 1 2 Steidl, Stefan (5 March 2011). "FAU Aibo Emotion Corpus". Pattern Recognition Lab.
  27. 1 2 Scherer, Bänziger & Roesch 2010 , p. 243
  28. ↑ Singh, Premjeet; Saha, Goutam; Sahidullah, Md (2021). "Non-linear frequency warping using constant-Q transformation for speech emotion recognition". 2021 International Conference on Computer Communication and Informatics (ICCCI). pp. 1–4. arXiv: 2102.04029 . doi:10.1109/ICCCI50826.2021.9402569. ISBN   978-1-7281-5875-4. S2CID   231846518.
  29. ↑ Caridakis, G.; Malatesta, L.; Kessous, L.; Amir, N.; Raouzaiou, A.; Karpouzis, K. (November 2–4, 2006). Modeling naturalistic affective states via facial and vocal expressions recognition. International Conference on Multimodal Interfaces (ICMI'06). Banff, Alberta, Canada.
  30. ↑ Balomenos, T.; Raouzaiou, A.; Ioannou, S.; Drosopoulos, A.; Karpouzis, K.; Kollias, S. (2004). "Emotion Analysis in Man-Machine Interaction Systems". In Bengio, Samy; Bourlard, Herve (eds.). Machine Learning for Multimodal Interaction. Lecture Notes in Computer Science. Vol. 3361. Springer-Verlag. pp. 318–328.
  31. ↑ Ekman, Paul (1972). Cole, J. (ed.). Universals and Cultural Differences in Facial Expression of Emotion. Nebraska Symposium on Motivation. Lincoln, Nebraska: University of Nebraska Press. pp. 207–283.
  32. ↑ Ekman, Paul (1999). "Basic Emotions". In Dalgleish, T; Power, M (eds.). Handbook of Cognition and Emotion (PDF). Sussex, UK: John Wiley & Sons. Archived from the original (PDF) on 2010-12-28..
  33. ↑ "Facial Action Coding System (FACS) and the FACS Manual" Archived October 19, 2013, at the Wayback Machine . A Human Face. Retrieved 21 March 2011.
  34. ↑ "Spatial domain methods".
  35. ↑ Clever Algorithms. "Bacterial Foraging Optimization Algorithm – Swarm Algorithms – Clever Algorithms" Archived 2019-06-12 at the Wayback Machine . Clever Algorithms. Retrieved 21 March 2011.
  36. ↑ "Soft Computing". Soft Computing. Retrieved 18 March 2011.
  37. ↑ Williams, Mark. "Better Face-Recognition Software – Technology Review" Archived 2011-06-08 at the Wayback Machine . Technology Review: The Authority on the Future of Technology. Retrieved 21 March 2011.
  38. ↑ J. K. Aggarwal, Q. Cai, Human Motion Analysis: A Review, Computer Vision and Image Understanding, Vol. 73, No. 3, 1999
  39. 1 2 Pavlovic, Vladimir I.; Sharma, Rajeev; Huang, Thomas S. (1997). "Visual Interpretation of Hand Gestures for Human–Computer Interaction: A Review" (PDF). IEEE Transactions on Pattern Analysis and Machine Intelligence . 19 (7): 677–695. doi:10.1109/34.598226. S2CID   7185733.
  40. 1 2 Picard, Rosalind (1998). Affective Computing. MIT.
  41. ↑ Larsen JT, Norris CJ, Cacioppo JT, "Effects of positive and negative affect on electromyographic activity over zygomaticus major and corrugator supercilii", (September 2003)
  42. 1 2 Benitez-Quiroz, Carlos F.; Srinivasan, Ramprakash; Martinez, Aleix M. (2018-03-19). "Facial color is an efficient mechanism to visually transmit emotion". Proceedings of the National Academy of Sciences. 115 (14): 3581–3586. Bibcode:2018PNAS..115.3581B. doi: 10.1073/pnas.1716084115 . PMC   5889636 . PMID   29555780.
  43. ↑ Bratkova, Margarita; Boulos, Solomon; Shirley, Peter (2009). "oRGB: A Practical Opponent Color Space for Computer Graphics". IEEE Computer Graphics and Applications. 29 (1): 42–55. doi:10.1109/mcg.2009.13. PMID   19363957. S2CID   16690341.
  44. ↑ Hadas Shahar, Hagit Hel-Or, Micro Expression Classification using Facial Color and Deep Learning Methods, The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 0–0.
  45. ↑ Ritendra Datta, Dhiraj Joshi, Jia Li and James Z. Wang, Studying Aesthetics in Photographic Images Using a Computational Approach, Lecture Notes in Computer Science, vol. 3953, Proceedings of the European Conference on Computer Vision, Part III, pp. 288–301, Graz, Austria, May 2006.
  46. ↑ Wu, Chih-Hung; Huang, Yueh-Min; Hwang, Jan-Pan (November 2016). "Review of affective computing in education/Learning: Trends and challenges". British Journal of Educational Technology. 47 (6): 1304–1323. doi:10.1111/bjet.12324.
  47. ↑ "In-Car Facial Recognition Detects Angry Drivers To Prevent Road Rage". Gizmodo. 30 August 2018.
  48. ↑ Collet, Christian; Musicant, Oren (2019-04-24). "Associating Vehicles Automation With Drivers Functional State Assessment Systems: A Challenge for Road Safety in the Future". Frontiers in Human Neuroscience. 13: 131. doi: 10.3389/fnhum.2019.00131 . ISSN   1662-5161. PMC   6503868 . PMID   31114489.
  49. ↑ Balters, Stephanie; Bernstein, Madeline; Paredes, Pablo E. (2019-05-02). "On-road Stress Analysis for In-car Interventions During the Commute". Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM. pp. 1–6. doi:10.1145/3290607.3312824. ISBN   978-1-4503-5971-9. S2CID   144207824.
  50. ↑ Yonck, Richard (2017). Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence. New York: Arcade Publishing. pp. 150–153. ISBN   9781628727333. OCLC   956349457.
  51. ↑ Projects in Affective Computing
  52. ↑ Shanahan, James; Qu, Yan; Wiebe, Janyce (2006). Computing Attitude and Affect in Text: Theory and Applications. Dordrecht: Springer Science & Business Media. p. 94. ISBN   1402040261
  53. ↑ Gilleade, Kiel Mark; Dix, Alan; Allanson, Jen (2005). Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me (PDF). Proc. DiGRA Conf. Archived from the original (PDF) on 2015-04-06. Retrieved 2016-12-10.
  54. ↑ Sykes, Jonathan; Brown, Simon (2003). Affective gaming: Measuring emotion through the gamepad. CHI '03 Extended Abstracts on Human Factors in Computing Systems. CiteSeerX   10.1.1.92.2123 . doi:10.1145/765891.765957. ISBN   1581136374.
  55. ↑ Nijholt, Anton; Plass-Oude Bos, Danny; Reuderink, Boris (2009). "Turning shortcomings into challenges: Brain–computer interfaces for games" (PDF). Entertainment Computing. 1 (2): 85–94. Bibcode:2009itie.conf..153N. doi:10.1016/j.entcom.2009.09.007.
  56. ↑ Reuderink, Boris; Nijholt, Anton; Poel, Mannes (2009). Affective Pacman: A Frustrating Game for Brain–Computer Interface Experiments. Intelligent Technologies for Interactive Entertainment (INTETAIN). pp. 221–227. doi:10.1007/978-3-642-02315-6_23. ISBN   978-3-642-02314-9.
  57. ↑ Khandaker, M (2009). "Designing affective video games to support the social-emotional development of teenagers with autism spectrum disorders". Studies in Health Technology and Informatics. 144: 37–9. PMID   19592726.
  58. ↑ Sahar, Yotam; Wagner, Michael; Barel, Ariel; Shoval, Shraga (2022-11-01). "Stress-Adaptive Training: An Adaptive Psychomotor Training According to Stress Measured by Grip Force". Sensors. 22 (21): 8368. Bibcode:2022Senso..22.8368S. doi: 10.3390/s22218368 . ISSN   1424-8220. PMC   9654132 . PMID   36366066.
  59. ↑ Janssen, Joris H.; van den Broek, Egon L. (July 2012). "Tune in to Your Emotions: A Robust Personalized Affective Music Player". User Modeling and User-Adapted Interaction. 22 (3): 255–279. doi: 10.1007/s11257-011-9107-7 . hdl: 2066/103051 .
  60. ↑ "Mona Lisa: Smiling? Computer Scientists Develop Software That Evaluates Facial Expressions". ScienceDaily. 1 August 2006. Archived from the original on 19 October 2007.
  61. ↑ Battarbee, Katja; Koskinen, Ilpo (2005). "Co-experience: user experience as interaction" (PDF). CoDesign. 1 (1): 5–18. CiteSeerX   10.1.1.294.9178 . doi:10.1080/15710880412331289917. S2CID   15296236. Archived from the original (PDF) on 2017-12-14. Retrieved 2016-02-02.
  62. 1 2 3 4 5 6 Boehner, Kirsten; DePaula, Rogerio; Dourish, Paul; Sengers, Phoebe (2007). "How emotion is made and measured". International Journal of Human–Computer Studies. 65 (4): 275–291. doi:10.1016/j.ijhcs.2006.11.016. S2CID   15551492.
  63. ↑ Boehner, Kirsten; DePaula, Rogerio; Dourish, Paul; Sengers, Phoebe (2005). "Affection: From Information to Interaction". Proceedings of the Aarhus Decennial Conference on Critical Computing: 59–68.
  64. ↑ Hook, Kristina; Staahl, Anna; Sundstrom, Petra; Laaksolahti, Jarmo (2008). "Interactional empowerment" (PDF). Proc. CHI: 647–656.

Works cited

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.