Decoding Emotions

There are two distinct perspectives on the nature of emotions. One view is that they are universal—and universally recognizable—responses that are automatically triggered by events and circumstances. The other view is that they are constructed by each of us moment-to-moment, based on our particular combination of physiology, temperament, and experience.

Charles Darwin’s The Expression of the Emotions in Man and Animals, published in 1872, wasn’t the first book to attempt to codify facial expressions, but it still serves as inspiration for what has been referred to by Lisa Feldman Barrett as the classical view of emotion.

[H]e claimed that emotions and their expressions were an ancient part of universal human nature. All people, everywhere in the world are said to exhibit and recognize facial expressions of emotions without any training whatsoever.

This idea was further advanced in the 1960s and 70s, by psychologist Paul Ekman and others, and remains a widely held view. It’s why many people believe there are six basic emotions—fear, anger, joy, sadness, disgust, and surprise—that are recognized by people across the globe. And it’s why the use of psychological tests for recognizing emotions from facial photographs is still quite common.

Furthermore the ability to identify emotions from facial expressions is a standard test of empathy, with implications in diagnosing various mental health conditions. So it’s relatively safe to say that a belief in empathy is based in the classical view of emotions.

But as Feldman Barrett points out:

Hundreds of experiments have shown that people worldwide can match emotion words to so-called expressions of emotion, posed by actors who aren’t actually feeling those emotions. However, those expressions can’t be consistently and specifically detected by objective measures of facial muscle movements when people are actually feeling emotion.

So if most (if not all) of the people who have been photographed for emotion detection tests are…emoting (i.e., acting), what do these tests actually reveal? At best, they appear to reveal some consensus agreement as to what those facial expressions are supposed to represent—stereotypes of emotions, if you will.

A Little Context, Please

Even if the photographs were not posed, focusing exclusively on faces excludes all the relevant contextual detail that the brain uses to form impressions and make interpretations.

Other things, including body movement, personality, tone of voice and changes in skin tone have important roles in how we perceive and display emotion. —Douglas Heaven, Nature, 2/26/20

Surrounding environmental context is even more critical. In a PNAS paper published on 3/21/19, Aleix Martinez offers the following images to support the importance of contextual information.

When asked to identify the emotions shown in these images, most people agree that the left image expresses sadness, while the right image is a clear display of anger. If asked whether these expressions communicate positive or negative valence, most people agree that both correspond to a negative expression. The problem with these assessments is that context is not observable, which may lead to incorrect interpretations. Images courtesy of (Left) Imgflip and (Right) Getty Images/Michael Steele.

Adding context to the facial expressions previously seen in [the previous figure] radically changes our interpretation of the emotion being experienced by a person. (A and B) In these two images, most observers agree that the people shown are experiencing a joyful event (i.e., positive valence). (C and D) When the face and body are blurred out, inference of valence and arousal is still possible. Images courtesy of (Upper Left, Lower Left, and Lower Right) Imgflip and (Upper Right) Getty Images/Michael Steele.

Martinez concludes that:

[w]e still do not know which brain regions of interest (ROIs) are involved in the recognition of affective context and how these communicate with already known areas. ROIs for the recognition of facial muscle articulations and biological motion as well as affective variables have been identified. It is logical to assume that ROIs involved in visual analysis of scenes, objects, and bodies are part of the visual interpretation of context. But does the brain employ other mechanisms to perform this inference? And, how do all these ROIs interact with one another to create the conscious affective percept we all experience?

How Many Words Is a Picture Worth?

Contextual information can not only lead to more accurate interpretations, it can also turn the tables on emotion recognition altogether. In writing about rethinking emotional intelligence a few years ago, I included this excerpt from The Storytelling Animal by Jonathan Gottschall:

In the early twentieth century, the Russian filmmaker Lev Kuleshov produced a film of unnarrated images: a corpse in a coffin, a lovely young woman, and a bowl of soup. In between these images, Kuleshov squeezed shots of an actor’s face. The audience noted that when the soup was shown, the actor emoted hunger. When the corpse was shown, he looked sad. When the lovely young woman appeared, the actor’s face was transformed by lust.

In fact the actor wasn’t emoting at all. After every shot, Kuleshov had inserted exactly the same footage of an actor staring impassively into the camera.

We often see what we expect to see rather than what’s actually there. Expectations play an enormous role in the interpretations made by the brain. So does our experience, as well as the language we speak and the range of our emotional vocabulary.

This applies to the interpretation of our own emotions as well as the emotions of others. When it comes to interpreting our own emotions, we don’t usually go about it by looking at ourselves in a mirror. We’re more likely to focus on internal cues first. But no matter how strong those cues are, they do require interpretation. For example, excitement and anxiety are what are called high-arousal states that have similar neurological and physiological symptoms, such as increased heart-rate, restlessness, rapid breathing, difficulty concentrating, and nervousness or tension. So context (the circumstances surrounding a particular high-arousal state) plays a role in our process of interpretation. Nevertheless, some of us are more inclined in general to interpret this set of sensations as anxiety and some of us are more inclined to interpret it as excitement.

Our personal interpretive tendencies are very likely to influence our interpretations of the emotions others are experiencing. Given what we understand about how the brain works, it’s hard to imagine how they wouldn’t.

Take Away

The idea that emotions are constitutive (constructed) has gained more and more ground as neuroscience research expands our understanding of how we function. There are so many things we take for granted, or at face value, without questioning them. The idea that there are six basic emotions that are universally recognized is an intuitive get. But there was actually substantial disagreement with the classical view of emotion prior to the 1960s; it just got swept under the rug. It wasn’t the first scientific concept that got swept aside and won’t be the last.

Emotions are real just as a sense of self is real, but neither an emotion nor a sense of self is a thing. Instead both are processes, continually in flux, a result of our interpretation—and open to our interpretation.