“it’s all in the eyes” and other lies: a critique on contemporary emotion research

felt and unfelt smile poll - authentic - genuine - inauthentic

the sisyphean reality of smiling

As casualties of Western media and polite positivity, social smiling has become an expected component of our day-to-day interactions. We smile for yes; we smile for no. We smile for hello, goodbye, please, and thank you. We rinse. We repeat.

Sisyphus and the smiley

These expectations do not leave us once we are behind closed doors. They extend beyond in-person niceties and nestle into our online personas. Static images of our faces regularly guide first impressions from peers and strangers alike. Whether it’s the business-casual beam for LinkedIn, the group-photo-grin for Facebook, or the carefree crinkle for Bumble – your smile is being judged, online and offline.

authenticity vs. perception

Because smiling is such a critical component of our social interactions, not just any lip corner levitator will make the cut. How others perceive our authenticity is essential. Our ability to assess emotional sincerity is not simply important for social survival but for survival in general. While in most cases, miscalculating the genuineness of a pleasantry is inconsequential, in some cases it can put us in life-threatening situations.

Many factors play a role in how we appraise the authenticity behind a smile, but how reliable are those factors?

putting it to the test

A few months ago, I posted two photos of myself smiling – one deliberately posed/non-response-based** and one spontaneous/response-based**. I asked audience members on all social media platforms which they believed to be the authentic smile and which they believed to be the posed smile.

I received over 170 responses; 85% of them were wrong.

Photo prompt from post shown below. See original post here.


felt and unfelt smile poll - authentic - genuine - inauthentic


 Answer revealed at end of post.

NOTE: While 90 responses came from platforms where other voter answers were visible, 80+ responses came from platforms where voters could not see prior responses. Regardless of platform and voter visibility, the 85% trend remained stable.

TERMS & DEFINITIONS

Because the word “authentic” can be interpreted in various ways, it is important to align reader definition with writer intent. I am thus defining my terms as such:

authentic / unposed / felt: Facial expression was not consciously forced. Expression was a spontaneous reaction to an internal or external stimulus. Resulting expression was evaluated – by expresser (person experiencing reaction / expression) – to be reflective of expresser’s internal affective experience.

inauthentic / posed / unfelt: Facial expression was deliberately forced in order to imitate an affective response and deceive an audience. Expresser’s internal affective state and external expression were dissonant. Expresser did not feel the resulting posed expression was reflective of their internal affective experience.
 
smile: The facial expression (either posed or unposed) featuring elevated lip corners. Typically generated by the activation of the zygomaticus major muscle. May or may not be accompanied by other facial muscles.

making sense of the confusion

So, how did this happen? How did 85% of voters get it wrong?
 

The purpose of this test was to demonstrate the fallibility of our assumptions on facial expressions of emotion – and it did just that. Had I included video clips rather than photos, I am certain the audience would have performed much better; however, it is important to note that still images currently serve as the backbone for an alarming portion of emotion research and technology. Emotions are full of intricacies and nuances that, even with video footage and motion, we struggle to truly understand.

emotions & facial expressions

Emotion research centered on facial expressions has been around for a good number of decades – centuries even, if you consider early contributors, like Charles Darwin. Regardless of where you begin your timeline, the mainstream foundation underlying our present understanding of the face-to-emotion relationship comes largely from psychologist and father of the Facial Action Coding System: Paul Ekman.
 
While Ekman’s research and writings have inspired many of us (myself included) – tech, academia, and entertainment have become too reliant on the work of this single contributor and his followers. The overwhelming majority of companies I’ve worked for or contracted with have either been wholly blind to and/or disinterested in alternative branches of thought. Being that I have worked for Big Tech and influential emotion tracking startups: This is unsettling.
 
With the progression of tracking technology and the increasing demand for realism in digital art, it’s beyond time that these industries (and likely many others) adopted a more comprehensive approach to understanding emotions and facial expressions.

take big claims with a grain of salt

salt shaker shaking emoji salts

While I still hold Ekman’s work with high regard, I have also learned to take a lot of claims on “true” emotion with a grain of salt. As with many big questions in science – it depends.
 
We need to move forward in time with new research and begin integrating more theories, approaches, and critiques from other sources into our products and practices. We need to stop pushing Ekman’s emotion research as unchallenged gospel and start pulling from other sources.
 
Ekman’s work was revolutionary for its time and remains extraordinarily useful; however, it is dated, incomplete, and regarded by numerous leaders in modern emotion research as highly subject to bias. Yes, it is certainly nice to believe we can reliably determine genuine emotion with rules like: “eye contraction in a smile makes it a true expression of joy” or “genuine expressions are more symmetric,” but it’s just not that simple.

orbicularis ocu-lies

In the Ekman and mainstream pools of thought, it is said that in order for a smile to be genuine and positive in affect, the orbital portion of “orbicularis oculi” (a sphincter-like muscle in the eye region) must be contracted in conjunction with zygomaticus major (the muscle typically used for mouth smiling). Such eye + mouth combo smiles are referred to as “Duchenne smiles.
 

Because of the popularization of the Duchenne smile, certain concepts tend to get parroted around in pop culture and the emotion/nonverbal behavior world:

    • The truth is in the eyes!
    • Real smiles show in the eyes!
It is true that many spontaneous expressions of joy, amusement, and other positive, smile-based expressions are likely to feature orbital eye contractions. However, it is also true that many unfelt and posed smiles feature this same muscle activation, while some felt and unposed smiles do not. In fact, recent studies revisiting the ideas behind felt vs. unfelt smiles have found that presence of orbicularis oculi contraction did not reliably predict felt-ness. Inferring emotion is not a matter of simple binary approaches.

In a large sample of spontaneous (i.e., non- posed) smiles, we found that knowing whether a smile included the Duchenne marker added very little new information about both self-reported positive emotion and observer-rated positive emotion when smile intensity was already known. Girard et al. 2020

Additionally, when considering the diversity of facial anatomy – the look of an orbital contraction is highly dependent on factors such as:
    • the shape and size of orbicularis oculi
    • the shape, thickness, length, and orientation of zygomaticus major
    • the spatial relationship between orbicularis oculi, zygomaticus major, and other surrounding muscles
    • facial fat content and distribution
    • dynamic wrinkle presence
    • age
    • the presence of additional muscles interfering with the orbital region
Melinda Ozel lecture series on advanced anatomy - orbicularis oculi and zygomaticus major overlap examples - abridged
abridged slide from my diverse expression & anatomy lecture series geared toward companies focused on photorealistic characters and diverse anatomy

In some people, orbicularis oculi and zygomaticus major muscles overlap; in others, they do not. There are also lesser-known muscles that inherently affect smile-based eye-mouth interactions such as the:

    • medial malaris muscle
    • orbitozygomatic muscle

These muscles are often omitted from anatomy textbooks due to their inconsistent presence in Europeans; they also remain unaddressed in emotion research centered on facial expressions. Such omissions have left us with large gaps in knowledge and a white European-leaning bias in a significant portion of Ekman-based emotion research. Variation in appearance affects detectability and facial coder reliability, rendering the methods of countless studies on spontaneous and deliberate smiles flawed and subject to false positives.

implications in academia, tech, & entertainment

These complexities are critical considerations for academic research, technology, and entertainment. In academia, we are attempting to gain deeper insights into our behavior. In tech, we are attempting to train machines to recognize and classify our emotions. In entertainment, we are attempting to recreate our movements and features down to the follicle. But can we really make headway when decision-makers resist heterogenous perspectives and cling to a frayed baseline of understanding?

If felt smiles of joy, the most widely recognized and arguably simplest emotion to break down anatomically,  are so difficult for us to grasp – imagine the abounding chaos for more complex and less-easily recognized emotions.

recommended readings and perspectives

ANSWER: THE IMAGE ON THE LEFT IS THE SPONTANEOUS, FELT, UNPOSED SMILE.

3 thoughts on ““it’s all in the eyes” and other lies: a critique on contemporary emotion research

  1. Pingback: Animation Tips for Smiles - Face the FACS

  2. Pingback: all about upper lid raiser – AU5 – Face the FACS

  3. Pingback: cheek raiser vs. lid tightener – Face the FACS

Comments are closed