So, how did this happen? How did 85% of voters get it wrong?

 

The purpose of this test was to demonstrate the fallibility of our assumptions on facial expressions of emotion – and it did just that. Had I included video clips rather than photos, I am certain the audience would have performed much better; however, it is important to note that still images currently serve as the backbone for an alarming portion of emotion research and technology. Emotions are full of intricacies and nuances that, even with video footage and motion, we struggle to truly understand.

Emotion research centered on facial expressions has been around for a good number of decades – centuries even, if you consider early contributors, like Charles Darwin. Regardless of where you begin your timeline, the mainstream foundation underlying our present understanding of the face-to-emotion relationship comes largely from psychologist and father of the Facial Action Coding System: Paul Ekman.

 

While Ekman’s research and writings have inspired many of us (myself included) – tech, academia, and entertainment have become too reliant on the work of this single contributor and his followers. The overwhelming majority of companies I’ve worked for or contracted with have either been wholly blind to and/or disinterested in alternative branches of thought. Being that I have worked for Big Tech and influential emotion tracking startups: This is unsettling.

 

With the progression of tracking technology and the increasing demand for realism in digital art, it’s beyond time that these industries (and likely many others) adopted a more comprehensive approach to understanding emotions and facial expressions.

While I still hold Ekman’s work with high regard, I have also learned to take a lot of claims on “true” emotion with a grain of salt. As with many big questions in science – it depends.

 

We need to move forward in time with new research and begin integrating more theories, approaches, and critiques from other sources into our products and practices. We need to stop pushing Ekman’s emotion research as unchallenged gospel and start pulling from other sources.

 

Ekman’s work was revolutionary for its time and remains extraordinarily useful; however, it is dated, incomplete, and regarded by numerous leaders in modern emotion research as highly subject to bias. Yes, it is certainly nice to believe we can reliably determine genuine emotion with rules like: “eye contraction in a smile makes it a true expression of joy” or “genuine expressions are more symmetric,” but it’s just not that simple.

In the Ekman and mainstream pools of thought, it is said that in order for a smile to be genuine and positive in affect, the orbital portion of “orbicularis oculi” (a sphincter-like muscle in the eye region) must be contracted in conjunction with zygomaticus major (the muscle typically used for mouth smiling). Such eye + mouth combo smiles are referred to as “Duchenne smiles.“

 

Because of the popularization of the Duchenne smile, certain concepts tend to get parroted around in pop culture and the emotion/nonverbal behavior world:

• • The truth is in the eyes!
  • Real smiles show in the eyes!

It is true that many spontaneous expressions of joy, amusement, and other positive, smile-based expressions are likely to feature orbital eye contractions. However, it is also true that many unfelt and posed smiles feature this same muscle activation, while some felt and unposed smiles do not. In fact, recent studies revisiting the ideas behind felt vs. unfelt smiles have found that presence of orbicularis oculi contraction did not reliably predict felt-ness. Inferring emotion is not a matter of simple binary approaches.

• • the shape and size of orbicularis oculi
  • the shape, thickness, length, and orientation of zygomaticus major
  • the spatial relationship between orbicularis oculi, zygomaticus major, and other surrounding muscles
  • facial fat content and distribution
  • dynamic wrinkle presence
  • age
  • the presence of additional muscles interfering with the orbital region

In some people, orbicularis oculi and zygomaticus major muscles overlap; in others, they do not. There are also lesser-known muscles that inherently affect smile-based eye-mouth interactions such as the: