bias in emotion tracking

AU23 - lip tightener - Facial Action Coding System - FACS

Biases in Face & Emotion Tracking

We seem to subscribe to the popular oversimplification that machines are less biased than humans; however, if you are familiar with the ways in which machines are trained to read and focus on different aspects of data, you will know: It’s just not that simple.

Machines are not free of bias if they are trained by humans.

The following is a presentation of the different types biases that can occur in face tracking and expression labeling. Many of these biases can be reduced; so I have also included suggestions for improved methods. If you are working on face & emotion tracking of any type, it is your responsibility to be aware of these biases.

View the slides below OR the video linked here: YouTube Video

Racial, cultural, and age related biases in face and emotion tracking
Racial, cultural, and age related biases in face and emotion tracking: Many face and emotion tracking companies try to use Paul Ekman’s Facial Action Coding System (FACS)

Text: Many face and emotion tracking companies try to use Paul Ekman’s Facial Action Coding System (FACS) 
– but many don’t take the time to use it right.

Racial, cultural, and age related biases in face and emotion tracking

Text: What happens when you don’t use FACS right?

  • incorrect expression classification
  • inconsistent expression classification
  • biased labeling (racial, cultural, age-related, etc.)
  • anarchy

Even if you use FACS properly there will always be bias and inconsistency – but by taking careful measures, these issues can be significantly reduced.

Racial, cultural, and age related biases in face and emotion tracking: Incorrect expression classification

Text: incorrect expression classification

  • Facial actions are subtle and difficult to differentiate without intensive study.
  • Most FACS references (excluding the original FACS Manual) provide incorrect FACS visuals – even sources considered credible.
  • Despite these inaccuracies, such sources are often used as references by face tracking engineers and researchers. 
  • Because tech companies do not invest enough in data-based roles, they likely do not possess the right staff or resources to differentiate important facial actions.
Racial, cultural, and age related biases in face and emotion tracking: Incorrect expression classification
Racial, cultural, and age related biases in face and emotion tracking: Incorrect Expression Classification

Text: incorrect expression classification

  • Basic shapes like “lip tightener” regularly get confused with actions  like “lip presser” and/or “lip pucker.
  • Lip tightener is important in: emotion expressions & speech production
Racial, cultural, and age related biases in face and emotion tracking: Incorrect Expression Classification

Text: incorrect expression classification

  • Above is a true representation of lip tightener.
  • This is just one of many shapes that fly under the radar each time they are:
    mistaughtmisclassifiedmisused
Racial, cultural, and age related biases in face and emotion tracking: Inconsistent Expression Classification

Text: What happens when you don’t use FACS right?

    • incorrect expression classification
    • inconsistent expression classification
    • biased labeling (racial, cultural, age-related, etc.)
    • anarchy
      •  

Even if you use FACS properly there will always be bias and inconsistency – but by taking careful measures, these issues can be significantly reduced.

Racial, cultural, and age related biases in face and emotion tracking: Inconsistent Expression Classification

Text: incorrect expression classification

The same issues surrounding incorrect classification also contribute to inconsistent classification.

Racial, cultural, and age related biases in face and emotion tracking: Inconsistent Expression Classification

Text: inconsistent expression classification

If tech companies don’t thoroughly invest in data quality, their data classification rules cannot be standardized.

Due to:

  • a lack of investment in hiring and/or training employees for data-based roles
  • a lack of quality FACS resources 
  • an inherent difficulty in differentiating facial actions

Labelers classify expressions inconsistently.
Trackers develop weird quirks, tying incorrect expressions to each other while confusing others.

Text: inconsistent expression classification

NOTE: This diagram was made to explain problems in shape activation for avatars, but the same basic concepts apply to face and emotion tracking.

Text: What happens when you don’t use FACS right?

    • incorrect expression classification
    • inconsistent expression classification
    • biased labeling (racial, cultural, age-related, etc.)
    • anarchy
      •  

Even if you use FACS properly there will always be bias and inconsistency – but by taking careful measures, these issues can be significantly reduced.

Racial, cultural, and age related biases in face and emotion tracking: biased labeling

Text: biased labeling (racial, cultural, age-related, etc.)

When expression labels are ill-defined and poorly understood, small mistakes can create big biases.

Racial, cultural, and age related biases in face and emotion tracking: biased labeling

Text: biased labeling (racial, cultural, age-related, etc.)

Before going into the biases that can exist in face tracking tech, consider the already-existing biases in feature identification tech: facial recognition.

Racial, cultural, and age related biases in face and emotion tracking

Text: biased labeling (racial, cultural, age-related, etc.)

If we can’t even get feature detection right, imagine how complicated bias can be in expression tracking. Expression tracking demands an understanding beyond facial features. It demands an understanding of subtle facial movements.

Racial, cultural, and age related biases in face and emotion tracking: Studies have consistently found that people are noticeably worse at interpreting faces from different cultures.
Text: biased labeling (racial, cultural, age-related, etc.) Studies have consistently found that people are noticeably worse at interpreting faces from different cultures.
    Blais, Caroline & Jack, Rachael & Scheepers, Christoph & Fiset, Daniel & Caldara, Roberto. Culture Shapes How We Look at Faces. (2008).PloS one. 3. e3022. 10.1371/journal.pone.0003022.
Racial, cultural, and age related biases in face and emotion tracking:

Text: biased labeling (racial, cultural, age-related, etc.)

  • People extract information differently when looking at faces and search for different clues when interpreting emotions. 
  • Culture deeply influences what we consider valuable information and clues.

Studies have found:

  • East Asian participants tend to focus on the middle of the face, around the nose, giving more importance to the eyes and gaze direction. 
  • Western Caucasian participants tend to look for expressions of emotion in the eyebrows and the mouth areas
  • These differences in attention create biases when participants look at faces with conflicting expressions
    e.g. When there are sad eyes with a happy mouth:
    – Japanese participants give more importance to emotion shown in eyes
    – American participants care more about the mouth area.

 

1. Blais, C., Jack, R. E., Scheepers, C., Fiset, D., and Caldara, R. (2008). Culture shapes how we look at faces. PLoS ONE 3:e3022. doi: 10.1371/journal.pone.0003022
2. Elfenbein, H. A., & Ambady, N. (2003). Universals and cultural differences in recognizing emotions. Current Directions in Psychological Science, 12(5), 159-164.
3. Matsumoto, D., Kasri, F., & Kooken, K. (1999). American-Japanese cultural differences in judgements of expression intensity and subjective experience. Cognition & Emotion, 13(2), 201-218.
4. Matsumoto, D., & Ekman, P. (1989). American-Japanese cultural differences in intensity ratings of facial expressions of emotion. Motivation and Emotion, 13(2), 143-157.
5. Marsh, A. A., Elfenbein, H. A., & Ambady, N. (2003). Nonverbal “accents” cultural differences in facial expressions of emotion. Psychological Science, 14(4), 373-376.
6. Yuki, M., Maddux, W. W., & Masuda, T. (2007). Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. Journal of Experimental Social Psychology, 43(2), 303-311.

Racial, cultural, and age related biases in face and emotion tracking: expression labeling

Text: biased labeling (racial, cultural, age-related, etc.)

Now consider what goes into expression labeling . . .

Racial, cultural, and age related biases in face and emotion tracking: lip corner depressor and elder bias

Text: biased labeling (racial, cultural, age-related, etc.)

  • If lip corner depressor is not understood correctly, labelers are more likely to incorrectly classify any shape with drooping lip corners as lip corner depressor. These misclassified shapes are often caused by the effects of chin raiserbut they can also include something worse: neutral faces (faces with no expression).
  • One likely outcome of this misconception is: over-detection of lip corner depressor in people with drooping lip corners. Often times, the elderly have drooping mouth corners due to long-term effects of gravity. 
Racial, cultural, and age related biases in face and emotion tracking: lip corner depressor and elder bias

Text: biased labeling (racial, cultural, age-related, etc.)

  • From improper labeling practices, older demographics may be subject to overdetection of lip corner depressor.
  • Lip corner depressor is a key component of sadness. Could this error lead to overdetection of sadness in older demographics?
  • What about other groups whose facial features are characterized drooping lip corners? Will they be incorrectly interpreted as “sad”?
Racial, cultural, and age related biases in face and emotion tracking: lip corner depressor and elder bias

Text: biased labeling (racial, cultural, age-related, etc.)

  • It may not seem like a big deal for a tracker to incorrectly detect lip corner depressor or sadness, but: What happens when face tracking is used for bigger things like mental health assessments or evaluating potential job candidates?
Racial, cultural, and age related biases in face and emotion tracking:

Text: biased labeling (racial, cultural, age-related, etc.)

  • AI Is Now Analyzing Candidates’ Facial Expression During Video Job Interviews – Unilever, IBM, Dunkin Donuts and many others are already using this technology
  • Emotion-recognition technology doesn’t work, but hiring professionals, others are using it anyway: report
  • ‘Emotion detection’ AI is a $20 billion industry. New research says it can’t do what it claims.
Racial, cultural, and age related biases in face and emotion tracking: Feature-based biases

Text: biased labeling (racial, cultural, age-related, etc.)

We talked about how improper labeling can incite bias against older populations . . . But what about other biases toward people with certain facial features?

Racial, cultural, and age related biases in face and emotion tracking: Feature-based biases

Text: biased labeling (racial, cultural, age-related, etc.)

“. . . facial recognition programs exhibit two distinct types of bias.

First, black faces were consistently scored as angrier than white faces for every smile. Face++ showed this type of bias. Second, black faces were always scored as angrier if there was any ambiguity about their facial expression. Face API displayed this type of disparity. Even if black faces are partially smiling, my analysis showed that the systems assumed more negative emotions as compared to their white counterparts with similar expressions. The average emotional scores were much closer across races, but there were still noticeable differences for black and white faces.”

Understanding the Hidden Bias in Emotion-Reading AIs
-Lauren Rhue

Racial, cultural, and age related biases in face and emotion tracking: How might this happen? Let’s revisit WHY IT’S IMPORTANT TO INVEST MORE IN DATA.

Text: biased labeling (racial, cultural, age-related, etc.)

How might this happen? 

Let’s revisit WHY IT’S IMPORTANT TO INVEST MORE IN DATA.

Racial, cultural, and age related biases in face and emotion tracking: Upper lip raiser

Text: biased labeling (racial, cultural, age-related, etc.)

Consider the FACS shape “upper lip raiser.” Upper lip raiser is a key component of emotions like disgust, angerand contempt.

Features of upper lip raiser include:

  • raised upper lip
  • rounded upper nasolabial furrow region
    (laugh line – see photo)

NOTE: On top of potential facial action labeling biases, there is also significant controversy over the Basic Emotion Theory (e.g. basic emotion prototypes like contempt, anger, etc.) in general. See “‘It’s All In the Eyes’ and Other Lies.

Racial, cultural, and age related biases in face and emotion tracking: Nasolabial furrow

Text: biased labeling (racial, cultural, age-related, etc.)

Some people have more curve in their nasolabial furrow.

This is simply a product of their naturally-occurring face structure.

Racial, cultural, and age related biases in face and emotion tracking: Nasolabial furrow

Text: biased labeling (racial, cultural, age-related, etc.)

A labeler not properly versed in the Facial Action Coding System can easily mislabel people with non-expression-based nasolabial furrow curves as having upper lip raiser.

These incorrect labels birth a tracker trained to identify faces with certain structures as expressing upper lip raiser – even when they are neutral or smiling.

Procuring a “diverse data set” is irrelevant if your labels are inaccurate.

Such training explains Lauren Rhue’s findings from Understanding the Hidden Bias in Emotion-Reading AIs.

Racial, cultural, and age related biases in face and emotion tracking: Holistic emotion labeling

Text: biased labeling (racial, cultural, age-related, etc.)

The presence of biased emotion labeling will persist even if you move away from a FACS-based approach and attempt to have labelers classify emotions holistically.

Racial, cultural, and age related biases in face and emotion tracking

Text: biased labeling (racial, cultural, age-related, etc.)

In case it is not clear why holistic emotion labeling is just as likely to fail (if not, moreso) – HERE IS A REFRESHER:

  • People extract information differently when looking at faces and search for different clues when interpreting emotions. 
  • Culture deeply influences what we consider valuable information and clues.

Studies have found:

  • East Asian participants tend to focus on the middle of the face, around the nose, giving more importance to the eyes and gaze direction. 
  • Western Caucasian participants tend to look for expressions of emotion in the eyebrows and the mouth areas
  • These differences in attention create biases when participants look at faces with conflicting expressions
    e.g. When there are sad eyes with a happy mouth:
    – Japanese participants give more importance to emotion shown in eyes
    – American participants care more about the mouth area.
Racial, cultural, and age related biases in face and emotion tracking

Text: What happens when you don’t use FACS right?

    • incorrect expression classification
    • inconsistent expression classification
    • biased labeling (racial, cultural, age-related, etc.)
    • anarchy
      •  

Even if you use FACS properly there will always be bias and inconsistency – but by taking careful measures, these issues can be significantly reduced.

Racial, cultural, and age related biases in face and emotion tracking: Mitigating Bias

Text: anarchy

If . . . 

  • culture influences how we interpret expressions and emotions
  • environmental conditions affect which clues we look at to evaluate faces
  • knowledge of facial actions affects how we label expressions

. . . What can tech companies do to reduce these human-based errors and prevent them from leaking into algorithms?

Racial, cultural, and age related biases in face and emotion tracking: Mitigating Bias

Text: What can tech companies do to reduce biases in face tracking?

  • Tech companies must recognize that face tracking has advanced beyond the scope of engineering.
  • Tech companies must accept tech’s vulnerability to bias and educate employees on these vulnerabilities.
  • Tech companies must invest more in data quality.
Racial, cultural, and age related biases in face and emotion tracking: Mitigating Bias

Text: Recognizing the advancements of face tracking tech, understanding its vulnerabilities, and investing more in data quality means . . . 

  • Actually allocating headcount toward roles focused on data quality – even if that requires taking space from engineering headcount.
  • Spending time, energy, and resources to find data specialists. If this is not possible, it is still the company’s job to spend time, energy, and resources to TRAIN data specialists. Classifying facial expressions is much more complicated than classifying basic objects like traffic signals and should be treated as such.
Racial, cultural, and age related biases in face and emotion tracking: reducing biases in face tracking

Text: Reducing biases in face tracking

Recognizing the advancements of face tracking tech, understanding its vulnerabilities, and investing more in data quality means . . . 

  • Engineering, research, and product should not simply tell data what they need – but LISTEN to what the data teams need. Data teams are the most knowledgeable about behind-the-scenes work required for functioning algorithms. THIS IS WHERE MANY COMPANIES FAIL.
  • Requiring employees with data-based roles to regularly interface with engineering, research, and product teams.
Racial, cultural, and age related biases in face and emotion tracking: proper training

Text: Reducing biases in face tracking

Recognizing the advancements of face tracking tech, understanding its vulnerabilities, and investing more in data quality means . . . 

  • Educating employees on the reality of bias in tech.
  • Taking the right precautions to standardize and define labeling.
  • Always considering when and where bias can appear.
    – Who is labeling?
    – What are they labeling?
    – What factors might impact their labeling?
    *
    mood * culture * experience
Racial, cultural, and age related biases in face and emotion tracking: static images

Text: Reducing biases in face tracking

Recognizing the advancements of face tracking tech, understanding its vulnerabilities, and investing more in data quality means . . . 

SIDENOTE: STOP LABELING EXPRESSIONS FROM STATIC IMAGES.

  • Accurate FACS and emotion labeling is largely contingent on seeing movement.
  • If movement were stressed more, static facial features such as drooping lip corners and curved nasolabial furrow would be less likely to activate detection of lip corner depressor and upper lip raiser (respectively).
Racial, cultural, and age related biases in face and emotion tracking: Closing Notes

Text: Reducing biases in face tracking

Closing notes . . . 

Racial, cultural, and age related biases in face and emotion tracking: Ramifications

Text: Reducing biases in face tracking

The more we believe in and rely on technology, the more ramifications these biases will have.

Racial, cultural, and age related biases in face and emotion tracking: AI Implications

Text: Reducing biases in face tracking

“… facial recognition software interprets emotions differently based on the person’s race.  …This finding has implications for individuals, organizations, and society, and it contributes to the growing literature of bias and/or disparate impact in AI.”

Rhue, Lauren, Racial Influence on Automated Perceptions of Emotions (November 9, 2018). Available at SSRN: https://ssrn.com/abstract=3281765 or http://dx.doi.org/10.2139/ssrn.3281765

Racial, cultural, and age related biases in face and emotion tracking: Data Responsibility

Text: Reducing biases in face tracking

Take responsibility.
Put in the work to reduce bias.
Invest more in data.

Racial, cultural, and age related biases in face and emotion tracking: FACS Cheat Sheet

Text:

In response to the prevalence of low-quality and inaccurate FACS references, I have created a free “FACS Cheat Sheet” to serve as a guide for artists, researchers, and engineers. It is available on my FACS resource site, Face the FACS. I am also open for consulting.

melindaozel.com/facs-cheat-sheet/

One thought on “bias in emotion tracking

  1. Pingback: “it’s all in the eyes” and other lies: a critique on contemporary emotion research – Face the FACS

Comments are closed