ARKit to FACS Translation

difficulties in AR development toolkits

(Scroll down to skip to the guide.)

If you or your team are using open-source face tracking kits to:

    • animate faces
    • overlay virtual content
    • create expression-based events

. . . figuring out what’s what can be confusing – especially if you or your team do not have a strong background in:

    • facial expressions
    • face tracking
    • the Facial Action Coding System (FACS)

vaguely-defined items

Face tracking software development kits (SDKs) can be confusing for a number of reasons. A large contributor to this confusion is the lack of detailed documentation available for users.

Human facial expressions are complex and rich with nuance. Understanding how to identify and break down expressions is not simple or intuitive. Despite this reality, face tracking kits often only provide vague definitions for the expression shapes in their libraries.

Poorly-defined expression shapes:

    • create room for user misinterpretation
    • increase user’s likelihood to confuse similar-looking shapes
    • limit the user’s potential to effectively use the product

the frown example

When I was at Facebook / Oculus, I worked with many revered research scientists in expression tracking. Even these top minds had trouble describing, identifying and differentiating facial expressions – subtle or not. My coworkers’ inability to clearly communicate the expressions they were referring to led to frequent misunderstandings and miscommunications.

One specific example of miscommunication arose whenever anyone on my team used the word “frown” to describe a facial expression of sadness or disappointment. Depending on where you are from, “frown” can have different meanings.

“Although most technical definitions define it as a wrinkling of the brow, in North America it is primarily thought of as an expression of the mouth . . . The mouth expression is also commonly referred to in the colloquial English phrase ‘turn that frown upside down’ which indicates changing from sad to happy.”
—- Wikipedia

This linguistic debacle is just one example of the many potential misinterpretations that can transpire when discussing facial expressions.

getting around the ambiguity

Expression shapes in most face tracking products (despite their names) are primarily FACS-based. If you wish to foster a better understanding of the face tracking products you are using, you should familiarize yourself with FACS.

FACS naming is standardized. FACS is consistent. Each FACS shape has a detailed, well-defined, and heavily researched description. If you are well-versed in FACS, you can equip yourself with the tools you need to compensate for the ambiguity of most expression libraries.


Whether or not you are FACS-savvy, if you want a clearer breakdown of ARKit facial expression shapes, here ya go:

(NOTE: I am starting with the brows & eyes and will add more to this next Monday – August 10, 2020.)

ARKit Translation Guide

the brows

ARKit term & description

browInnerUp

________________

FACS-friendly term and description

AU1 – inner brow raiser

other notes:

  • Combined L/R shape recommended. 
  • Not common to see this action separated. 
  • May only be useful to separate for complex shape creation.

ARKit term & description

browOutterUpLeft & browDownRight

The coefficient describing upward movement of the outer portion of the left [and right] eyebrow.

Apple's Typo -

other notes: It seems Apple has misspelled “outer.” My nitpickery knows no end.

________________

FACS-friendly term and description

AU2 – outer brow raiser

  • The upward movement of the outer portion of the eyebrows.
  • Caused by the contraction of the outer portion of the frontalis muscle.
  • Due to frontalis muscle variability, AU2’s effects on the eyebrows (e.g. where the main point of focus for the outer pull is in relation to the inner pull) may differ across individuals. (See Frontalis Variation.)
  • For more on AU2:

other notes:

  • AU2 can be confused with full brow raise (AU1+2).

ARKit term & description

browDownLeft & browDownRight

The coefficient describing downward movement of the outer portion*** of the left eyebrow.

*** See “other notes” below.

________________

FACS-friendly term and description

AU4 – brow lowerer

  • The downward movement of the brows which may move straight down or diagonally down.
  • Caused by any combination of the following muscles: corrugator, procerus, depressor supercilii. 

other notes:

  • ***AU4 does not affect the outer brow portion in isolation. In fact, much of the movement occurs centrally.
  • Combined L/R shape recommended. 
  • Not common to see this action separated. 
  • May only be useful to separate for complex shape creation.

the eyes

ARKit term & description

cheekPuff

________________

FACS-friendly term and description

AD34 – puff

NOTE: AD = action descriptor – In the current FACS manual, action descriptors (unlike action units) represent actions that do not yet have a specified muscular basis and does not yet have distinguished behaviors identified.

Because this shape is not specified in FACS, there is not much more to elaborate on. We can fill our cheeks with air in various ways. Consider this shape representative of any general air-in-cheek shapes. That being said, there is one point of caution to consider when using this shape:

In older versions of FACS, the shape now known as “sharp lip puller” was formerly named, “cheek puffer.” “Cheek puffer” / “sharp lip puller” is entirely different from AD34 – puff. Be cautious of this distinction. Do not mistake “puff” for “cheek puff” or vise versa.

AR Kit cheeksquint

ARKit term & description

cheekSquintLeft & cheekSquintRight

________________

quick rant about the word “squint”

Referring to eye-narrowing actions as “squint” is one of my greatest pet peeves. In its most basic (and contextually relevant) definition, “squint” means: “to look with the eyes partly closed.” These parameters are much too vague. There are many ways to “look with the eyes partly closed.” You can:

A) employ cheek raiser
B) employ lid tightener
C) drop down your upper eyelid
D) activate both A and B
E) activate both B and C
F) activate both A and C
G) activate all three – A, B, and C

In the FACS manual, “squint” is used to describe AU7 – lid tightener when it is at maximum intensity. Colloquially, “squint” tends to take on the same definition as provided earlier: to “look with the eyes partly closed.”

Basically, “squint” is an unnecessary word that only ends up confusing laymen and non-technical FACS users as to what anyone is actually talking about.

Another problem I have with the word “squint” is – it is just one typo away from transforming an innocent Google search into an extravagantly NSFW collage of graphic visuals. Type with caution and stay away from the R key.

FACS-friendly term and description

AU6 – cheek raiser (maybe?)

Because “squint” in both colloquial use and FACS-official use most often refers to AU7 – lid tightener, Apple’s choice to combine the words “squint” and “cheek” is baffling.

Based on the information we have to go off: a vague definition (The coefficient describing upward movement of the cheek around and below the left [and right] eye.), a contradictory name, and a single, low-contrast image, it is not immediately clear which action Apple is attempting to articulate here. Let’s investigate further.

To determine whether Apple is representing cheekSquint as cheek raiser, lid tightener, or a combination of the two, I had to overlay the neutral and cheekSquint images and toggle between them in PhotoShop to see the movement. This what the movement looks like:

cheeksquint ARKit

From this GIF, we can conclude Apple is using “cheekSquint” to refer to cheek raiser; however, I rarely see Animoji pick up this action and when it does pick up the action, the animation is extremely mild.

Now that we have confirmed the meaning of cheekSquint . . . 

actual description

AU6 – cheek raiser

  • The pinching and lifting of the skin around the eyes and cheeks.
  • Movement is centered in the outer portion of the eyes; however, it also often dramatically affects the cheeks, pulling them upward toward the outer eye corner.
  • Caused by the contraction of the orbital portion of the orbicularis oculi muscle.
  • Because the orbital portion of orbicularis oculi often overlaps with parts of the brow and the cheek area, cheek raiser can sometimes pull down parts of the outer eyebrows.
  • Learn more about cheek raiser here – Cheek Raiser vs. Lid Tightener.

takeaways & suggestions

“Squint” is confusing. Don’t use it.

If you are looking for alternatives to FACS-official terminology, do not use colloquial language. Be descriptive.

If I had to rename cheek raiser for a face tracking product,  I would call it: orbital pincher. Going with an anatomical and action-based descriptor is straightforward, and it minimizes room for confusion.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.