How do you compare FACS to what is done in deep learning?

Most face tracking and emotion tracking companies rely on FACS for machine learning. They use FACS to map out their data pipelines. They collect data with FACS-based expressions. They train their machines with FACS-based classification, etc. Even if you don’t use a FACS-based approach, you still need FACS to identify and communicate areas of failure in order to assess data needs to improve the tech.

Are there any existing FACS datasets we could use to train a machine learning model with face and speech?

From what I’ve seen working in tech on face tracking, most datasets are riddled with impure data. You would need to be able to scrutinize the data and clean up a lot of trash. If you have the resources, creating in-house data collection and labeling systems is recommended. If you are building your own data teams, please read Bias in Emotion Tracking.

Do you recommend any applications that extract FACS robustly? Can you recommend any app for FACS extraction?

If by application that extracts FACS you mean – an automated FACS detector / tracker – I’d say no. I don’t strongly recommend any applications. Some applications can detect very simple, high intensity expressions; however they often fail at detecting subtle or layered expressions. I think we are a ways away from something that’s actually robust, especially due to the data negligence of most leading tech companies. Read more here and here.

iPhones has 52 expressions, is that enough for good animation? If not, how do you use an iPhone to add more FACS shapes?

Are you going for a semi-stylized to very stylized character with basic movements? If so, it could be fine. If you’re going for more photoreal and expressive characters, it’s not going to get you to the level of quality you will need. As someone who is more on the observation and critiquing side of things, I could not tell you how to do add more FACS shapes; however, I am available for consultation if you would like to tell me what needs you have for your characters / what you want to accomplish in terms of expressivity / what shapes you might need to add to get what you want: facetheFACS@melindaozel.com

We are having a boom of 3D live avatars, but they are missing a lot of facial expressions. How do you think we can improve real-time digital human performances?

There are two parts to this:

1. The tech. Face tracking tech is not fully where we want it to be. Successful live avatars can be done by harmonizing the tech with the art. Extensive research and observation needs to be done to assess how the tracking tech responds to different faces. Additionally, the goals of the avatars need to be evaluated. Based on the context, what expressions are priority? Depending on the situations the avatars are being used for, different expressions will have different priority levels.

2. The art. Even though the tech is not where we want it to be, you can fake a lot of it through art. Faking it through the art side can be done once you have a clear idea of what’s not working in the tech and what you want to prioritize for your avatar expressions. When I was at Meta, I worked intensely with the data and studied how the tech performed on different faces. Being able to identify those deficiencies allowed me to effectively work with a modeler to mask recurring issues through the art side. Most successful-looking avatars are achieved by art masking tech issues.

How can different accents be supported by FACS, and how do different pronunciations impact facial expressions / movements for stylized characters?

The smallest meaningful units of sound in a spoken language are called phonemes. Different languages are made up of different phoneme sets. In art and speech tech, phonemes are grouped together by how they look when they are produced. In other words, phonemes are grouped together based on the general shapes our mouths form to produce them. These visually grouped phonemes are referred to as visemes. An example of distinct phonemes that form a single viseme are the bilabial sounds: B, M, and P. Though B, M, and P are distinct phonemes, when we produce those sounds, our lips take on essentially the same configuration. The production of B, M, and P looks the same; therefore, B, M, and P make up a single viseme. FACS shapes can be used to grossly represent a viseme, though finer details may be missing (discussed at the end of this answer).

Accents arise when different phonemes are applied to the same word. For example, consider the word “car.” The American English version pronounces the “a” in car differently from the British English version; therefore, American English uses a different phoneme Additionally, the American English version pronounces the “r” while the British English version does not. 

Though “car” is the same word with the same meaning in both American and British English, its pronunciation is made up of different phonemes depending on which English is being used.

If FACS shapes are used to create the skeleton of a viseme, and a word pronounced with different accents is made up of different phonemes, then that word is also often made up of different visemes. The varying viseme formula means different FACS shapes can be applied to the same words under different accents. This break down means it’s very important to consider the sounds of speech and phonemic options of words – not the letters that make up the words themselves.

The main issues with FACS and speech will come from phonemes produced outside of facial expression mechanics – e.g. guttural languages. Additionally, FACS in general, has a few missing links when it comes to speech shapes – even for American English. This deficiency is why I’ve define sub action units like “vertical lip tightener” and “Z-axis dimpler.” (See FACS Cheat Sheet for details.)