Realistic facial animation remains a cornerstone of visual effects, enabling filmmakers to create compelling characters and immersive storytelling experiences. Facial animation has come a long way since the days of animatronics, as filmmakers now have access to a choice of advanced facial motion capture systems. Technologies such as performance capture and motion tracking, generative AI and new facial animation software have played a central role in the advancement of realistic facial animation. World-leading animation studios are utilizing these tools and technologies to create more realistic content and breaking new ground in the way characters are depicted.

Vicon’s CaraPost single camera tracking system tracks points using only a single camera. Single-camera tracking works automatically when only one camera can see a point. If a point becomes visible again in two cameras, the tracking reverts to multi-camera tracking. (Image courtesy of Vicon Motion Systems Ltd. UK)

A wide range of facial capture technology is in use today. One of the pioneers of facial animation technology is Vicon’s motion capture system, which was vital in its use in films such as Avatar and The Lord of the Rings trilogy. Faceware is another leading system that has been used in various films, including Dungeons & Dragons: Honor Among Thieves, Doctor Strange in the Multiverse of Madness and Godzilla vs. Kong, as well as games such as Hogwarts Legacy and EA Sports FC 24. ILM relies on several systems, including the Academy Award-winning Medusa, which has been a cornerstone of ILM’s digital character realization. ILM also pioneered the Flux system, which was created for Martin Scorsese’s The Irishman, as well as the Anyma Performance Capture system, which was developed with Disney Research Studios. DNEG, on the other hand, uses a variety of motion capture options depending on the need. Primarily, DNEG uses a FACS-based system to plan, record and control the data on the animation rigs. “However, we try to be as flexible as possible in what method we use to capture the data from the actors, as sometimes we could be using client vendors or client methods,” says Robyn Luckham, DNEG Animation Director and Global Head of Animation.

Developing a collaborative working understanding of the human aspects, nuances and complexities and the technology was particularly challenging for Framestore on 2023’s Wonka. “Part of that was building an animation team that had a good knowledge of how FACS [Facial Action Coding System] works,” remarks Dale Newton, Animation Supervisor at Framestore. “While as humans we all have the same facial anatomy, everyone’s face moves in different ways, and we all have unique mannerisms. When it came to generating a CG version of an actor as recognizable as Hugh Grant, that raised the bar very high for us. At the core of the team was sculptor and facial modeler Gabor Foner, who helped us to really understand the quirks of muscular activation in Hugh’s face, [such as] what different muscle combinations worked together with what intensities to use for any particular expression. We ended up with a set of ingredients, recipes if you like, to re-create any particular facial performance.”

Masquerade3 represents the next level of facial capture technology at Digital Domain. “This latest version brings a revolution to facial capture by allowing markerless facial capture without compromising on quality,” Digital Domain VFX Supervisor Jan Philip Cramer explains. “In fact, it often exceeds previous standards. Originally, Masquerade3 was developed to capture every detail of an actor’s face without the constraints of facial markers. Utilizing state-of-the-art machine learning, it captures intricate details like skin texture and wrinkle dynamics. We showcased its outstanding quality through the creation of iconic characters such as Thanos and She-Hulk for Marvel. Eliminating the need for markers is a natural and transformative progression, further enhancing our ability to deliver unmatched realism. To give an example of the impact of this update; normally, the CG actor has to arrive two hours early on set to get the markers applied. After each meal, they have to be reapplied or fixed. The use of COVID masks has made this issue infinitely worse. On She-Hulk, each day seemed to have a new marker set due to pandemic restrictions, and that caused more hold-ups on our end. So, we knew removing the markers would make a sizeable impact on production.”