De-Aging the Iconic Actors of Martin Scorsese’s The Irishman

In 2015, visual effects supervisor Pablo Helman was working with Martin Scorsese on Silence, a stirring film about the Christian faith during 17th century Japan. The two ended up having a conversation that ignited a technological advancement within the visual effects industry—one that will likely become a new standard in how images can be captured and processed during filmmaking.

“We were talking about a project and I mentioned making one of the characters younger. Marty [Scorsese] said it wouldn’t be good for it but he was doing another thing called The Irishman and it sounded like something that they may want to pursue,” says Helman, who then immediately read the script that’s based on Charles Brandt’s nonfiction book “I Heard You Paint Houses.” A detailed account of mob enforcer Frank Sheeran’s life who admits to killing Jimmy Hoffa—the quintessential voice of labor unions of the time. The murder is still considered unsolved.

Helman’s reaction: “I’m in.”

For The Irishman, the characters played by Robert De Niro (Sheeran), Joe Pesci (Russell Bufalino) and Al Pacino (Hoffa) would span across four decades, and all being in their 70s, it would require them to appear younger using digital doubles. The only technology available required actors to perform wearing facial markers and a helmet rig or the performances were created through CG animation.

“If you put yourself in 2015 and take a look at the projects, everyone was using markers including myself,” says Helman. “Marty said that he didn’t want to use technology that was intrusive to the performances so I went back to ILM and proposed we create something that’s completely separate from the actors.” What they designed was a markerless approach to digital de-aging.

To initially test the idea Helman and De Niro reshot a scene from Goodfellas, specifically the one where Jimmy confronts Johnny Roastbeef (Johnny Williams) about purchasing a pink Cadillac for his wife. “Ten weeks later we presented the work to Marty and Bob [De Niro] and the movie got greenlit from that test,” says Helman.

For the next two years, a group of about thirty ILM creatives developed a new camera system and companion software called FLUX – “F” meaning face and “LUX” for light. The system is able to extract the performance data through lighting and texture and translate it into fully renderable versions of an actor.

To make it work, each actor was first shot separately in a controlled environment capturing a wide range of facial movements. The information was used to create 3D models of each actor as they look today. On set, cinematographer Rodrigo Prieto shot scenes with a special three-camera rig. The main director’s camera was a RED Helium that was attached and synced to a pair of ALEXA Mini “witness cameras” that recorded data for the VFX team. The witness cameras focused solely on actors’ faces capturing the performance from multiple angles while recording in infrared and throwing infrared light onto the actors and set. “Because of the way infrared neutralizes the lighting on the actors’ faces, the software is able to create real 3D renders of the performances for every single frame,” says Helman.

With any VFX-driven shot, the more image data you have to start with the better. The ALEXA Mini cameras recorded ARRIRAW at 3.4K resolution. The cameras also recorded at a different shutter speed and depth of field than the RED so the images would be tack sharp. “We always wanted the focus to be on the actors’ faces even when the center camera wasn’t,” notes Helman.

After recording on set, ILM applied the 3D facial scans they created from the controlled environment to the set performance without using keyframes. This turned the physical performances into digital ones, but they were still old. To make them younger, ILM retargeted the performance using the younger models which were created referencing photographs and videos of the actors. The last step was taking the newly created digital head and pairing it to the live-action body.

“We did this in a way so it’s all layered, renderable geometry,” says Helman. “It has pores, skin, light, sheen, interaction…once you put it all together, we’re really creating geometry out of light and texture and comparing it.”

Even with the advancement in technology, the VFX team did not touch the performances. “It’s something Marty feels strongly about and agonizes over so it was important we did not intervene,” says Helman.

However, with the technology burden lifted from the actors, it did shift to other departments. Multiple cameras meant additional operators while production designer Bob Shaw needed to consider the width of sets to allow the wider camera rig to pass through. Prieto also needed to consider lighting choice as the infrared technology worked better with LED fixtures over incandescent.

“Everything we did was to support the performances,” says Helman. “That’s really the achievement here. To have the actors arrive on set without markers so they can do what they do best.”

For more on The Irishman, check out our interview with The Irishman cinematographer Rodrigo Prieto and producer Emma Tillinger Koskoff.

Featured image: Joe Pesci, as Russell Buffalino, de-aged. Courtesy Netflix.