Gradually over time, the film industry has turned towards digital technology to solve problems in efficiency and effectiveness when it comes to film production. Filmmakers have gone from shooting on film to capturing footage digitally, from painted backdrops to using green screens, and now, from physical production to virtual.
The term “virtual production“, however, is not describing a change in the physically collaborative nature of filmmaking; instead, it refers to the method of blending augmented reality and video game technology with live-action footage to visualize scenes with heavy VFX before going to post-production. James Cameron and more recently, Jon Favreau are two of many directors that have used virtual production on their sets.
What is Virtual Production?
The adoption of VFX in film and video has made productions remarkably complex. Since visual effects are typically finished in post-production by VFX artists, production crews work on set with only green screens and mocap suits. There is a visual disconnection that forces filmmakers to make an educated guess on how the final image will look, leading to a degree of uncertainty among crew members. More often than not, there are inconsistencies in areas such as lighting, color, and editing.
Virtual production eliminates uncertainty among production crews by enabling them to see an accurate depiction of the final image on set. The method combines virtual reality, computer-generated images, and game-engine technologies to create digital assets that can be viewed in pre-production or on set in real-time. The digital assets could be anything from CG character models to virtual environments that will be used as a setting for a scene. Virtual production gives crew members a collective vision to work from.
Different Types of Virtual Production Tools
Virtual production does not refer to only one new approach to film production but rather a variety of different tools that can leverage VFX to aid each phase of production. These are 4 of the most common types of virtual production:
Previsualization or Previs
Previsualization is the process of visually mapping out a scene before it is filmed. The traditional methods of previs are known as shot listing, storyboarding, animatics, and photography. Virtual production revamps the previs process by using a video game engine to construct sets digitally in virtual reality. Within the virtual environment, filmmakers can use a virtual camera feature that allows the user to use a device such as a tablet as a lens to view the virtual space and frame shots.
Jon Favreau is leading the charge with this approach and he calls the process virtual cinema. For Rogue One, Gareth Edwards used virtual cinema to plan out the epic space battle sequences. The crew constructed the space battle set in a digital environment and Edwards used a virtual camera (iPad) to plan out the framing and camera movement for each shot.
Virtual Scouting
Location scouting is a fundamental part of the filmmaking process. However, video game technology like Unreal Engine has enabled crews to map out realistic digital environments. Virtual scouting is the digital method of identifying locations and reconstructing them virtually. These virtual environments can be fictitious or realistically constructed based on a real-life location. Crew members can experience these environments in virtual reality and use the experience to map out entire sequences ahead of time.
Performance Capture
Motion capture is the process of capturing the movement of objects or actors and using that data to build CG models. Whereas performance capture is the process of capturing not only body movement but also facial movement and subtle expressions. Simulcam is one of the most well-known approaches to using performance capture and it originated on the set of James Cameron’s Avatar. Simulcam is a method that shows the results of the performance capture in real-time by superimposing the virtual environments and characters over the live footage. Simulcam is an intersection of both the physical camera on set and the virtual camera in the digital space.
LED Wall
Painted backdrops were commonly used in 20th-century cinema. Now, LED walls are the modern equivalent to painted backdrops and are one of the most widely-recognized forms of virtual production. The term “LED walls” refers to an enormous LED screen designed to be used in a film studio as a backdrop. The LED screen displays live photorealistic images of environments that work as the setting for a scene.
The display can also link up with the digital camera to render in real-time based on the camera’s position. In other words, as the camera moves, the LED screen shifts the perspective of the image to match how it naturally would shift in real-life. LED walls have proven to have massive benefits and the potential to completely change the filmmaking process.
Jon Favreau has enthusiastically used this technology to film almost half of the scenes in The Mandalorian. The LED wall used on the Disney plus series is one of the biggest out there and Favreau calls it “The Volume“. The Volume is a 20 foot tall, 270 degree, 75 foot long LED screen. With The Volume, the cast and crew could visualize how to approach the scene and were better immersed in the environment.
Virtual Production Is Hollywood's Post-Pandemic Solution
A year ago, the world was at a standstill. Industries everywhere were facing unique challenges because of the Covid pandemic. The film industry, in particular, was confronted with an unprecedented challenge. How do you create a film without bringing people together? And, how do you bring people together safely? Unexpected challenges naturally inspire innovative solutions which was an idea Hollywood embraced in 2020 with virtual production. Many films used virtual production to continue working safely and remotely. It’s clear the pandemic expedited the adoption of virtual production in the film industry and one must wonder what that means for the future.