What is Virtual Production: A Complete Guide to the Technology That's Changing Filmmaking

Avatar photo Marco Sebastiano Alessi | December 9, 2023

Advancements in technology have revolutionized the field of filmmaking, enabling us to craft incredible worlds, breathtaking landscapes, and fantastical creatures that were once beyond the scope of traditional filming techniques.

Historically, visual effects (VFX) were applied during post-production. Nowadays, many filmmakers have embraced a novel approach that has not only reshaped their creative process but also has the potential to supplant the use of the green screen in the future. This technique marks a pivotal moment in the evolution of the movie industry.

In this guide, we'll dive into the realm of virtual production, what it is, its cons and pros, its stages, and what’s necessary to create a successful setup for a virtual environment.

Let’s dive in!

What is Virtual Production?

At its core, virtual production combines traditional filmmaking methods with new technologies such as virtual reality, motion capture, CGI, augmented reality, game engines, and other immersive technologies and digital visuals to create computer-rendered environments, characters, digital assets, and visual effects.

These compositions are then projected onto LED screens, allowing the production teams to see how a scene will look in its final form in real time

In a virtual production, the real and virtual environments appear simultaneously, allowing the production crew and actors to see the entirely composed scene with effects as they film it on the set, make changes in the lighting, camera angles, and atmosphere, and even move or remove objects and assets from the scene without having to wait until post-production to notice the flaws.

Furthermore, it also helps actors to interact and react in a set more naturally.

Virtual production makes use of technology borrowed from the gaming field to create virtual sets. Game engines like Unreal Engine allow developers to build the worlds for their games from scratch, and now filmmakers have discovered that the same technology can be applied to create virtual locations for films and television.

  • The Origins of Virtual Production

    Looking back, it’s easy to see the precursors of virtual production in the front and rear projection techniques, which required using pre-recorded footage projected on a screen to use behind the actors in a shot. For instance, it was common in driving scenes, making it easier to shoot dialogues.

    The first examples of virtual production used in movies can be traced back to the early use of CGI in films such as Tron (1982) and The Abyss (1989), where computer-generated elements were combined with live-action footage.

    However, the concept truly began to take shape with the rise of digital filmmaking tools in the 2000s. In particular, Avatar (2009), directed by James Cameron, utilized a virtual camera system that allowed the director to view the CGI world in real time while filming. 

  • Game Engines

    Unreal Engine is the most used game engine in video, film, and television. Epic Games developed it for first-person shooters and PC video games, and later, the technology expanded to other fields, such as mobile, consoles, and virtual reality platforms.

    Creating a real-time virtual world allows filmmakers to control the landscapes and visualize camera angles, framing, and sequences for their films. Furthermore, the 3D environment reduces the cost and the hassle of creating props.

    In a 3D world created with Unreal Engine, filmmakers can track the in-game camera to sync it with the in-real camera on stage. This allows directors to match the angles of the screen with the ones from the real camera, fine-tuning perspective, and parallax, something impossible with rear projection.

  • What Does LED Volume Mean in Virtual Production?

    You will hear a lot about LED volume or simply volumes in virtual productions. It refers to the screens where the 3D environments created on Unreal Engine are projected and where the actual filming takes place.

    Once the virtual stages are created, these are projected on an LED screen, LED walls, or an LED volume. LED walls are built from high-resolution LED panels and a flat ceiling with LED lights. LED volumes can also be circular stages with high-resolution LED panels around them and a flat roof.

    Think of volumes as gigantic circular LED screens where the virtual world generated in the game engine is projected and operated in real-time by a VFX team.

    ARRI's Mixed Reality Studio - A Cameraman In A LED Volume

    With virtual productions, shooting in imaginary locations becomes not only possible but also convenient. Furthermore, you’ll be able to shoot in the sunlight or at night at any moment you need, empowering you with a more flexible and creative schedule, all thanks to virtual production. LED volumes allow shooting in multiple scenarios at any time of the day without building a physical set that will require plenty of time and budget.

    However, LED volumes are not limited to the film industry. LED walls and big LED panels have already been used in advertising, broadcasts, and music videos.

Pros and Cons of Virtual Production

  • Pros of Virtual Production

    • Time-Saving

      One of the greatest pros of virtual production is how time-saving this technique is. Virtual production eliminates the time required to build physical sets, travel to locations to shoot, multiple shots to try different camera angles for the post-production team to work with, and finally, less work during post-production, allowing you to deliver a final cut much faster.

    • Cost Savings

      Since all the backgrounds and environments are created digitally, the cost of building the sets, travelling, and expenses connected to production, filming, and post-production corrections are reduced without giving up on quality.

    • Early Visualization

      Virtual production allows generating the virtual scenarios earlier in the process, which means you can visualize an environment earlier in the process to make decisions on lighting, real-life props required, atmosphere, framing, sequence direction, and more.

    • Shoot Anywhere, Anytime

      With virtual production, you have more flexibility to shoot different scenarios like a forest, desert, mountains, beach, under the sea, space, in the sky, and more that would take much time, cost, and preparation to film with traditional filmmaking techniques.

      You can shoot in multiple scenarios on the same day without having to travel to a different place or take longer shootings in the daylight without the restrictions of time, lightning, and weather.

    • Immersive Performances

      Having the virtual environments ready before the shooting begins allows actors to practice their performance with a better understanding of what’s going on in the scene, allowing them to immerse in the 3D world during the actual filming.

      It also helps actors look in the right direction, where the digital assets are. The performance will look more natural since actors will be able to see the VFX in real time.

    • Live Compositing

      During the filming process, the volume control team can make changes in real-time to the three-dimensional world. They can adjust the lighting, create reflections, add or remove assets from the world, change the color grade, etc.

      Normally, you wouldn’t be able to correct these settings until post-production, but now, all this can be done in pre-production or during the shooting.

    • Lighting

      Lighting control is another benefit that virtual production provides over traditional filming techniques. The LED wall can produce a mix of traditional and digital lighting in which you have more control by adjusting your virtual environment's brightness, shadows, and contrast.

    • Atmospheric Control

      The volume control team has total control over the time of the day and the weather and can create atmospheric conditions such as rain, fog, intense wind, clouds, and smoke. You can also include sounds to reproduce a specific setting, like the sounds of waves, wind, explosions, or the silence of the space.

    • Digital Assets

      Virtual libraries allow you to buy virtual assets to customize to your needs and save time building them from scratch. You can even store digital assets to use them in future productions. Imagine having locations like a spaceship, a desert island, or a facility underwater that you can modify to meet your production needs.

    • Remote Work and Global Collaboration

      Virtual productions allow teams to work remotely, collaborate in real-time, and have your crew spread out globally. There are no more geographical limitations with virtual production.

Virtual Production Stages

The traditional filmmaking process starts with the development and writing in pre-production, followed by the production when the filming takes place. Finally, the post-production, where VFX artists add visual effects and digital assets.

With the virtual production pipeline, this process has changed. Here is a general overview of the virtual production stages.

  1. Pre-Production

    In a virtual production, most things must be done in pre-production, contrary to the traditional process where most of the film was created and fixed in post-production. In this first stage, you create the idea and start writing.

    The art department, VFX artists, the pre-visualization team, and virtual art departments craft the world, scenery, and assets in the game engine, as well as all special effects required for the final product.

    It’s at this stage where most of the VFX work takes place.

    The pre-production process involves teams creating content for the three-dimensional world with photogrammetry techniques to build 3D assets, deciding on the size of the volume, the camera tracker, and more.

  2. Filming

    Now, the actual filming begins. The render is projected on the LED screens, and actors perform in the volume.

    The shot captures the actor and the LED screen together as a whole. Then, the director decides whether to adjust the environment, change the lighting, and color grading, as the volume control team works in real time on the set, reducing post-production editing, cost, and deadlines.

  3. Editing

    The editors will take all the video content and edit it for the post-production team to add additional VFX elements and other required corrections.

  4. Post and VFX

    At this stage, the post-production team ensures the blend with the virtual backgrounds, actors, and physical props looks realistic and applies any necessary corrections to enhance the footage.

  5. Film Distribution

    Finally, the film is distributed for projection in cinemas, broadcast TV, or streaming platforms.

    As you can see, with the new virtual production pipeline, the main change is that most of the post-production work is now done in pre-production when multiple teams create content for the filming.

What Do You Need for a Successful Virtual Production Setup?

A virtual production setup requires specific tools and equipment that will allow you to successfully create content for film, television, and even for shorts and indie productions.

  • Game Engine

    The game engine is the most crucial tool that you will need, as it’s where you’ll create your whole universe in a three-dimensional environment. Unreal Engine and Unity Engine are both excellent engines that offer an asset marketplace where you can buy digital assets for world-building.

  • Camera Tracking Technology

    You will need virtual reality gear to track camera movement from the in-game world. VR headsets and some Apple devices with LiDAR sensors will certainly do.

  • Cameras

    Just because you’re working in a virtual location doesn’t mean you won’t need to shoot in real life! With that said, the process involves at least some type of performance capture.

  • Audio Equipment

    You’ll require microphones, a boom arm, an audio interface, and headsets. Determine your needs and budget to see if you’ll need to buy or rent more equipment. Planning around the recording area will also be required. You need perfect circumstances to record audio uninterrupted.

  • LED Wall or LED Volume

    Decide whether an LED wall or volume will be required for your production. An LED wall or a big LED screen will be enough for smaller production.

  • Computer

    You’ll need a powerful computer to run the game engine. Ensure your machine has enough CPU power and invest in a high-quality graphic card. The NVIDIA GeForce RTX 4090 or the RTX 6000 will be your best options; however, consider your budget and the size of your LED wall.

  • Storage Device

    You will require plenty of storage for your video files, digital assets, audio files, and other production files. Using a solid-state drive (SSD) is preferable due to its faster read and write speeds. Plus, consider using a cloud service to exchange files with your team.

Final Words

Virtual production is a cutting-edge technology that allows filmmakers to bring their ideas to life in a way that was unthinkable before and create high-quality content while saving production costs and time.

More films and television series are starting to use in-camera visual effects to create photorealistic and controlled digital environments instead of the standard green screen.

Although this technology has been available for a while, only recently have filmmakers started using virtual production pipelines to create outstanding movie productions, and we should all look forward to seeing how this technology will evolve and change visual storytelling.

FAQ

What is an example of virtual production?

The Lion King (2019) film was created using virtual production techniques and filmed using real cameras to control the in-game cameras. Disney's The Mandalorian was one of the first major live-action productions to use LED walls over a green screen.

Rogue One: A Star Wars Story and Star Trek: Discovery also used virtual production extensively.

Other examples are Epic Games’ Fortnite concert for Travis Scott, and recently singer Jackson Wang made his music video using Unreal Engine.

Travis Scott's "ASTRONOMICAL" Event In Mega-Popular Video Game "Fortnite"

Is Virtual Production VFX?

VFX is just a part of virtual production. A virtual production process includes more techniques such as motion capture, camera tracking, world-building, virtual scouting, virtual reality, and virtual design. The 3D whold is then rendered to an LED volume where the filming takes place with the actors, the director, and the crew.