Bringing virtual production to the Middle East

Fractal Studio opened for business in Dubai last month. The multi-million-dollar set-up is the UAE’s first fully integrated extended reality (XR) and virtual production (VP) studio, with a 24x5m customised LED wall taking centre stage at the 650sqm facility. XR and VP offer the opportunity to create hyper-realistic 3D environments paired with powerful real-time render software solutions thus far limited to the gaming industry. With the opening of Fractal Studio, the technology will now become accessible for TV commercials, dramas, feature films, broadcasting and virtual events in the region.

Fractal Studio founders Thierry Louesse and Renaud Barbier explain the thinking behind the facility: “Digital content has become an important part of our lives. There is a growing appetite for documentaries, movies, features and the likes of Netflix, OSN and HBO have some fabulous series. But this means there is also an increasing need to produce more content with more quality, but within a shorter time and budget. The green background studio has considerable limitations specifically within the immersive environment. The use of XR with advanced technologies and an integrated data flow can provide a quick, cost-effective and high-quality output today. With Fractal Studio, we want to be able to provide producers, agencies and so on with a ready-made structure that offers all XR services under one roof so they can be free of logistical constraints and be able to fully express their creativity.”

Fractal’s state-of-the-art infrastructure might be the most advanced in the world, not just the Middle East. The 24m LED wall is 5m high with a 192° curvature and 2.5 pixel pitch, complemented with a rendering system powered by Disguise that allows up to 8K live rendering with infinite possibilities. Accompanying that set-up is an Optitrack camera tracking system, a 100G Mellanox network and Lightware connectivity solutions.

Bespoke infrastructure

For the installation of the curved 120sqm LED, Fractal worked closely with INFiLED, which created it to Fractal’s specifications using next-generation cabinets and 2.5 pixel pitch LED. Since it was a bespoke structure, INFiLED was able to optimise it for specific colour spectra that are critical to Fractal’s camera use but unavailable in off-the-shelf LED panels. This customisation also allowed the curve of the screen to be placed where Fractal wanted it without having to change the entire supporting structure of the wall.

“Customisation is key to the whole XR process,” says Gheysen. “This technology is so new that there is no buyable for it. It’s very experimental. You cannot categorise it and say follow these steps to get to the final product. If you really want a good product, you have to customise it and ours took eight months to make.”

The quality of the LED screen is the most critical element of a virtual production. Low-quality screens lead to visual artefacts, flickering, scan lines, aberrations and mis-matched colours.

“If not done right, it could become the weakest link in the entire VP workflow. If it’s not able to accurately project a wide palette of colours or if its refresh rate is too low, if the matrix is too reflective or has a low angle of vision, the camera will see it. There’s no better way to understand your LED product than to point a camera at it as early as possible, since cameras will reveal all the issues,” Gheysen remarks.

Fractal’s LED gives it a significant lead over any upcoming competition, not just in terms of infrastructure but technology acquisition too. The surge in demand for XR and VP has already put pressure on manufacturers, with wait times currently stretching from six months to a year.

Disruptive tech

XR and VP or any other technology for photorealistic environment creation is useless without real-time rendering. Fractal’s USP is its ability to render 8K images live, through the multiple layers of networking in the installation.

The first step in the process is the actual video information, where the rendering of the actual image takes place on rendering nodes that funnel their ‘tile’ to the controllers taking care of the final projection. These video feeds are then transferred via an L3 100G fibre switch, with each controller receiving up to 50Gbps of data to process.

Control of the system takes place on a separate 10G network, while a third 10G network handles the distribution of media and data transfers that are not time-bound. Due to the need to keep so many rendering nodes and edit stations up-to-date with projects ranging from 25 to 250GB, with the probability of multiple users live editing as well (locally or remote), the media distribution layer is a massive IT infrastructure that allows XR providers to receive tracking information, and to render and project images with minimal delay so the camera can’t see the difference between the real and virtual world. Advanced IT infrastructure is thus another important element of the XR process.

Fractal’s superior XR capabilities are possible thanks to the use of Unreal Engine’s gaming technology, which generates real-time photorealistic 3D scenes with live CGI projection. To tackle the high rendering power needed to manage the LED wall’s 24 million pixels, Fractal selected Disguise for rendering.

“Our decision was guided by the short deadline of the project, the great knowledge that the Disguise team has in the XR world and the fact that their architecture is extremely flexible and scalable. We are now capable of scaling the rendering power depending on the project at hand, ensuring that we have power on demand as opposed to a fixed installation,” says Gheysen.

“With Disguise’s RenderStream technology, the Fractal team is able to virtualise the resources and allocate them where they are most needed, depending on the project and the scenes at hand.”

Another defining trait of Fractal is its XR stage’s tracking ability. Using the OptiTrack solution, they can not only track multiple cameras, physical lights and random objects in the scene but can also undertake full motion capture. The video in/out switching on the premises is handled by Lightware, which reportedly has the best 4K matrices on the market.

“We work with Arri and Red cameras primarily. Dubai-based Filmquip Media is our key partner for all film equipment. They’ve been with us from day one to ensure we develop our product using tools that the film industry is already familiar with. The cameras are as per what the client needs. We can scale from one camera to four or more cameras. Most commercial shoots need one or two cameras. For the tracking cameras, because we track the whole volume, there are multiple cameras around the studio,” explains Samarmand.

Extended benefits

Fractal’s value proposition consists of an entire package to shoot and produce video content, says Samarmand. “Virtual production has so many technical and creative elements involved, so we provide a whole package for those who want it. We can handle all the arrangements in terms of technical environment required for creation, to projection and final pixel on the camera.”

The benefits of using XR and VP compared to traditional filming production are manifold. It encourages a more iterative, non-linear and collaborative process. Creating previs imagery via a real-time engine means sequences can be quickly updated and output at very high levels of image quality. This allows directors to replace the visual effects with working imagery far closer to the final pixel, making iteration and experimentation simpler, cost-efficient and agile. The use of XR and VP also helps filmmakers eliminate the uncertainty of traditional pre-production, empowering them to collaboratively iterate on visual details in the moment, not deferring all these decisions to post.

Fractal’s virtual production can bring any location in the world into the studio. Their team can carry out a shot-by-shot analysis of the entire script, advising the producer on what kind of backgrounds are available. Whether it’s the deserts of Arabia or the Golden Gate Bridge of San Francisco, they can all now be shot in Dubai.

Apart from a library of ready-made backgrounds, the Fractal team also works with international UFX/VFX partners to create bespoke backgrounds for clients. The process of choosing or creating a background can be likened to the location recce traditionally conducted by directors. In virtual production, this is done digitally. There’s a lot of flexibility with whichever background process filmmakers choose as any element of the background – lighting, and the position of natural elements like the sun or buildings or shadows, can be changed with just a few clicks.

XR creates highly immersive production environments. For instance, when using green screens to create virtual scenes, one of the biggest limitations actors or presenters face is that they cannot actually see the environment they are supposed to be interacting with.

Traditional green screens also require intensive and expensive post-production iterations for things that would have been easily fixed if discovered during shooting.

XR is even more beneficial in a live environment, as studios can make use of CG graphics in the moment. By plugging camera tracking data into Disguise’s software, engineers can render content from the camera’s view. Using camera tracking technology, real-time content is generated into the LED walls based on camera positions. This holds the key to extending these small sets into large virtual spaces. When set extensions are switched on, the virtual world continues beyond the LED screens for the viewer at home, rendering in all directions around the stage and accommodating the largest possible programming scale.

“These XR backgrounds are so real that even the camera gets tricked into not seeing the difference between the virtual world and the real one,” says Samarmand.

XR and virtual production have the potential to change the whole filming workflow, says Gheysen.

“Traditionally filmmakers had to go on set, shoot, and then post-production takes anything from two weeks to two months. Now this has all been moved up to pre-production. The XR LED allows directors, actors and other creatives involved in the filmmaking process to do a full visualisation of the entire commercial before even getting into the studio. When the crew come to shoot, it’s not a new location for them. They’ve already seen the environment and know where to place the cameras and shoot from. The director understands better what he is going to see as the final scene, and there are no surprises.”

Unlike a traditional shoot, where logistics take up a lot of the cost, a virtual production incurs only a digital cost, says Samarmand. “Producers can save on logistics cost, location permits and time delays. This is more cost-effective once people understand the workflow and what they want to do.”

No boundaries

Fractal is working with several international partners to ensure it can provide services on a par with global standards – its VFX partner is in South Africa, its tracking system provider is in the US, and its service systems are in the UK. Samarmand and Gheysen are quick to reiterate the importance of collaborating with good partners.

“The industry as a whole is seeing tremendous growth, and the partners are what are going to make it successful. Although the technology itself was developed in Europe or in the US, vendor support teams are always available online via Zoom or remote access software. In our case, we had the team from INFiLED and OptiTrack come down to guide us personally. It’s these things that have allowed us to build and deliver a superior product, because any time we ran into a bump in the road, these guys were there to support us,” says Samarmand.

Marco Bruines, Senior Vice President of INFiLED EMEA, says the company is “delighted to be part of this amazing project, as exciting times are coming with the revolution of the XR industry. We’ve been working closely with Fractal and all other parties involved to achieve this impressive, customised set-up. The combination of the seamless curve, high refresh rate and vivid colours of the massive LED backdrop offer state-of-the art 8K resolutions and allow outstanding quality on- and off-camera.”

The human element, of course, remains at the core of the whole process. Fractal is helping to create greater demand for 3D artists, VFX supervisors, colour and vision supervisors, Unreal Engine artists and video engineers.

“XR needs all these talents to generate its photorealistic content, and the challenge now is to find them. There’s also a newfound emphasis on the need for a director right from start of the filming, including the scripting and studio hire process,” explains Samarmand.

“XR and VP rely heavily on art direction and how to realistically design sets to have seamless stitching between physical sets and virtual environment. The new challenge is to have them on projects since day one.

“XR-VP is a tool. What you can do and how you develop it depends on the team you are with. You have to start imagining and writing scripts that fit the use of that tool. Technology is changing every month, allowing us to do more and more, so any limitations will be lifted soon. We are only limited by our imagination and how to use these tools. It is going to take time for directors and storytellers to understand this technology. It was important to onboard film technicians as early as possible to be ready for production. XR-VP is a great ship, and we need a knowledgeable crew to steer it.”

The road ahead

Both Samarmand and Gheysen agree that creating awareness and educating industry peers remain central to the success of XR in production.

“This is more than cinematography meeting technology,” explains Samarmand. “It is putting top-notch technology together with a multitude of critical elements and then balancing them to get a hyper-realistic final shot. If even one element is out of place, the whole workflow will fail.”

Gheysen seconds this. “Most of our industry peers are set in the linear workflow, so to change their mentality and convince them to change the workflow is a big challenge. But filmmakers who visited us have revisited their scripts after seeing what they can do with our technology. They went back and changed their entire approach to the shoot itself, and that’s what is needed to achieve something great. Those who take the time to understand tech are at the forefront, they are the ones who will gain the most from it.”