A virtual world

US-based virtual reality expert, Lewis Smithingham, was in Dubai for DIFF recently, to explore the use of VR technology in the region. In an interview with BroadcastPro ME, Smithingham discusses VR and where it is headed 

Recognising the growth of Virtual Reality (VR) in the UAE, the Dubai International Film Festival (DIFF) held a number of sessions on the state of VR and its potential for immersive storytelling. One of the speakers at the event was Lewis Smithingham, CTO of 30ninjas, a pioneer in VR filmmaking who has worked on VR content around the globe for the past four years. He recently completed Invisible, a VR mini-series with Doug Liman, producer and director of films including The Bourne Identity.

First, can you give us a quick overview of VR?

VR isn’t new; it’s been around since the 1990s. It’s only in the last few years that it has been consumerised. VR is a realistic and immersive simulation of a three-dimensional environment. The viewer can literally look and go anywhere in the film. There are two key points in VR. First, the viewer has to be wearing an HMD (head-mounted display) to be right in the middle of the action or immersed in the film or content. Second, it is completely different for the video storyteller because you give up some control to the viewer and he or she can look and go anywhere in the story.

It sounds dangerous for creatives, but actually they can put cookie crumbs or sounds to guide the viewer in the film. It’s not something that is taught in film schools yet, so most VR filmmakers do an awful lot of shooting and experimentation to find out what works and what doesn’t. But when it’s done right, the viewer experience is really thrilling and rewarding.

For those who didn’t attend DIFF, was there a particular highlight of the event for you?

One of the highlights for me at DIFF was the Embodied Media Project from Japan. In talking with UAE filmmakers during the event, they were just as enthusiastic because they could see the potential for them and the consumer/viewer of tomorrow. Embodied Media’s goal is to create future media technologies that record, share, enhance and create experiences that entertain, enchant and empower us.

Is there really a VR scene in Dubai?

Before I visited Dubai, I thought it might have been in the gestation stage – between conception and birth – and for that, I apologise. Dubai is one of the fastest growing and most progressive cities in the world, and from my brief visit I saw some very progressive studios and marketers who are seriously looking at how they can use the technology to entertain people and promote products.

I know you were in Amsterdam for IBC and am not certain if you had an opportunity to visit the Virtual Reality Cinema, but I would guess that there are probably creative groups working right now to open facilities soon in Dubai. Knowing how progressive IMAX is and how quickly they are moving to open the next-generation theatres, I’m certain the city is on their roadmap.

Why cinemas, you might ask? It is because of the richness of the content you participate in. Every time you put on an HMD and enter an immersive story, it is never the same viewing twice. As I alluded to before, you can go right, left, look up, look down and every time the film is different. The viewer controls the story and tries new avenues. It’s amazing.

I visited Giga Works while I was in Dubai, and they are currently doing some excellent work in the areas of documentaries, travelogues, streaming 360/VR work, and are pushing the technology into marketing consumer and business products/services.

Pixel Hunters is a Dubai-based animation company that I unfortunately didn’t have the time to visit but have been told are doing some great animated VR segments. Animation was tailor made for VR, because you can immerse a person into a virtual world that only exists in the mind of the creators, so you can let them go into outer space or visit the centre of the earth, interact with non-existent characters, run 100kph, fly unassisted and vanquish monsters or your enemy. It’s not just a cartoon; the viewer is right in the middle of everything and it feels and sounds remarkably realistic.

There are other firms I heard of that are using VR to design and sell building projects. Once the person puts on the HMD, he or she can look inside, outside of a building – for example – from the top down, inside and out, walking through rooms that your mind says are real. So you know before the ground is broken that you’re going to really like the completed work.

Are there applications and opportunities for VR in the broadcast industry?

Talking with DIFF attendees and the people at Giga Works, it is already in the experimental usage phase in Dubai. Broadcast opportunities have been in the testing trial phase through most of last year for us, as well as several partners and broadcasters around the globe.

For example, one of our European partners produced several livestream soccer matches with our assistance and then we jointly critiqued the results to provide the viewer with the feeling that they were at the match and in the middle of the action. Most of the streaming concerts and sporting events – including the Olympics in Brazil – have been and are being done in 180. It’s good, because it’s an improvement from standard flat image capture/streaming. But from our tests, we know rich 360 coverage is possible.

The NBA and Golden State Warriors regularly stream 360 coverage of games so you can feel you are right in the middle of the action, not just sitting courtside but able to view a play or shot from every angle.

We’ve been working with Nokia Ozo on projects for Ozo Live. I feel full 360 capture and streaming is necessary to give the viewer the honest feeling of being at the event, viewing the play and shot from every angle. The same is true of streaming a concert or play, because well planned and executed VR capture, production and streaming gives the viewer a rich, satisfying experience that is hard to explain until you see it and feel it.

Networks, studios and specific shows have also been doing 360-degree specials that have been enthusiastically reviewed by the specials’ producers and the audience.

Last year, our firm produced the Conan O’Brien 360 special. We shot the entire show in 360 with multiple camera rigs, with post-production being done behind the scenes to finalise the entire special in record time for delayed broadcast streaming. It is a tremendous effort on the part of the production/post-production team, but the show owners couldn’t believe the outstanding response they received from the audience.

All of this right now is still in the experience development phase at the present time, but I am completely confident that you’ll see more and more of it over the next two years, and soon it will be standard viewing fare. There have already been sporting events and TV show specials produced here in Dubai, and the volume is only going to increase.

By 2018, I am confident that every network and studio will have regular 360 specials, including dramas, documentaries, regularly scheduled shows, concerts, plays and sporting events that people throughout the UAE will be putting on their HMDs to enjoy. Once you experience it, it is just too compelling!

What other uses for VR do you see here in Dubai and elsewhere around the globe?

They say a picture is worth a thousand words, but VR lets you experience a product, a service and a location. The technology, properly executed, I believe, will be a major marketing tool in the coming years.

You can experience a travel location. In fact, in a couple of hours, you could visit a dozen exotic places and almost feel as though you had been there. You can do breathtaking, risky adventures like climbing the Himalayas, skydiving, scuba diving in spectacular locations… almost anything you can think of in the privacy of your HMD. You could take off your HMD and be as exhausted as though you did everything in the location in five action-packed days.

Want to test drive multiple cars but don’t want to waste weeks and months doing it? You can go to a kiosk or stream the content to your home and put them through their paces, so you know exactly which you want before you buy. You can try new furnishings in your home, test out new tools, try new layouts to your house before you remodel and know you’ll be completely satisfied.

Will VR be mainstream production and viewing this year?

Let me first be a little cautionary, because too many people have oversold and overhyped VR, saying it is the complete solution for tomorrow’s media and entertainment industry. The cameras are very good today, the HMDs are good and getting better, and the prices are now very reasonable for consumers. The prices will continue to drop and the gear will get better. As I mentioned earlier, I feel by 2018 everything will be feature-rich and ready for everyone to experience and use without even thinking twice about it.

Over the past four years that I’ve been doing VR, we have tested and used almost every piece of shooting and production gear that has been introduced. The professional cameras available today are very good. People are amazed at how much content we capture when we’re doing a 360 or VR project, and you can easily use 25-50 terabytes of storage a day. It’s an important part of the VR equation and as we shoot in 4K and HDR, it’s only going to increase. High-performance computers and high-throughput, expandable storage are mandatory!

We are still learning a lot about what works and what doesn’t for the viewer. When we produced Invisible, we shot and reshot hours of content, reviewed it – and some worked and delivered the viewer experience everyone wanted, while some didn’t.

Production and editing is a labour for every filmmaker, and the key to a great 360/VR film is that we constantly think about the audience. This is completely new and unique, and requires a different approach for the director and the filmmaker.

The most frustrating part of the 360/VR creative process is stitching [the process by which multiple camera angles are fused together into a spherical video panorama]. While the software is steadily improving, it is still very laborious and expensive – often $10,000-plus per finished minute.

But once you put on an HMD and immerse yourself in really good content… you’re sold.

Why am I sure it will succeed? This is one of the few technologies in history that all of the major players – Intel, Qualcomm, Nvidia, AMD, Samsung, HTC, Adobe, Avid, Google, Facebook, Sony – have weighed in and invested in early. That has had a profound effect on how quickly solid and economic products have become available.