'The Voice Middle East' has continuously raised the bar in MENA broadcasting with fresh elements added to the show each year. Visuwalls walks Vijaya Cherian through some of the key technologies that led to successfully creating AR for the show.
The Voice Middle East, MBC’s much loved and popular Arabic version of the famed Dutch format, has over the years, become an intense and emotionally charged show as participants battle it out on stage to be crowned the next voice sensation of the year.
Each season, the production teams and the broadcaster aim to keep the show refreshed to continue to draw the millions of viewers it has attracted since it debuted in the region in September 2012. In one edition, that was aired earlier this year, the teams successfully introduced Augmented Reality (AR) for the first time into the show, with sunflowers, words and letters extending the physical environment and adding a brand new dimension to the show. The iconic hand that represents the voice was also recreated this time in AR.
Beirut-based technology solutions provider Visuwalls, which has worked with show producer ITV Studios Middle East on the last three seasons of The Voice Middle East, by providing the set design and construction, automation control, custom software, motion graphics, LED screens and media servers as well as any additional lighting equipment and scenic lighting required, was responsible for executing the AR vision of the team. This was also incidentally Visuwalls’ first AR project.
Nicholas Malbon, Head of Department for Media Servers and Augmented Reality at Visuwalls, who was primarily responsible for the AR execution, says a good technical strategy and robust solutions are critical to getting it right.
“First, you must have an idea of what you want to achieve,” says Malbon.
“We looked at the AR element for the show, offered suggestions and what was achievable. Your technical strategy must be sound. From there, you need to first get your cameras and lens sorted out. We had a Mo-Sys star tracker mounted on the techno crane and a static camera with a Mo-Sys E-sensor Head. We chose Mo-Sys because it is independent of everything else. Once you map the studio, you can map that to any device. Mo-Sys has a very reliable and robust solution and once it is set up, it is pretty simple. The calibration is probably the most time consuming and once that is done, you can sail through. They also offer very good support and are extremely helpful,” Malbon says.
Besides the Mo-Sys tracking solutions, the core equipment for this project included a host of Ross Video solutions including Ross Voyager, UX and Video Routing as well as Visuwalls’ own custom servers with AJA Corvid 88 HD-SDI cards.
The tracking data captured by Mo-Sys was fed into a Ross UX system at the front of the house with Ross Voyager servers — that were remotely operating from the rack room of the studio — taking the camera footage and compositing the AR. All AR compositions were internal to the server, so the gallery simply had a feed with AR and a feed without. Ross UX was used to control the servers and manage tracking data.
“Synchronising all the data and video streams can be quite challenging. Good solid genlock is a must in such a project,” explains Malbon.
One key element that the team had to ensure worked well was the lens, which Malbon reiterates, “is critical within any AR setup”, and points out why.
“Within a virtual solution, the lens is perfect. Within the software, the lens is perfect. But in the real world, lenses are good but not necessarily perfect. You need to have a good lens calibration file and good lens data, or it just looks wrong.
“You have to get a distortion table, which matches the real lenses to the virtual lenses. This is where a company like Ross Video comes in. They have several real lenses mapped already with the virtual lens. The reason why this is important is because every lens is unique. When they map a lens, they model the lens. In the ideal world, you would send the lens to Ross Video and they would custom map it for you. That would be an expensive proposition if you were using a lens just for one show but now, we have invested in our own lenses.”
Visuwalls has invested in new Canon lenses and Grass Valley cameras to provide a pre-calibrated service.
“Now that we have our own lenses, we will get them properly mapped to ensure complete precision,” explains Malbon.
The next step to executing an AR strategy lies in the software.
“Again, we chose Ross Video here because it is based on Unreal Engine. You could argue that most available software solutions today are based on Unreal Engine but we went with Ross because of the way they have configured the system, how they interface it into the unreal engine platform, how they track the system and how they manipulate things within their projects. Their control software is really good. It all boils down to interface and support.”
Malbon admits that having never bought AR software before, the company was looking for a straightforward solution, and claims that Ross Video fit the bill.
“The Ross solution was pretty straightforward, but you don’t lose any of the power. Once the interface is set up, pretty much anyone can operate it and that was the plan. We wanted to ensure that the operations would be easy for anyone without involving a big learning curve,” he points out.
Once that is sorted, it is a matter of getting the 3D artists to make beautiful models, says Malbon.
Integrating the AR into the graphics and the set was the most fun part of this project, says Daria Bertinelli, Head of the Graphic and Motion Design Department at Visuwalls.
“As creatives, we at Visuwalls have dreamed many times that our graphic creations could somehow come to life, and this is what is slowly happening with the help of AR.
“We constantly work closely with the Art Director, who gives us an idea of the vision he wants to achieve. We then translate that vision into something visually and emotionally appealing. Doing that with the integration of AR was quite challenging as it was our first time, and technically we had to meet some requirements that don’t always agree with our imagination. But that was part of the challenge.”
Bertinelli says the team “timidly” started with some virtual graphics that “appeared to be flying around the studio only at certain moments” but then gradually gained the confidence to “work with something more obvious on stage towards the last episodes”.
“Anyone who has watched the broadcast from home will have seen that the singers were surrounded by something special, not obtainable with lights, projectors, screens, and with the classic equipment of a studio. Of course, we didn’t want to create an exaggerated environment. We needed to maintain consistency with reality and adapt the setting to the context,” she remarks.
The team used some 3D software solutions such as Blender, for the main modelling, texturing, and animations. Blender helped prepare the 3D scenes for integration into the live environment.
“In this step, sometimes everything works immediately, while at other times, there are continuous adjustments to make until the expected result is achieved. Then it’s time to integrate this within the larger picture to ensure a seamless visual experience. This starts during the setting up inside the 3D software, then, as Nick mentioned, we use Unreal to manage the imported 3D models, creating the right materials, lights, shadows, animation and all the programming necessary to match the actual environment as close as possible.”
Bertinelli goes on to add that AR has already been used extensively in the video games and film industry. “It was missing only on television and we are happy we have been part of this global transformation.”
But AR has its limitations “because it is all about overlay and influx,” cautions Malbon.
“You have to be very clever about how you position things and how you move the camera past objects. Once we had a brief from the creative team, we were doing it in conjunction with motion graphics. We would take 3D models from the graphics and extract them and place them within the AR frame. We would then enhance the graphics with some AR elements to make the picture better. We were really keen to keep it simple so as to ensure that we did not overwhelm the director,” adds Malbon.
The result was sunflowers extending and adding a further flourish to an already glamorous environment. In some instances, words floated across the screen. Of course, with a new project and wanting to ensure that everything went smoothly, the team ended up creating a lot of AR that they did not eventually use but it was a great learning exercise.
“In the past, we have always dealt with physical construction. As AR was new to the show, we were trying many things out – some never made it to the final broadcast due to time constraints or simply because of artistic choices. In hindsight, we know now what questions to ask, what to propose or not to propose the next time,” explains Malbon.
He goes on to add that while some things worked quite well, “there were others that required specific camera movements to work and those were tough”.
“We had a couple of things that were too close to the performers and that became challenging. We discovered that if the camera was even slightly in the wrong place, it could break the illusion. There were a couple of areas, where we had to clip the frames. As AR progresses, there are tips and tricks we are likely to figure out. Lens calibration and tracking solutions are very important and having your virtual and real life perfectly aligned will make or break the frame. You can cheat on some things but not the above,” adds Malbon.
Overall, the project was so successfully executed that the company invested in a lot of new equipment to ensure that Visuwalls could make it part of the portfolio of services it offers, CEO Pietro Haj says.
Visuwalls typically provides turnkey solutions for any event primarily on the live side, and these include props, the set, LED screens, lighting equipment, graphics, customer software, automation for the chairs and so on.
“This is what we have typically provided for The Voice Middle East and this is the third season that we are working with them. MBC took care of the lighting, cameras and so on. We are mostly into AV and lighting and can be considered a good production resource, but in recent times, we have found ourselves working on more and more broadcast projects as well,” explains Haj.
“We have worked on Arabs Got Talent, The X Factor and several live events such as the Opening Ceremony of the World Air Games in Dubai and Julia in Concert in Lebanon and Jordan. We expanded into supplying lighting and projection equipment. The move into AR came from a realisation that it was the up-and-coming technology, and we already had a graphics team who often worked in 3D anyway,” explains Haj.
With warehouses in Lebanon and Saudi Arabia as well as a well-supported technical department and construction unit, the Lebanese solutions provider has a 360-degree solution for live events. Haj comments that in the future, the company will also probably invest in new media servers that will integrate into its AR world.
AR projects are here to stay as it has allowed for enhanced storytelling and greater interaction between participants and graphics objects. With companies such as Facebook, Intel, Alphabet, Inc., Comcast Ventures, and Samsung Group investing heavily in the AR market, we are likely to see more such deployments moving forward.
