Creating mixed reality animation at Blink Studios

Claiming a global first as makers of a mixed reality animated series, the team at Blink Studios Dubai shares its challenging journey to master the new immersive technology with BroadcastPro ME.

There is something about the animated characters Karim and the lantern, Noor, on the wooden floor of Blink Studios, beside a manic vending machine spewing chocolates, that blurs the distinction between a hard-nosed trade journalist and a six-year-boy at Comic-Con in Dubai.

“‘What is he doing?’ his mother asked, surprised to see her son trying to pick invisible stuff off the floor,” recalls Nathalie Habib, creator and Executive Producer at Dubai-based Blink Studios. “The boy looked up at his mother and said, ‘Mom, I am trying to pick up the sweets falling out of the vending machine.’”

That the boy could see his mother while still engaging with the animated action on the floor of ComicCon, was the prime reason Producer and Director Hani Kichi and his team at Blink, including Habib, decided on the medium of mixed reality (MR).

“With virtual reality, the child enters another world and can move around, potentially hurting himself. Mixed reality allows the child to engage with content within his or her own real environment,” says Kichi, in conversation with BroadcastPro ME.

MR is just the newest technical vehicle for Blink to take forward an idea conceived by Habib in 2016, to convey social, emotional learning, through animated characters Karim and Noor. What started as a linear animated series has now morphed into an MR series supported technically by Microsoft’s HoloLens, described as the first self-contained holographic computer, enabling users to engage with their digital content and interact with holograms in the world around them.

BroadcastPro ME met the Blink team at a very busy Microsoft stand during GITEX 2017. As a steady stream of trade visitors are wowed by the virtual world of Karim and Noor , Habib briefs us on how the collaboration came about.

“A chance meeting in Abu Dhabi in 2016 with Playing Forward, a Los Angeles-based technical development company, started it all. We had just completed the first linear animated season of Karim and Noor and we were prepping for the second season. When they watched the episodes, they were keen to have the series on their global platform as mixed reality content. We were then introduced to the world of the HoloLens. The company was working on mixed reality and enhancing what Microsoft had already built.

“The revenue-sharing, co-production deal that we struck required us to do the production, with Playing Forward taking on marketing and distribution. Season 2 of Karim and Noor is one of the four intellectual properties for global distribution on Holotoons, an app for kids to immerse themselves in edutainment content.”

With Kichi’s background in engineering and 3D artists at Blink having gained experience in CGI and visualisations of architectural walkthroughs, since 2005, the team found it “easy to embrace mixed reality”.

The actual journey from linear to mixed reality, however, is described as “funstrating” by Lea Badro, Creative Producer at Blink.

“We have been producing content ranging from linear to 3D for the last 12 years. While the commercial arm helps us keep the lights on so to speak, we have travelled the world showcasing our own IP. We have co-produced a number of shows, such as Iftah Ya Simsim Season One, the Arabic adaptation of Sesame Street, and Driver Dan’s Story Train, among others. However, when it came to script writing for mixed reality, the learning curve was steep but fun.”

The questions the two script writers hired from Ireland asked were the first indications that the team at Blink was not just looking at new technology, but a mental shift in story-telling, Kichi says.

“Mixed reality is a new way of consuming content. It is a new way of producing content. It is a new way of storytelling. The international writers we hired for the second season came back with a list of questions that appeared laughable. But the questions were completely valid, considering we were grappling with new technology where virtual characters engage the viewer in his or her own environment. Where do they come out from? How do we start? Can we cut to another scene? Can we fade in and fade out? Can the character go out of frame? There is no frame. Can the time pass quickly? Can you fast forward to another scene?”

A typical six-week timeline for each two-minute episode of Karim and Noor in the linear world stretched to two whole months of Skype sessions between the Blink team and their counterparts in Los Angeles.

On the back and forth across continents, Habib says: “While the script writers grappled with questions as to whether Karim can shower or not, we had to get our 3D artists who had worked on Maya and 3ds Max up to speed. The technical team from LA cross-checked our work, advising us on how to deal with the production challenges.

“The team had to deconstruct their traditional approaches and start with basic building blocks. Initially it was frustrating, but then there was the mental switch. The production pipeline started to emerge. Our first episode took the longest time because it was trial and error, with a lot of back and forth to get it right from the visual perspective, story perspective, production perspective and so on.”

The 10% that the Blink team takes pride in investing in internal projects was spent largely on training its 3D artists to restructure their thinking in terms of rigging, texturing and modelling for mixed reality while introducing them to software such as Unity and Microsoft Visual Studio.

Understanding the technology was centre stage, but Kichi and Habib knew they still had a story to tell.

Understanding the technology was centre stage, but Kichi and Habib knew they still had a story to tell. Only now, in mixed reality, the curious Karim and the lantern robot Noor were imparting their values in the viewer’s living room or bedroom.

On the radical shift in canvas, Kichi says: “When we started working with the team in LA, we did not know much about the HoloLens. And while they knew the technology, what we discovered was because it was the first time they were tackling storytelling, our questions posed challenges we both needed to find answers for.”

Mixed reality is defined as an extension of VR and AR, creating environments in which digital and physical objects can interact. This definition was cold comfort for the scriptwriting team faced with issues of Karim having to shower or the need for time to pass, issues that are solved easily in any linear production.

Habib explains the production conundrums: “For example, Karim and Noor can’t just come out of nowhere. You can have Karim drop into space, but you have to write, ‘He drops into space and lands on the ground.’ While there are limitations, new avenues of storytelling opened up. For the passage of time, we could not have the character say, “Oh, see you at three in the afternoon”.The dialogue-free nature of Karim and Noor placed an extra burden on us. Time lapse was used to show passage of time.

“At the end of the day with mixed reality, if you are not engaging with your reality, why do it? You need a story that makes sense in your reality. Mixed reality is not a gimmick. The 3D props we created saved the day for us. Our characters engage with the props, taking the story forward.”

The mixed reality workflow, currently down to six weeks from the original two months, began to emerge – albeit a workflow with no post-production, Hani explains.

“We don’t have traditional post production with mixed reality. In linear production, you have compositing software. Here, you are doing real-time 3D rendering. Everything needs to be prepared ahead of time. Your texturing is the final texture. Real-time rendering is the final render. Noor, the lantern robot with light emerging from his belly, posed a particular challenge for us. Light is a visual effect that we normally do in post-production. With mixed reality, we need to do it as part of the programming.”

While the world of AR, VR and MR is set to become a $165 billion industry by 2020, with MR described as a paradigm shift by experts, not many fully understand this fast developing digital phenomenon. Describing the process as plunging into the abyss with just questions and more questions, Habib says: “I told Hani we need to switch off the light in the script. However, unless the viewer switches off the lights, there is no light to switch off. It is weird. There are tricks you can deploy. Light is projected through the screens in front of the eyes, and if you add a darker colour, the light can be dimmed for that segment. The technical roadblocks did take time and effort in figuring out.”

From a technical point of view, producing in mixed reality means the team has to walk the fine line between normal TV production and gaming technology.

Kichi elaborates: “When you go into normal production, the computers with their massive RAM can handle any modelling object regardless of resolution. With modelling for mixed reality, you need to be careful – with a high polygon count, you will hit the ceiling imposed by the current HoloLens technology. Polygon, in modelling parlance, is the number of faces inside the topology of an object. You need to be specific and smart in using polygons and where to put an extra polygon. If you exceed the limit, the content will run with difficulty on the HoloLens.

“At the same time, we are not producing a game. In gaming, people don’t really care about the characters. With Karim and Noor, we are still producing an episode where the character needs to engage with the audience through storytelling. So there is that fine line we need to walk. For instance, texturing has to be more enhanced than what an average game demands. Texturing is the process of colouring and laying the details on top of the 3D objects. Questions such as what is the type of material, is there a reflection, is it rough … in normal production, it is left to a traditional rendering process. With mixed reality, it needs to be done in real-time render as the audience engages with the virtual character.”

Badro adds: “With the 3D nature of the action, script writing is not typical. When you are writing that Karim is doing one thing, you need to write how Noor is reacting as well. The storyboard artist needs to be aware of all the characters at all times.”

Experts say believability is a core design principle in mixed reality. Transparent lenses, spatial sound and an understanding of the physical environment, Kichi explains, allow holograms to look, sound and behave like real objects, evoking an emotional engagement from the viewer. As we don the HoloLens and watch Karim and Noor grapple with the vending machine, we could choose to sit on any of the sofas, stand by the door, resize the characters or bend down on the floor and watch the action. There is no telling what we will do as a viewer and therein lies the challenge, according to Habib.

“The number one challenge of mixed reality is you don’t have a camera with you. The child can decide from which angle he or she wants to watch the action. He can put the content wherever he wants. He can place the characters on the dining table, on the floor or on the bed, and watch the content. While we can add some virtual elements as props, the kid is seeing everything in his environment.”

Mixed reality brings into play the critical concepts of spatial mapping and spatial sound, says Kichi.

“If a virtual ball drops into the scene, it can first bounce on your table before touching the floor. A real-time scan by the HoloLens allows the unreal to become part of the real. As the viewer moves away from a sound-producing virtual object, the sound will lower in volume. Spatial mapping and spatial sound allow the experience to be immersive.”

Mixed reality sounds and looks like magic, but Kichi assures us that it is all down to analysing data via the HoloLens.

“The HoloLens scans your environment and creates a 3D model of your reality. In this model, we know where our holographic character is and where the virtual objects are. When the person wearing the HoloLens moves in his environment, a virtual camera literally replicates his movement – so when he gets closer to the holographic character, the virtual camera goes close to its 3D model.

“The HoloLens is a computer, at the end of the day, but it is a computer that enables you to interact with the content within the Windows application in a holographic way. From your internet browser to an Excel sheet, to video and photos or social media, you can literally put all that on the table or on your wall and work, design or be entertained.”

Habib likens the experience to the Iron Man character and his virtual screens, without the world-saving abilities.

“Swiping the phone was an alien concept to us not long ago. I am sure hand gestures of air tapping and blooming that you need to get your HoloLens working will become second nature soon. Forget gamers and kids, the average housewife will end up wearing some sort of headset in the very near future to watch her favourite cooking show.”

From car design to medical education, the HoloLens is already being applied in various fields even as the team at Blink is poised to launch Karim and Noor on an immersive journey into space.

There are constraints, Kichi admits, citing a limited field of view and RAM space, among others. He, however, points out that the HoloLens is still part of a developer’s kit and is being improved. Happily, MR now comes on different platforms and devices. Google Tango is a different way of experiencing MR through tablets and phones, Habib says.

“You can project Karim and Noor from your phone on to any device and you can watch it as enlarged as you want. You are still experiencing mixed reality, but without the headsets, making it ideal for a younger audience. Imagine the child projecting Karim and Noor on his bed and watching an episode – the potential is endless.”

With commercial interest already buzzing around MR, the team at Blink is optimistic about the future. And unlike the Middle East trend of adapting to new technology after it is established globally, the team at Blink has decided to take the plunge even as the technology is still evolving.

Kichi says: “We knew from day one that the ROI would not be immediate. We decided to proceed because of a passion for innovation. Secondly, we believe that this is the future of content.”

Justifiably proud of being a global pioneer in developing MR content, Habib predicts: “We have the production workflow in place, having unravelled the secrets of the technology over one and a half years of painstaking trial and error. In two years, this technology will be mainstream and we will be ready.”