With the Covid-19 outbreak creating a huge appetite for binge-watching services, entertainment platforms experienced massive traffic, compelling media houses to ramp up their in-house technology services to cope with the spike. As a result, cloud, IP solutions, remote productions and other new technologies that may have been on the to-do list were moved to top priority by some broadcasters. The BroadcastPro Tech Summit addressed some of these new developments through panel discussions and presentations.
BroadcastPro ME’s two-day Tech Summit included several takeaways for broadcasters, content providers and backend technology suppliers. The virtual event held over June 14-15 covered topics on cloud adoption, migration to IP-based systems, coping with the rush on streaming services and maximising the potential of data to offer a better user experience. Besides panel discussions, the first day featured presentations from OSN and Pebble, while day 2 included an interesting presentation on cloud playout and streaming from qibb by Techtriq.
Peter Riz, CTO, and Zahra Zayat, Senior VP – OTT Digital and Telco, spoke about the relaunch of the new OSN streaming app and presented a case study of the app’s development and journey so far.
Pebble Director of Sales and Marketing Alison Pavitt and CTO Miroslav Jeras held a discussion about solving the challenges of connection management in an IP world. While they agreed that there are many noted benefits to the wholesale move to IP, they also admitted that it comes with the “unwelcome spectre of increased complexity”. In this context, they looked at how the industry could navigate from one state to the other seamlessly.
Day 2 of the Tech Summit began with a presentation by Hans Baumgaertner, Senior Solutions Architect Cloud at qibb by Techtriq. Playout solutions are increasingly moving to the cloud owing to the speed, scalability and cost efficiency of launching pop-up channels and distributing content across different platforms. Given this perspective, Baumgaertner took us on a journey to show how this can easily be achieved with out-of-the-box SaaS solutions, and how an individual cloud playout architecture can also be built with a vendor- and infrastructure-agnostic cloud application management and orchestration platform.
The keynote panel at the BroadcastPro Tech Summit brought together industry leaders to share their experience during the lockdown and the best ways to leverage the cloud for operational efficiency and greater elasticity, while ensuring it remains budget-friendly. We had end users and solution providers on the panel, representing regional and international players, which ensured a good mix of opinions.
Nicolas Hans, Partner, Middle East and Africa for Broadcast Solutions, moderated this panel, with Faraz Arshad, VP of Technology & Infrastructure at StarzPlay; Manish Kapoor, VP of IT and Broadcast at Zee TV MENAP; Chris Fellows, Senior Manager, Solutions Engineering, International, Zixi MENAP; and Jared Timmins, Senior VP of Solutions at TVU Networks, joining the conversation.
During the lockdown, StarzPlay experienced 400 times the traffic it was used to and coped with that “tsunami of traffic” primarily because of the “efficient cloud-based CDN and hybrid solutions” it has had in place since 2017, maintained CTO Faraz Arshad.
“With Covid, user engagement behaviour and consumption patterns have changed. So, we began to offer recommendations based on their consumption profiles. We looked at churn and the longer life cycle of the customer. Those computer recommendations were generated on the edge on local servers instead of at the origin, and that was hugely beneficial for a consummate user experience.”
Zee TV in Dubai transitioned to a complete remote operation with IP solutions, and moved to the cloud during Covid to avoid any negative impact on its production and scheduling, Manish Kapoor said. “Due to our operations using a hybrid cloud solution, our post-production was done in India and cloud playout was in another part of the world. We could deliver our channels to the US and other countries.”
But Covid didn’t just help media houses accelerate the move to the cloud; it also led to the development of new business models, commented Jared Timmins from TVU Networks.
“Covid took away that 24-month transition period that may have been available before to move to the cloud, and now that many have moved, I don’t see any reason to go back, given the advancements we have been able to provide operators and content owners to do things differently. We are no longer bound to the old workflows.”
With cloud transitions being more than just flicking a switch and requiring a move from a CAPEX to an OPEX model, everyone agreed that a complete shift in mindset was critical to efficient cloud use.
“It’s not a lift-and-shift job, so you won’t get benefits of using the cloud if you do that,” Fellows explained. “Cloud is built for elasticity. The first step would be to look at the different vendors with whom you can host. Most clients want redundancy across different vendors with a simple user interface that spans all those vendors. It is critical to identify the right vendors, get a good user interface and reduce the compute requirement. By using a sensible streaming codec and protocol, you can reduce that requirement and reduce your bills each quarter.”
Kapoor agreed that Zee had seen significant cost benefits from its move to the cloud. “We started delivering channels over IP using the cloud with different protocols like Zixi or the SRTs, and during Covid we increased the pace of implementation. We realised significant cost savings in terms of satellite space, uplinking and even fibre to a large extent. We plan to implement more features of live content replacement on the cloud in the immediate future.”
The discussion then moved to data and how it is processed to build better business cases and user experiences.
“The important thing is to classify the data you collect so it can become useful to the stakeholders in the organisation, from content to marketing, and you can see what features to offer to your audience,” explained Arshad. “It helped us achieve our KPIs in terms of discovery, engagement, performance of the customer, sentiment analysis, pricing and so on.” He added that it also helped them understand how people in KSA, for instance, skewed towards some packages while UAE residents showed a different preference.
Hans asked: Has all this made broadcast engineering a data scientist’s business?
“People typically like to see video on a monitor wall, although you get more information from data off the stream,” explained Chris Fellows. “Zixi, for instance, is introducing our intelligent data platform and can look at jitter on a network connection and detect pits in a video stream – all those things you can’t get from a monitor wall. If we use that information in an intelligent way with machine learning (ML) and AI to predict whether there are problems on that stream – say, for instance, if it will go off in six hours … that’s another way in which data can be used.”
Timmins seconded that and explained that data, cloud, AI and ML are all interconnected, part of a new ecosystem that could take broadcasters to the next level.
“It’s about how we handle data, not just around the customers or the data integrity of the streams, but also data based on EDL information of when you made a switch between two sources and why you made the switch. At a lower level, it’s what we can do with object and facial recognition, transcription, even scene recognition for sports. AI is really moving so fast in terms of what we can do with it. If you look at what Google and Amazon and Microsoft are doing with it … these tool sets are already available, which we are not fully exploiting in broadcast. When you talk about the financial element of the cloud, it changes everything. Before, we had to build facilities for the worst possible scenario, where every input was maxed on a router, and every switcher and operator station utilised all the time. Now, I can build functional elements and spin those up as needed. It is about the art of creating microservices and creating them in a way that works with customers differently than just saying that this is a product that everyone must consume. We can now build adaptable technologies and solve our customers’ problems. Working with them helps them to figure out how to revolutionise their business into the future.”
He added that one worry is that people will simply take things as they are in a traditional set-up and move them into the cloud.
“My biggest concern with the cloud is that we are just taking the routers and the switchers, the replays and the playout engines, and taking them exactly as they are in a traditional set-up, and putting these in the cloud and keeping the same operational paradigm. We are no longer relying on multiviewers and people to understand how our systems are performing. Our business choices will be informed by a new level of data and collaboration and our ability to scale up and down. That whole operation will breathe in the cloud.
“We have a brand-new template that we can work on. Since broadcasters have moved to the cloud, we don’t need to go back to what we had in the traditional set-up. We have way more cool things we can set up and work on if we get rid of our old operator mentality.”
The panellists agreed that changing mindsets across the board is critical to successfully implementing a cloud solution within a company. Fortunately, as a young entrant, this was not difficult for the likes of StarzPlay, since most of the people it hired already worked in cloud environments.
“We never had to convert anyone from a traditional broadcast setup. Most people we hired already worked in the cloud, so we didn’t experience those challenges. To automate our workflows, we worked with people who had worked with ML. We focused more on automation and training our staff on the overall delivery of the project. We looked at ML models to deliver more assets with automation,” explained Arshad.
Fellows commented that Zixi takes three billion data points from video streams and “you can’t get that from traditional SDI and infrastructure”.
Timmins explained how new experiences are being shaped with the new cloud ecosystem.
“We are building new experiences, and a lot of it has been forced upon us. We see a lot of user-generated content (UGC) taking over with its own form of infrastructure. With sports teams, you can create a whole new tier that you can sell because people can have a closer experience with the players. Content is becoming bi-directional – not just voting and polling, but allowing audiences to interact with the players. So, for instance, you can high-five people on the stands, or the players can drop in on the box seats and chat. Everything is moving towards a personalised experience.”
He pointed out that there are so many disruptions in the cloud.
“If the broadcast industry doesn’t see the opportunity, small niche startups or large cloud companies will take over, because they are coming at it from a different angle. If we don’t treat live production in the cloud like game engine companies treat video games in the next five years, we will have missed the opportunity for broadcast to transition to the next level. The concept of having fixed cameras, fit-for-purpose gear that you spin up for just one kind of utilisation case with a broadcast-to-many output will not sustain the revenues or the growth that broadcast companies need, moving into the future.
“Today, our phones can do what broadcast cameras do today with 5G components. In five years, we will not know what broadcast is doing. Deepfakes and everything that is happening outside our industry is something for us to look at. We should use the new toolsets instead of copying yesterday in the pursuit of tomorrow.”
Although regional broadcasters have not exploited AI’s potential to its fullest, its ability for audio detection, violence and nudity detection, facial recognition, scene comprehension, removing people unauthorised to be on shows, the ability for automated clipping and so on will be hugely beneficial in a region that is particular about culturally sensitive content.
Fellows pointed out that monitoring also plays an important role here.
“We need to monitor those linear streams and the software in the cloud. You need a decent control plane to be able to do that, and you want to limit the number of systems with just a single platform monitoring all that. So, install a system that can utilise everything without reinventing new monitoring solutions, and have monitoring for each system.”
Some of the key takeaways from the discussion: the move to the cloud is inevitable; data is the key to unlocking the mind of the customer and ensuring more efficient processes; exploring AI and ML beyond its basic functions will lead to greater success; and the industry’s tech approach will be more micro-app and talent-based, rather than skills-oriented. “I don’t say we throw out the old, but we have a new world to explore and can reinvent the businesses, whether it is sports, playout or OTT – and we must be ready for that,” Timmins concluded.
PANEL 2: THE EVOLVING MENA OTT STORY
The second panel took an in-depth look at how tech teams could help improve the user experience and reduce churn within an OTT ecosystem through data, recommendation engines, payment gateways, delivering a broadcast-quality experience and exploring regional telco partnerships. Moderated by Suhail Ahmed, CEO of One Diversified LLC, it brought together Bachir Boumediene, Director Streaming & IPTV – Telecom Partnerships MENA, OSN; Barry Mehdizadeh, CTO, Mawarid Media & Communications Group; and Salloum El Dahdaah, CEO of White Peaks Solutions.
Data was hailed as all-important to improving the user experience on OTT platforms, with artificial intelligence (AI) and machine learning (ML) serving as the tools for more accurate user profiling and recommendations, to ensure less churn and greater stickiness.
The panellists also agreed that data is the perfect sales tool to help make good business decisions. With data, a platform provider has greater access to who is viewing what and when, enabling media houses to better segment their users intelligently and make recommendations.
“Without data, we cannot drive the ship and see what is happening on our product,” commented Boumediene. “Using data in analytics to support our ML and AI in the backend helps us offer a much better customer experience.”
El Dahdaah stated that it is so important that “data analysis is the centre of our consultancy” – without it, “you are operating completely in the dark”.
In analysing how to create a bespoke personalised experience so as to “create that immortal customer who never jumps ship”, as Ahmed put it, without causing search fatigue, Mehdizadeh said that to make more successful recommendations, platforms need to go beyond personalisation to analysing viewers’ moods.
“People on a weekday may behave differently from a weekend or have a different mood on a weekend with friends compared to without. You listen to different types of music depending on your mood. The same is applicable to viewing as well. Moods change even during the day, so the ability to gauge that and make a recommendation would likely yield better results,” he said.
El Dahdaah said his team already applies this to some extent. “When we see a lot of navigation on the first page and a person has been searching for five minutes and not started watching anything, a pop-up appears asking, ‘What is your mood like, what kind of movie do you want to watch?’ and so on, and the platform starts making suggestions based on that.”
In the case of OSN, Boumediene said the new streaming app has up to five profiles to cater to different kinds of household, as well as different kinds of appetites within a household. OSN has also invested heavily in improving its search and implementing smart search “so you can search on the app by the actor, director and what you are viewing … This ensures more stickiness from a user perspective and more recommendation. We can push more content based on what they watch.” In case anyone forgot content in the midst of a tech discussion, El Dahdaah hastened to add that a platform can have the best technology and personalisation elements, but is only as good as its content: “If you don’t have a good and varied portfolio of content on your platform, nothing else matters.”
Social media was also lauded as a great marketing tool for generating interest in specific titles on platforms. Boumediene, for instance, cited Friends: The Reunion as a case in point for OSN. “There was a massive buzz around Friends: The Reunion on social media. People hopped in just to see that.
El Dahdaah cautioned, however, that social media is best reserved as a tool for marketing purposes. “I advise those in OTT to use social media only for marketing and feedback. Otherwise, the platforms are getting your users’ data but you are not getting much in return. It’s not the right platform for a sign-up.”
With several platforms seeing an opportunity to entice users to their services through games and a reward system, Ahmed asked if panellists were open to using games to lure more subscribers. While everyone agreed about the natural tie-up between content and gaming, El Dahdaah remarked that gamification is less applicable in an SVOD environment; it is a “better fit for sports platforms and may apply to AVOD for engagement”.
Boumediene described the situation as “tricky”.
“Games are also important to kids. You want to create the right content for kids, and you want to make the space pleasing and engaging, but you also don’t want to create addiction – and there is just a thin line between the two. This market is not stringent in terms of what you show kids. Making sure the content that you are providing meets the social and ethical needs of the community is important. There are people who work with us to just ensure that our kids programming is good,” he said.
The discussion then turned to payment gateway and its regional and legal challenges. In a region where credit card usage is still quite low, most streaming services have had to explore other payment options to woo new subscribers, such as partnerships with telco operators, e-vouchers, and weekly and monthly payments through their mobile services. The other big challenge lies in the fact that while the wealthier GCC countries are on a par with global players on several counts, including bandwidth connectivity, the rest of the Arab world is way off the mark.
“We work in economies that are driven by cash and don’t have digital payments to the scale that we need. So apart from Google and Apple integration, we are ramping up our telco integration, because this is huge. We have also gone with e-vouchers that have established a distribution system. Being able to enable e-vouchers through them, whether it is cashless or cash payment, has been hugely beneficial,” explains Boumediene.
“However, we need to recognise that there is a massive opportunity in terms of subscribers, but connectivity issues remain a challenge. From OSN’s perspective, our core market is still the GCC, so we haven’t faced any issues just yet. As we ramp up our efforts in other regions, CDN challenges will start coming up.”
This brought the discussion to the current challenges with CDNs and the lack of support from telcos, a sore point for several reasons with most broadcasters. The lack of a good regional IX to access all the GCC ISPs and the lack of a commercially viable pricing point from regional telcos has led to many challenges for local media players.
“There is no GCC IX. I want an IX where I can access all the GCC ISPs. When we reach that point, we will be able to provide an amazing quality of service,” commented El Dahdaah.
Also, unlike their global counterparts, regional telcos have not been open to exploring better bandwidth pricing options with media houses.
“Telco operators need to look at media houses from a different perspective. Broadcasters cannot provide 40-50% of their income for broadband. Technology is only around 10-15% of their costs. Content eats into most of their investment budget. If a telco is serious about being a partner with broadcasters, it must cut their cost and help them with more sensible prices,” El Dahdaah said.
The panellists added that there is global disruption in the telco space – if local players don’t get their act together, they will lose a good opportunity.
“We know that the likes of Starlink are emerging as a global player, and so is Amazon. What if they come and give Starlink free with Netflix as a bundle? Telco operators will lose audiences. Telcos should see things from a different perspective before it is too late,” El Dahdaah cautioned. He urged the telcos to “work with us and provide us with the space to provide nodes at a good rate, so we can in turn offer a good quality of service”.
Boumediene said, “The business case for telcos is straightforward … The ongoing challenge is that some telcos are trying to defend their market position and they are proactively trying to develop their own OTT and apps to cater to this space. Slowly but surely, they will conclude that they must focus on what they do best … if they established their connectivity and sold their core services, that would be better, rather than competing with OTTs and taking on their roles. Partnership and aggregation would be best, so they become more relevant to their customers as the ecosystem evolves.”
Mehdizadeh seconded this, adding that partnerships between OTT players and telcos will become increasingly relevant. He called for greater collaboration and cooperation.
“Telcos should do what they do best: selling broadband, core access and so on. It won’t always be a case where telcos need OTT. Both need each other, so they must see how they can leverage each other’s strong points to offer a better quality of entertainment to the MENA region,” he concluded.
PANEL 3: CLOUD IS THE FUTURE
Most solution providers had been touting IP and cloud solutions for the preceding couple of years, but these truly became invaluable during the Covid-19 outbreak, when remote working became the norm for most companies. This panel aimed to analyse how companies in the local market adapted to the new normal, what challenges they were up against in considering remote operations, and what tweaks to their in-house technologies were necessary.
The moderator was Hasan Sayed Hasan, Managing Director of UAE-based consultancy Master Media, and the panel consisted of Raad Haddadin, Head of Engineering & IT at Roya TV, Jordan; Jean-Claude Rahme, Head of OB Engineers, LIVE, Abu Dhabi Media Company; Deepraj Sandhar, Regional Manager at Ross Video Middle East; and Peter Van Dam, founder of Audio Video Consulting UAE.
It was established at the beginning of the panel that Covid-19 posed a lot of similar challenges, but also some that were unique to each panellist, and that each had dealt with them differently. All of them represented companies that had to work remotely to varying degrees, and use their time differently.
In the case of Roya TV, the editorial and graphics team were asked to work from home, with equipment provided to ensure they could function effectively. However, engineering and operations staff had to continue work from the office.
“We separated them into three groups, so if anyone had Covid, we could isolate just one group and the rest could continue,” commented Raad Haddadin.
Most Abu Dhabi Media Company (ADMC) staff, by comparison, worked from home with a VPN connection. ADMC also used the opportunity to refurbish one of its trucks, as most of them were sitting idle during the lockdown.
Peter Van Dam, CTO of Alamiya in Saudi Arabia and owner of Audio Video Consulting UAE, said the production house moved to ENG work and awareness campaigns in the absence of other work.
“Edits were done remotely and then uploaded to one of our servers. We also used the time for refurbishment and for 100% cloud productions.”
Ross Production Services, the production arm of Ross Video, looked at how it could innovatively support sports events in North America when its traditional operations came to a standstill.
“During the pandemic, commentators became critical to most live events, but they could not travel to the studio. So, we looked at how we could help and potentially provide valuable contribution to the viewers. Since Teams, Zoom and the like offered pretty low-quality video, we developed our own version of a contribution tool – a cloud-enabled solution. This way, most of the commentators and analysts could work from home but could talk to other contributors and producers off-air on the same platform. If the studio wanted to stream the video back to the talent, they could; and this way, we were able to give them the same experience they would have if they were in the studio,” said Deepraj Sandhar.
Having tasted the efficiency of operating remotely and the success of remote productions, the moderator asked if companies would ever be willing to go back to a full traditional OB set-up for live event coverage.
It was unanimously agreed that because of the unique power of telcos in the MENA region, where connectivity continues to be prohibitively costly, OB production will continue to play a big role.
“To rent connectivity is tough, so we can’t do remote production,” commented Jean-Claude Rahme, though an opportunity at the upcoming Expo will help the team move towards its goal. “We are building a full-IP solution and gallery for our Expo coverage. This will be native IP all the way, from the camera to streaming, and our plan after Expo 2020 is to move this IP kit to refurbish one of our OB trucks. We aim to put all our equipment in a support truck, so the main OB then functions like a control room. We can link both OBs over fibre and this will move us to the next stage of remote production,” elaborated Rahme.
Saudi Arabia, however, seems way ahead in terms of remote production, according to Van Dam, who said that remote production was approached differently by various parties. In some cases, people were able to keep all their tech services in one place while sourcing talent from various countries. The other remote production was through a full cloud solution. He explained that DAWIYAT KSA has established fibre connectivity in most of its football stadiums, which boast 6GB connectivity and can go to 40GB if needed.
“I expect to see a hybrid situation in the near future with regard to remote production, where we will probably see three OBs instead of 10 for an event coverage and better remote production tools.”
Sandhar said that remote production must ultimately culminate in having one’s entire service production in the cloud. “As IP becomes more accessible and telco providers open up and the transmission of video over SMPTE 2110 or NDI, etc becomes more stable, we will begin to see a transition from remote production at a local level to cloud-enabled remote production rather than point to point. Point to point is what we are seeing at the moment.”
Although a complete cloud-enabled environment may be difficult to track at present in MENA and IP ecosystems remain rare, most media houses are taking their first steps into these spaces, thanks to the lockdown. Roya TV, for instance, is clear that it will stick to its current OB arrangement and will not consider remote production. However, it still moved all its archives to the cloud during the pandemic.
“We also had to do playout from the cloud, and we did a POC with one company which went successfully, so our operators could operate from anywhere,” said Haddadin. “We did, however, have some challenges with the cloud. They say upload is very cheap, but they make downloads very expensive. You can’t download very high-quality files, especially when you have 4K files, because of the cost involved – so you need to stick to your remote access, local workstations and local storage
“Regarding studios, we can’t go from the cloud. My OB van did serve as my DR gallery, which could be looked at as remote production in case my station went off-air, but it was not in the cloud.”
Public broadcasters in the GCC specify that their content cannot be located outside the region. As a result, cloud has thus far served in a limited capacity.
Van Dam pointed out that Alamiya used cloud in two ways.
“One was for archive. They wanted to do quality control and monitoring. There was a request that content was available for one of our international partners in Italy, who were making a weekly programme, and they wanted it available in the cloud. We had an Alamiya Media Cloud and we created a proxy so it was easy to browse and edit the files.
“We also gave access to the clubs, and this was more interesting. Before, if you had an archive on the premise, it became difficult to access. By having this in the cloud, we could give them limited access to their assets and we could decide how to do it – whether to offer only three minutes of a match or whether we wanted to just give highlights. There were options.
“The other was with TVU. Last August, we did projects with TVU and we used their Producer solution where we created programmes with international guests. It was a great step in the direction of what was possible in a full cloud solution. For tier 2 and tier 3 productions, where budget was limited and platform was OTT, we used this solution.”
Ross has prioritised which parts to move into the cloud, commented Sandhar.
“Non-video-based applications like MAM and newsroom systems that didn’t require much bandwidth and don’t require that much video were moved to the cloud. Likewise playout, which has less impact on real-time workflows and archive connectors, went into the cloud. Slowly, we were pushing towards real-time production. We started using these for the cloud as well.
“Now we have an asset management company under our umbrella who also do a cloud-enabled asset management system. We will see some real-time videos start to transition as well, like editorial, production switcher workflows, and then it comes down to how to get that video up or down – and IP is the answer to that. IP is the enabler of the cloud and gives people an opportunity to move to the cloud quickly.”
Hasan concluded this segment with the remark that MENA broadcasters are presently using select services typically within contribution, distribution and maybe some playout for the cloud, though core production and broadcast remain on-prem.
Van Dam, however, predicted that everything will move into the cloud, including routers.
“The next phase will see us scale down operations on-prem. You don’t need your graphics or audio operators in the same place anymore, and you can potentially make three different programmes at the same time. You can integrate all the commentators and you may sit at home or the stadium. You may want an international commentator, and that’s now possible. Even a director can be sourced from any part of the world, and this is coming soon. It won’t be for tomorrow, but very soon.”
As with the previous panel, the lack of cooperation from telcos in terms of both pricing and speed of response to media industry requirements was cited as a serious deterrent to progress.
Van Dam called it “a lack of long-term vision of the business case … In Saudi Arabia, we went through Dawiyat, a supplier who had big fibre connectivity and it was great. The UAE already has fibre optics, but sadly its telcos lack the vision.”
The discussion then turned to IP and the status quo in terms of converting on-prem structure from baseband and SDI to IP signals.
Although Haddadin agreed that everyone will move eventually to IP, he said Roya TV has no plans to undertake this migration at this point.
“It is risky. You need a lot of investment, training, redundancy, good IT experience and a secure network. I guess once we have greater confidence that there will be no downtime and this is fool-proof, we will move to IP. However, with the current budget and the management’s vision, we prefer to stay with baseband.”
Jean-Claude Rahme remarked that ADMC has decided to move ahead with a full-IP solution, as “it is no longer the future, but the present”. All panellists agreed that IP will see greater traction once standardisation has been established. They also agreed that the whole objective of moving to IP must be to optimise resources and be able to do more with less.
Sandhar added that Ross has been developing software-defined processing platforms for the last five years, where licences and resources can be shared. “It doesn’t matter whether the signal is coming across an IP board or an SDI board. The customer can decide whether they want a 2ME switcher or a 4ME switcher.”