MBC’s private CDN: Living on the edge

Shahid has been MBC Group’s blue-eyed star in recent months with the broadcast network’s CEO Marc Antoine d’Halluin announcing at the platform’s relaunch in January 2020 that if the company had invested $1.3bn in the streaming service over the last ten years in acquiring and producing approximately 46,000 hours of Arabic-language entertainment content and 20,000 hours of TV series, it had plans to double that spend within the next couple of years. The new and revised platform includes a host of Shahid Originals, new drama series and other exclusive content. More recently, it became the first SVOD service to offer exclusive Arabic-language blockbusters prior to their theatrical release.

All of this content addition, a brand-new interface and the relaunch has dramatically increased Shahid’s subscription base – which has meant ensuring that its content distribution network (CDN) remains equally robust to deal with the traffic while offering its viewers a superlative viewing experience. To ensure that, MBC Group recently completed the addition of a private CDN to its existing multi-CDN setup. With this, MBC joins a very small coterie of sophisticated streaming services in the region that have added a private CDN to their content delivery mix.

Dominic Farrell, Shahid’s Chief Technology Officer says MBC Group is constantly monitoring and evaluating options to improve the quality of the service it offers users.

“With the arrival of global streaming giants into the region, there has been a step change in consumer expectations around both streaming quality and reliability that we have to meet if we are to remain competitive in the market. Our team has been evaluating the option of adding a private CDN to our multi-CDN architecture for some time, but it wasn’t clear as to when would be the right time to execute the first phase of this strategy,” Farrell says.

With the relaunch of Shahid earlier this year, together with the investment in original productions and the rapid increase of new content onto the platform, the team felt the time had finally arrived.

“The huge growth we’ve seen means that relatively small problems become magnified: if a specific buffering issue affects 1% of our users, that’s still hundreds of thousands of people impacted. And with paying subscribers, there is a very direct link to satisfaction and therefore our revenues. They rightfully expect to receive perfect HD streams without issues, and it’s our job to do everything we can to make that happen,” explains Farrell.

Adriaan Bloem, Head of Digital Infrastructure at Shahid who initiated and oversaw the whole project, alongside MBC’s DevOps manager, who ran the PoCs; and the Media Engineering lead, who set up the routing and runs the operations, explains why the team chose a private CDN this time.

“Delivering reliable video streams to end users in this region is still quite challenging. In Europe and the US, it is basically a solved problem. You contract it to a good CDN, negotiate a good price and your video will work as intended. In emerging markets like the Middle East and North Africa (MENA), the landscape is much more heterogenous. Most of North Africa has no dedicated CDN nodes from any of the major suppliers and the quality of home internet connections ranges from pretty-low bandwidth in Egypt, to the high-speed, high-quality fibre we get in the UAE. This takes constant tweaking and chasing CDNs and ISPs and challenging them to improve the delivery.”

An additional CDN has long been in the planning phase, explains Bloem. In fact, the team ran its first POC with Equinix five years ago, with the prototype built by MBC’s DevOps team in Jordan. Although it proved the concept would work, it was not urgent enough for the team to properly resource it at the time. The relaunch of Shahid in January 2020, and the apps that were built ground-up in-house were the kind of things they wanted to get out first, he elaborates. Hence, there was a long waiting period before the CDN was deployed.

MBC has had a multi-CDN ecosystem since 2013 with all the usual suspects like Akamai, CenturyLink, Limelight, Microsoft and AWS CloudFront primarily “because it is the only way to fill in the gaps between the different suppliers and their regional coverage”, says Bloem.

“We’ve used various different tools to switch to the right CDN for the right location and time, including our own APIs,” he adds.

The private CDN is operated by Medianova as a managed service and this addition will help MBC provide much better video delivery.

Medianova’s CEO Serkan Sevim steps in to explain that while 85% of content today is delivered with a multi-CDN approach making it the industry standard, “a private CDN should be one of the important delivery points for OTT”.

“A private CDN PoP provides more control, better quality by not fighting for the cache space, and more agility by letting you run your own microservices at the edge,” says Sevim.

“While we recommend that clients keep the existing multi-CDN architecture in place, we explain that a private CDN platform improves the quality of content delivered in the region and offers greater control on where to cache their valuable content. This platform can also be considered as a private cloud at the edge where OTT providers can run their own microservices such as packaging, DRM, real-time data analysis or machine learning applications. This way, the key data and operations stay in the region without going back and forth to Europe or other locations,” he elaborates.

For MBC, which has a relatively reliable setup in place, the addition of a private CDN was meant to help make “the smaller incremental improvements”, says Bloem.

“You want to start shaving off a second of the video startup time and increase the percentage of people that get our highest bitrates and video quality. We work closely with our CDN partners to achieve that, but having our own dedicated nodes makes it easier to steer on some metrics that are very particular to Shahid,” he elaborates.

The team, therefore, wanted to have the ability to put nodes where they matter most to Shahid, explains Bloem.
“That objective is not always aligned with the priorities of global CDNs that need to satisfy multiple customers,” he points.

“Crucially, our library is enormous; we have much more content available than some well-known global streamers. That ‘long tail’ of content is much harder to deal with, because the number of plays on an asset may be relatively low; an ‘out-of-the-box’ CDN would deprioritise that content and wouldn’t keep it in the cache very long. So, every time it gets played, it is like it’s being played for the first time. This offers a terrible user experience. Having our own CDN allows us to set the rules, and our VIP library gets the priority it deserves,” explains Bloem.

Sevim seconds this. Besides the fact that global CDNs may not have enough POPs to support an entertainment provider’s specific target market, existing POPs may also have old and slow equipment with not enough cache, and CDNs may not necessarily have all the features that a user needs, he points out.

“One CDN could be great at image or video optimisation features but not so great on latency. Another could have good latency but not have flexibility on integrations required for the agile world. So, there are many use cases where a multi-CDN approach is required to ensure a stable service, but it may not be enough,” he clarifies.
It is this search for greater control and a more stable service for its regional subscribers that led MBC on the path to search for the right CDN. The team, however, also realised in its initial PoCs that building out their own CDN would require a very specialised skillset. It would take time to build this in-house and would also mean dealing with a lot of hardware again.

“Shahid is very much based on a cloud-native infrastructure,” explains Bloem. “Working with hardware again would mean taking a step back into the past and we didn’t necessarily want to take that on ourselves. So we looked at how we would be able to build this ourselves (for instance using Varnish or Nginx). We also had a look at specialist companies that could, at least initially, run it for us. Medianova stood out because they’re familiar with the region and have the experience to do this.”

Having had a multi-CDN setup since 2013, adding one more to the existing ecosystem didn’t seem a Herculean task, Bloem says.

“You add it to the rulesets and monitoring, and then jump immediately into analysis and tweaking, which are the operational concerns of maintaining a solid QoS across the various channels,” he elaborates.
MBC had the new CDN up and running in a few months.

“The main delay was because we were also simultaneously dealing with a massive increase in traffic on Shahid before Ramadan. It’s running in production with several ISPs peered through the UAE-IX internet exchange here in Dubai now, and we’re slowly ramping up traffic. It’s in production now and we’re expanding the coverage. Then we’ll start on the second phase expanding into other exchanges as well as moving on-net with individual ISPs, which is almost inevitable with a service that has the kind of traffic Shahid has,” explains Bloem.

For consumers, the addition of a CDN will reduce the video start-up time. The time it takes from when a user clicks play, and the first frame of the video is shown will be hugely improved. There will be reduced buffering, and a huge improvement in the bitrates users get.

“This will especially be visible on big screen TVs,” explains Bloem.

“So it helps us improve the overall quality of the experience for Shahid users, particularly on VIP, the platform’s premium SVOD package.”

Unlike a global CDN, the addition of a private one will allow the team to scale up much more quickly, and with Shahid’s traffic levels peaking, this will be a more economical strategy for the team.

Perhaps what is unique about the setup is that it is customised in terms of cache rules and where it gets deployed, “without us immediately having to master all of the details of how to make it work”, explains Bloem.
“We compete on video quality with a company like Netflix, but they have years of experience with this already and probably about a dozen people dedicated to their OpenConnect CDN for MENA alone. With the enormous growth of our traffic, this allows us to catch up with that, and do better than our competitors.”

Of course, it is not without its technical challenges but the Shahid team is gradually building its internal expertise.
“We have very competent teams — particularly our DevOps and Media Engineering teams. But running a CDN requires very specific skills especially with regards to routing – something that not many people outside of a telco environment have much experience with. Medianova has helped us to kickstart this and in time, we will build up the expertise internally.”

This does not mean that global CDNs don’t play a significant role.

In fact, Bloem reiterates that with Shahid being a global service, the broadcaster will “continue to work with global CDNs”.

“For instance, I discovered that even in Greenland, we have a few hundred Shahid users. It will not make sense for us to deploy a CDN node there. The plan, therefore, is to focus on our core volume markets, such as KSA and Egypt, and get good coverage with the ISPs.”

But the team has no plans to stop there. With the aim to conquer the region’s streaming market if not global just yet, Bloem says the team has the Maghreb and Levant on its radar next.

“For that, we will likely take increasingly more control of the infrastructure,” he says.