MBC Group has forged ahead with a full-fledged virtualised business continuity centre that is equipped to take over should the main site be compromised. BroadcastPro ME gets an exclusive first-look.
While most regional broadcasters are still toying with POCs for virtualised playout projects, MBC Group has forged ahead with a full-fledged virtualised business continuity centre that is equipped to take over should the main site be compromised. BroadcastPro ME gets an exclusive first-look.
MBC Group recently deployed the Pebble Beach Systems Orca, which serves as the core playout solution at the broadcast networks virtualised Disaster Recovery (DR) facility. The virtualised DR deployment is reportedly a regional first, as opposed to the proof-of-concepts (POCs) or test channels at most of the facilities so far.
We look at this facility primarily as a business continuity centre rather than a DR unit, clarifies Nader Mokhtar, Director of MCR and broadcast technology at MBC and the project manager of the DR/Business continuity site.
This is not a pure DR system. I prefer to call it a business continuity centre because it functions just as well as the main site. MBC is very proud of this site as it is based on a modern and flexible technology all the way from the MCR to its DVB/muxing capabilities and the Orca virtualised playout system. They are all IP-based, where you can monitor and control everything from either our Headquarters in Dubai or anywhere else, says Mokhtar.
Nick Barratt, Senior Broadcast Manager at MBC elaborates. The traditional approach with DR has been to have a very scaled down operation compared to what you traditionally have at your main facility. Your media options will be limited, and you will probably run some evergreen generic material that buys you time to get a playlist of what you should ideally have on air. With business continuity, operators can move seamlessly from the main site to the next just that they have equipment in front of them for one and they are remotely managing the other.
Barratt points out, however, that a real installation is a far cry from anything undertaken at the POC level. The MBC team worked closely with the Pebble Beach Systems team and he says it was a steep learning curve for both parties, as the technology is still young and the experience in the field is still nascent.
Last summer, the teams from MBC and Pebble Beach Systems started talking about a real installation and over the last several months, the teams have reached various milestones. By late 2017, the Business Continuity system had been fully deployed by MBC, replacing an expensive third-party service and bringing the function virtually in-house, thereby delivering both cost-savings and full business continuity.
By early 2018, 13 of MBCs most important channels across both sites had been synchronised with the broadcaster aiming to replicate the rest of its channels as well. By February, additional features like the ability to do live sources had been rolled out. The team will soon be ready to roll out 3D graphics as well.
The 13 channels are totally synchronised with playlists in Dubai and we can seamlessly switch remotely between the main and the back-up sites through IP switches, explains Mokhtar. We do not have full continuity graphics on there just yet. Each one of those channels are simulcasting in SD and HD. Orca channels run in a virtual machine (VM) in a private cloud with IP inputs and outputs. These are the key features compared to other installations.
Samir Isbaih, Regional Director, Middle East & South Asia, Pebble Beach Systems, says the real game changer is Orcas ability to run the full broadcast chain in IP on a VM.
Orca offers a software-only, all-IP, virtual implementation of the Pebble Beach Systems Dolphinintegrated channel platform. Channels can be launched and integrated into a running Marina automation system very quickly, making it ideal for temporary and event-based services. This MBC deployment cements its application for DR environments, as it runs 24/7 in parallel to the main channels being broadcast from Dubai.
MBC already has a Marina installation in place, so deploying a Marina-controlled Business Continuity solution was the obvious next step.
The chain of discrete devices controlled by Marina at MBCs playout centre in Dubai is duplicated at the DR location in a software-defined environment with compressed IP inputs and outputs replicating the baseband best-of-breed functionality in one box, including plug-ins for third-party Pixel graphics, and loudness running on the same VM as part of the same IP broadcast pipeline, explains Isbaih.
With Orca claiming a flexible design and shared memory concept, Pebble says industry-standard, third-party plug-ins can be incorporated into one software-defined TX chain. MBC evaluated other solutions but eventually settled on Orca, because its users were already quite comfortable with the Marina experience and would find the Business Continuity system easier to operate.
Orca offers remote domain list synchronisation, even over a potentially unreliable network. It replicates all playlist edits and actions at the remote site. Both systems are monitored from Dubai using Pebble Beachs Lighthouse remote management and monitoring tools.
Lighthouse is a web-based application and the plan is to give customers an option to operate their multiple systems on separate domains from a common user interface, rather than having to use VPNs across multiple sites. It uses a Linux server, so security is better and it can eventually connect to both the Dubai site and the other site, claims Isbaih.
Barratt says the operator has a graphical view of both sites, making monitoring easier, and can see it on any device, even remotely.
In addition, the operator does not have to manage the playlist on a day-to-day basis. We load a playlist in Dubai every night and as soon as we load that playlist, a list update gets sent across to the business continuity site, and any changes we make during the day or for live events are picked up within half a second of someone doing it in Dubai. The key thing is that nothing changes for our operators in Dubai. They continue doing the same thing they were doing here and the playlist is automatically changed at the business continuity site. Effectively, all we have is a multi-viewer in Dubai from the business continuity side, so if anything happens, they can step in and take control if they need to. Barratt, however, points out that the Lighthouse deployment still has some way to go. We are in the process of deploying Lighthouse with the ability to basically control a playlist and load the playlist on the Tx side. We are still to get to a point where it will come close to replacing the existing clients; however, we are using it to monitor the global view of the timeline and other such elements.
“MBC is very proud of this site as it is based on a modern and flexible technology all the way from the MCR to its DVB/muxing capabilities and the Orca virtualised playout system Nader Mokhtar, Director of MCR and broadcast technology & Project Manager of the Dr/Business Continuity site, MBC Group
Although the Orca is the playout solution, Mohktar remarks that this is indeed a collective project, which includes solutions from other manufacturers as part of the workflow. For instance, encoding and multiplexing is from Ateme, while there is nearline storage from the likes of EMC/Isilon. Eventually, there will be also a lite version of a MAM solution at the DR site.
There is a lot more to this project than playout. There is the whole networking and connectivity that has been undertaken. There is the transcoding to deal with and the movement of files from Dubai to the DR site, and the actual contingency plan of how we can get media if we lose Dubai and re-purpose it at the DR site. From an encoding and multiplexing POV, we put out a CVR 20MB for HD and a CVR 10MB for SD, and thats reencoded and stacked/muxed in the Ateme system so we can have something to be multiplexed and uplinked, and obviously that can be repurposed to go over multiple satellites.
MBC will also eventually undertake remote editing.
This MBC deployment cements its application for DR environments, as it runs 24/7 in parallel to the main channels being broadcast from Dubai Samir Isbaih, Regional Director, Middle East & South Asia, Pebble Beach Systems
We hope to have the ability to have a VPN connection at the business continuity site and access the entire system as if they are sitting in front of it. There are some interesting bits and pieces we are putting in there to make it a globally accessible business continuity site. This is also helping us with a POC for a longer-term migration of whatever systems we can take into the public cloud.
The whole point is that the majority of the technology we are using is all available in public cloud, and this is allowing a slightly more controlled environment to get by with users. We are controlling the actual connectivity with the site as well as allowing us to tweak and play with it more.
In essence, MBC has deployed a virtual environment, using IP inputs and outputs. It is business continuity running in step with the main channels. More importantly, Isbaih points out a trend that is becoming increasingly common among end customers as they move into software environments.
Today, we see more customers supplying their standard IP hardware and the virtual machine environment. MBC also delivered the hardware and the environment, and our software runs easily in their environment.
Of course, an installation like this came with huge challenges.
We had issues with latency and being able to get the IP inputs. Its working now and we have given our feedback to Pebble Beach Systems. Its different from other deployments, and therefore the biggest challenge all around was that it was a big learning curve for everyone, says Barratt, adding that it was also a huge experience for all those involved.
Although there are no cost savings at this point, as many manufacturers like to tout, Barratt points out that space and storage are obviously more optimised.
It has been a huge learning process for both sides on deploying and tuning the whole system and adding functionality over time Nick Barratt, Senior Broadcast Manager, MBC Group
The cost is comparable. It was an easier deployment in that we were not doing an SDI infrastructure. It was just running Cat 6 Ethernet cables everywhere. The fact that it didnt cost us any more allowed us to learn more and produce something useful. It fits into where we see the future going. Of course, there were significant space and machine savings with two racks being more than sufficient to run 13 channels, says Mokhtar.
It has been a huge learning process for both sides on deploying and tuning the whole system and adding functionality over time. When you stick a box in, its expected to do something. The huge learning curve was how to tune and get it to do what you need it to do reliably. Those did create significant challenges. There is a level of trial and error and learning that is different from a POC. And when you are starting to use it, you begin to realise that a weak POC is not good enough to understand how the thing must be tuned to achieve a certain functionality.
He concludes: Off the back of what we have done and what Pebble has done in DMC, we will have a better idea of what a starter kit should be for something like this in the future.
