Archiving the “Big Data”

During CABSAT, BroadcastPro Middle East teamed up with EMC Isilon and INC to conduct a roundtable that was intended to bring together a small group of key players in the market to share information and encourage collaboration on digital archiving and storage.

Some 15 broadcast executives from across a number of media organisations gathered at the Fairmont Hotel on 29 February in Dubai for the roundtable. David Sallak, who is based in the US and works as a business strategist enabling broadcast entities to adopt new technologies, led the discussion. Having worked with ITV, UK; channels nine and ten in Australia; NBC and ABC in the US and Televisa and Mega in Central and South America among many others, Sallak brought a wealth of knowledge to the discussion, filling the gaps and answering queries while encouraging local players to talk about the challenges they face, the trends they are looking to adopt and if they are ready to undertake archiving projects at their respective facilities.

 Representatives from Abu Dhabi Media, Al Jazeera, DMi, Ashorooq TV, MBC, OSN, WRN Broadcast, Ten Sports, twofour54 and Master Media discussed how they manage long term archives today, and the challenges and opportunities for archiving in the future.

 Key to the discussion was the most appropriate way to archive existing content, the ideal format for archiving, how to monetise one’s archives and whether cloud was a safe option presently for storing premium content.

Sallak began the roundtable with a short presentation that set the stage for the  roundtable. He began by asking how many people used videotape, data tape and/or disk to archive their content.

Most people in the room indicated that they used disk to some extent as an archiving medium, but added that they continued to maintain the use of videotape and/or LTO (Linear Tape Open) method of traditional data archive while some of them also went one step ahead to house tapes outside of their building for added security. Video tape was considered a legacy medium that is quickly losing its place both for acquisition and storage, due to the increasing cost to store media on video tape. However, it’s still used heavily in broadcast data storage even today, due to its reliability.

David Travis: The real advantage of using tape is that you can ship it to other locations.

David Sallak: Tape is based on utilisation. How many times do you need to access that particular content? With disk, it is easy to transfer content digitally from one place to another. When it is stored in a remote space, it can be restored quicker because it is stored as data.

Hasan Sayed Hasan: The general consensus is that videotape is dying quickly. File-based acquisition and workflows are becoming prevalent in the regional broadcast industry, so there is a huge need to digitise content and build a robust archive.

So what kind of storage would you prefer?

Disk or tape?

Hassan Chahine: Most big players around the world use disk as well as tape archives. In the last few years, we have seen the utilisation of spinning disk not just for internal archives but for external archives also including web IPTV for customers.

A number of networks are using spinning disks to make their data available to customers. But with spinning disk, there is the risk on power, cooling systems failures and so on. But you seem to be pushing to archive to spinning disk.

David Sallak: I’m merely suggesting new ways of thinking. Most companies maintain a balance between disk and tape, based on their business model. For example, 95% of content can be stored on tape and 5% on disk, based on your business model where maybe, 95% of that data is not touched in three years.

Hassan Chahine: So it is based on utilisation!

David Sallak: Exactly. Tape is very efficient when the data is very rarely accessed.

David Travis: To some extent, I agree. In an ideal world, we would have effectively replicated online storage in some way. That will always be the perfect scenario for every broadcaster. But from an operational cost of having cloud storage or replicated storage, it is usually something you cannot support because the density of LTO tape keeps getting better and better so you can have more storage in a smaller footprint. Therefore, there is less need to switch to disk-based storage, particularly in the short-term.

Mounir Adghoughi: One of the issues we have with the LTO is that it has a proprietary file system. LTO is good but we will have LTO6 coming to market. If a tape is broken, it is broken. If a tape is lost, it is lost.

Omran Abdallah: I think the choice of storage and archive is dictated by the strategy, the cost, the business, the technology and the operation. The main issue is file integrity. What we are archiving now needs to be accessible in a decade and that cannot happen if we do not store it in a file format that is future proof. Before we start thinking about disks or LTOs, we must look at the process from end-to-end to see whether we are looking at short-term archiving or long term archiving (tape). If we are thinking short-term, perhaps disk is the answer but for the long term, tape is ideal because of the new technology with LT05. The new LTFS feature allows us to put everything onto different tracks. This allows for easy searching and retrieving of material and it’s even faster than a disk.

David Sallak: One of the advantages of Linear Tape File System (LTFS) is that it is now being proposed into SNIA [Storage Networking Industry Association] as a standard within its specifications because data integrity across all devices in the future — to be able to read the data — is important. The future ability to read the data is crucial. LTFS is a tape-based file system, and this is a goal that’s being sought by the tape industry. The objective is to present the data on tape in a directory-system manner that can be accessed openly.

Melvin Saldanha: Both disks and tapes have their own sets of pros and cons. LTOs are relatively cost effective when compared to spinning disks and come with the ease of replicating content across backup tapes for disaster recovery. However, every three years, we have a new generation of LTOs with better tape capacity and bandwidth. That involves migrating content to the new generation to keep pace with technology. If you look at spinning disks, storing petabytes of data on disks would involve substantial costs for disks, support for replacements of drives, power and air conditioning. A right combination of both disks and data tapes would be ideal.

David Sallak: Okay, so we have agreed that tape and disk are important based on the business model we adopt. What about archiving on the cloud?

Melvin Saldanha: Cloud computing comes with its own set of risks. A broadcaster could end up giving its 30-year-old content to a competitor via the cloud if encryption of high res content on the cloud is not handled well. Security is an issue. Moving content to and from the cloud would be the key, knowing bandwidth is expensive in this region.

Hasan Sayed Hasan: But banks also have secure content on the cloud and it is becoming more common these days. If it’s secure enough for them, surely it must be good enough for the media as well?

Hassan Chahine: I second that. In 10 years, 50% of all media and broadcasters will be using cloud for storage. You cannot run away from it; accessibility of media is very important and security is increasingly getting better.

Hasan Sayed Hasan: The technology is different from the service. Cloud services have been the buzzword recently but having another service provider or another site where your data is safe has emerged for other industries. This can apply to the media and broadcast markets. I do agree though that the cost of connectivity in the region is very high at the moment.

Melvin Saldanha: Bandwidth is very expensive.

David Sallak: That is a barrier to adoption. There are very different approaches to content security depending on what your role is. If you are a state broadcaster, you will want your content to reach everywhere and by all media, while commercial broadcasters and pay TV operators and content rights owners want more security and restricted access to their content and delivery streams.

Mounir Adghoughi: A shut down can close your access to a site. Even if you put something in a cloud, if there is a notoriety issue, it can be shut down. Look at the Megaupload scenario.

Abdulla Alhashemi: The issue in the region is primarily the cost of connectivity to disk-based storage. The majority of broadcasters are still reluctant to use cloud-based archiving or collaboration tools as they are not yet comfortable with security and availability.

Mounir Adghoughi: Expensive bandwidth is the barrier to cloud storage adoption.

David Sallak: Let’s take the Megaupload scenario. I have protection from that event because I have maybe, a private cloud infrastructure. It is a cost balance definitely because these external services have significantly lower cost. This is, of course, separate from bandwidth in this region, where moving data up the wires is quite high. In the US and Northern Europe, the cost to transport is dropping significantly and the cost to store it on whatever infrastructure is also very low.

I’ve run into customers that look at the cost balance sheet. One could be using a remote archive of some type, and converting the data into digital or these days, into file-based workflows as video is already in the digital domain as files. So instead of dubbing it to tape, when it was never created or processed on tape, the data lives on the disk as a finished product. Delivering it to remote storage (aka cloud) as a trickled-out delivery can optimise my bandwidth cost. I can remotely store video data in two places, creating a RAID1 cloud. But the combination of these two services can cost the same or less than tape infrastructure.

I have some customers who look at this and see tape infrastructure as reliable over many years. In the digital file format, we rely on LTFS and other formats such as Archive eXchange Format (AXF) that will be standardised to help access that data using a future tape interface to the physical tape device. The question is how reliable will this storage be for my business.

With different strategies, what are the thoughts about how to go forward from here on how you are archiving based on the fact that you are not creating any more content on video tape but only as file.

Hasan Chahine: As a state broadcaster, we produce more than 75% of our content locally. More than 85% of this is available on the cloud, and can be accessed using mobile, tablets, smart TV and all media tools.

We know content needs to be available on multiple platforms. It is based on the business model. I don’t mind if other people download my files or watch my content. We are an FTA (free-to-air) network.

Frank Kerrin: As a pay TV network, we find it difficult to subscribe to that idea. We are not happy if people are downloading our content. We archive locally and externalise LTO tapes and store them off site. In the future, we would look for something on the cloud but our concern has always been the connectivity apart from the costs.

There is the feeling among broadcasters today that content would be exposed because it still uses the internet and can be compromised if it sat on a cloud network. For OSN, this would lead to contractual problems with studios.

David Travis: The other problem is that everyone approaches rights differently. Different networks have different content. Pay TV networks use content that comes from the studios. It is bought and licensed. The Disaster Recovery (DR) and archive plan is their archive and their cloud and you are accessing that as your backup.

David Sallak: You’re pointing out that your archive strategy is not under your control … because you don’t make the content, i.e. you are a distributor of other people’s content, you have a rights restriction. You must prove that it has been deleted from your environment once the rights window is closed. So you are depending on their archive strategy and their provisioning of their data to you as an extension of your ability to distribute it based on the agreements and rights.

Melvin Saldanha: What happens when the content is available only on one particular source? We have a cricket match in Sharjah that was held in 1984. No one else in the world has it. It is on a one-inch tape but I want it on LTO5, and keep a backup elsewhere. When you compare your nearline and deep archive, the question to ask is, can I put it on disk? Is it too expensive? What about power and cost?

Can I put it on LTO5? Yes you can, but every two-and-a-half years, there is a new generation and if I skip two generations, I have to migrate the data again.

My data has significant value. Being a rights holder, no one else would have a copy of this content. Hence it is imperative that this kind of data is stored securely either on disk or on tape or on the cloud. We’ve ensured that these are backed up on LTOs and moved to a Disaster Recovery location in addition to having the original.

Mohammad Saifi: So what is the recommendation? Tape or Disk?

David Sallak: Is it an either/or discussion? We do not have a customer that uses one exclusively. It’s a combination of tape and disk. Seventy-five percent of our customers use disk for archive nearline process. The business value of the data itself dictates the degree of risk or higher cost infrastructure applied to storage. A very large Hollywood movie studio customer of ours has a three-petabyte archive of disk. They wanted to get out of the tape storage business and it is the value of the data itself that justifies the use of disk — not tape — as an archive.

Their content has extremely high business value. It’s worth pointing out that this is an extreme example. They do have a DR scenario where they DR to tape but they do not restore from tape. They check it occasionally but their restore process is always from disk. In the case of the majority of our broadcast customers, video tape and data tape are heavily used for archive. The ratio of the balance and the value of what we do is important. News footage, for instance, has very low business value unless an event occurs that makes the archived data relevant again. In most cases, it could be an interview with a state leader and when that leader passes away, the data has great value for a broadcaster because of its news relevance. So essentially, it’s the value of the data itself that determines what cost will go into storage.

Mohammad Saifi: In terms of reliability of saving the data for companies, how can we convert? Disk?

David Sallak: Both have improved their reliability quite dramatically. How would you measure that as a value, the success of disk compared to LTO tapes? Would you ask ‘I have had no data lost using this method?’ The migration from video tape to disk and LTO will happen over the next 10 years. What format will you use that you expect you can read 10 years from now? The format you create content in may not be the format that other networks want to use it in.

Mounir Adghoughi: I prefer in the future to go with a neutral deep archive. We have a lot of programming that needs to be played out for a period of five days. Why should I spend my money on disk archive, which is expensive?

David Sallak: What Master Bitrate (MBR) file format would you consider using given that you would want to read this video data ten years from now?

Munzer Rushdie: Many customers are choosing high-quality edit formats such as Apple ProRes, Avid DNxHD, or more-open formats such as AVC-i and XDCam for their Archive MBR.

Mohamed Abdel Azim: We all agree archiving is important for media entities. The question we ask today is more fundamental. What software and what format would be ideal for us to standardise our archives?

Omran Abdallah: At twofour54, we have DNxHD for archiving and XDCam HD 422 50Mbit for archiving. We have to look at how we can integrate with multi-format archiving and be productive for our business.

David Sallak: Let’s take a step back and look at the goal of a media entity. I shall use a broadcaster in Central America as an example. They had a limited archive strategy in the past, but they have expanded their archive strategy dramatically and selected an MBR for the archive, where it makes sense for workflow. They analysed how they make the data; then determined who uses it after they make it, which is usually in a format that is not the one that they use so they have to change it when they want it delivered to the client that has licensed it for their market. If they want to use it themselves, they can always use the original MBR.

Melvin Saldanha: It is advisable to keep the video data in the highest quality bit rate and resolution – DNxHD or AVC-Intra 100.

Hasan Sayed Hasan: Now, some content creators are also talking about JPEG2000.

Omran Abdallah: If you want to archive, I recommend that you keep the original codecs. If you transcode, you have to archive the compressed data.

David Travis: In Europe, AVC-Intra is the preferred MBR for 10bit. Most end users including some of our clients have signed up for that. We will archive in this format.

When you talk about file formats, there are two questions to ask. Are you talking about a broadcast infrastructure of the now or are you talking about a broadcast infrastructure for the future? We operate a 10Gbit network; so we’ve got complete flexibility. We play out AVC-Intra; we transfer that to our playout servers and play it out. If you are on a 1Gbit network, we use MXF50 for the content. So it depends on what the digital infrastructure restrictions around your network are.

Hasan Sayed Hasan: We should not only look at the current and future acquisition and production file formats when planning for digital archives, but also at historical and old content such as films, one-inch and two-inch tapes and other legacy tape formats, and at what quality and using which codecs this content will be digitised.

David Sallak: You have to look at what data format is appropriate so you can read it in the future and attach the metadata around it. I find that with archive strategies, the more simple you make it, the more likely it is to be adopted and maintained through different generational changes of leadership. Otherwise, will the integrity of the archive be maintained past the person that designed the complex system?

That’s how most of the media is likely to be managed. Carrying that forward and keeping it simple gives you the flexibility to choose the right cost infrastructure of disk and tape. We have a consensus here that LTO continues to be a good way to go forward for deep archive, with the appropriate amount of redundancy.

Look at LTO formats and writing multiple copies of them; checking them periodically; and having a migration strategy to go into LTO7 whenever that comes out to migrate that data. Recognise there is a cost to that in terms of labour and equipment to move the data. When it’s on disk, some companies choose to use that as an archive because it’s easier to move the data. There’s less infrastructure involved. So the flexibility and cost to move it over time and maintain the life of it is also a consideration.

Before, it was very easy to archive when it was film and video tape, but film and video tape cameras are no longer being manufactured so I need to migrate that data out of this technology because the parts are going to be difficult to buy 15 years from now.

Abdallah Alhashemi: There are lots of factors to consider. Which archive company is good and which company would make it easy?

David Sallak: In this region, there is the consensus that there should be a standardised format, or an MBR, for the broadcasters to use in order to make the transition to archive within a safe and measured environment.

Videotape worked well and we will carry on labelling tapes. It is important to carry this methodology forward when establishing an archive infrastructure. We all agree that we will carry on using tape LTO formats and multiple copies of them. But take on as much of the process as makes business sense. Build on strategies and make it more sophisticated as we go forward, by evaluating the various archive software vendors. Review cloud storage providers and their value as a possible DR service for local archive, or a means to restore archive content remotely while keeping it secure.

Key Points of Discussion:

Disk Archive or Tape Archive – Right combination of both could make good business sense. Ideal method would be to introduce a layer of disk storage in front of the tape archive. This disk storage will provide redundancy with content available for use in the event of any issues encountered when accessing the tape archive. Disk archive will also add a layer of protection to tapes as archived data on tape will be accessed less frequently, hence maximising both LTO tape and LTO drive life cycle. In addition, the nearline disk storage would give you the opportunity as well as the capacity to open up your archive to a brave new world of IPTV and Over-The-Top (OTT) delivery.

The introduction and development of Linear Tape File system (LTFS). This is in its second generation, intended to address tape archive requirements defining the organisation of data and metadata on tape using an open system architecture that enhanced the integration of Tape and Disk Archive. Data tapes written in the LTFS format can be used independently of any external database or storage system, allowing direct access to file content data and file metadata. This “file system on tape” allows accessing the files stored on the LTFS formatted media similar to the method of accessing files on removable flash sticks or spinning disks. Consider LTFS as a means to integrate disk with tape for a common archive platform.

Disk archive or LTO? It’s ultimately a business decision depending on costs and size of assets being archived, complexities of restore time, the workflow benefits to restoring an archive while meeting SLA needs, and the role of the broadcaster as a public or private business operation that creates or licenses content. Disk and tape have their certain limitations based on speed and cost, but as technology evolves with more and more broadcasters adapting to file-based workflows, the two together with cloud could be the ideal mode of archive.

Video essence and file metadata are the key factors in long-term archive solutions. Preserving media in the best possible format attached with metadata today would secure a comprehensive MBR archive in future. AVC-Intra @ 100 Mbits seems to be the ideal choice, based on input from the European broadcast and post community.

Cloud technology needs to offer a securely encrypted option, with a means to move data to and from the cloud efficiently while providing high availability. Cloud for high-res content (i.e. Big Media Data) could be an ideal platform for expanding an archive. However, bandwidth costs and security seems to be a concern in the GCC. The cloud will continue to evolve and become a better option for deep archive and DR, as bandwidth costs decrease in the Middle East region.