Well, how about losing an entire feature film and a year's worth of work? While this isn't a new story, as it was featured in the special features on the DVD, the tale of the failed Toy Story 2 backup is fascinating. It just proves that even when you're spending millions of dollars, a few mistakes can be disastrous.

Here's the humorous animation of the potentially disastrous situation:

I know many of you might be thinking that just because something is deleted, doesn't mean it's gone forever, but I have to believe that the animation and the story was greatly simplified from the actual situation. It seems to me that whatever networked storage they were using would have made it very difficult to recover what was lost, or that the time it would have taken to recover that material would have delayed the movie by months. It's certainly possible for a few IT people to screw up the backups, as you have to be diligent with large storage arrays to make sure that they are not corrupt.

While the story, in this case, had a happy ending, I'm sure many of you out there have been on the wrong end of file corruption or bad hard drives. I know I've had a few cases of bad hard drives or misplaced footage, so I think it's worth bringing up the idea of having a good backup strategy when making a film. If you're shooting 4K, you already know it's going to be a headache, but for most of us, there's no excuse not to have our footage in two, if not three, different places. Embarrassingly, the reason that the rest of the 5D Mark III/D800 review is not out yet is because I thought I'd had footage backed up in three places, but only had it in two, and it was accidentally deleted from one of my drives, leaving the only remaining copy on a completely different drive that belonged to a friend - who happened to be across the country. Good times.

It's pretty easy to have an accident like that if you're not careful, and being on set is one of the more stressful places to do that sort of work. So if you're shooting a film (or any sort of project really), a couple of things you should always keep in mind:

  1. Have a dedicated backup person on set, ideally someone who isn't the Director or Director of Photography. If you can, make a separate space as far away from set as possible (within reason), so that the data wrangler has a quiet and stress-free place to back up your important footage.
  2. Keep the ENTIRE FILE STRUCTURE of the SSD drive or CF/SD cards. Don't ever just pull individual clips from a card -- always back up the entire drive or card. This can lead to issues in post if the file structure is changed.
  3. At a minimum, have the footage backed up to two separate drives, not just a RAID drive. The issue with just using a RAID is that if the RAID is corrupt in any way, it could possibly corrupt both copies -- not good if that's your only backup. If you can get it on three drives on set, you'll be in excellent shape for any possible catastrophes.
  4. When you're using solid state media, like SSDs or CF/SD cards, wait as long as humanly possible before needing to reformat those cards. Ideally you would use separate cards for all of the footage on one day, but if that's not possible, rotating two or three cards will help keep your just-backed-up footage from being reformatted for at least a few hours.
  5. Develop a system with the data wrangler on set to ensure that cards are only formatted when the footage has been backed up. Using a color-coded tape system is a great way to do this, as detailed by Evan Luzi, who runs a great blog called The Black and Blue.
  6. If you use some sort of backup manager to automate the process on set, make sure that the program will check the integrity of the backups against the originals. I've often done backups manually, as time and resources has dictated it be done this way, but a checksum can certainly help prevent bad data on your backups.
  7. When your production is done, take that third backup to a completely different location. Fires and other accidents can, and do happen. While it is tragic, a good way to prevent your hard work from being lost forever is to have another copy in a completely separate location.
  8. If you've got the money or the time, LTO tape backups are a great option as an additional backup, but many times there isn't the budget or the time to do them.
  9. If you've got hard drive backups and they aren't SSD drives, DON'T let them sit on a shelf and do nothing. This is the easiest way for backups to go bad, because drives need to spin every so often to keep the integrity of the data.

While I'm not an expert on data backups, developing a strategy is important to keeping all that hard work safe. In my experience purchasing hard drives, it's always best to read reviews about failure rates and try to buy the cheapest one that seems to be working the longest. The best way to purchase drives is to not buy the latest and greatest, as reviews will most certainly be limited, but also because it usually takes a little bit for hardware manufacturers to work out the kinks for a particular model within a brand.

The other way you can minimize purchasing bad drives is not buying two of the same drive from the same retailer. If a specific production run of drives has an issue, and you buy several from the same place at the same time, it's possible to get multiple bad drives from that same run. That can spell disaster if you were planning on using all of those drives for the same backups.

As your budget level increases, so should the complexity and redundancy of your storage array. If you've got a fairly large project or a series of projects, you should look into building a storage array that will not only speed up the access times for your data but also give excellent redundancy. If you're interested in building a SAN (storage area network), Rich over at Digital Cinema Demystified is currently building one and is detailing the process on his blog.

Something to always keep in mind: when it comes to backing up, data is never safe, so the more places you've got it, the better chance you'll have against a complete catastrophe. You can always buy new hard drives, but trying to reshoot days or weeks is almost never an option.

Again, a lot of good tips have been detailed on this blog before, so you should check out this link, and also go to Evan's blog as often as possible to read about his experiences being a camera assistant and data wrangler.

Links: The Black and Blue & Digital Cinema Demystified

[via FreshDV]

ditdatadatawranglingbackupharddrives
Editing & Post-Production