Description image

A Beginner's Guide to Backing Up Footage and How 'Toy Story 2' Almost Disappeared Forever

05.13.12 @ 4:53PM Tags : , , , ,

Ever lost important footage because your backups failed, or maybe you didn’t back up at all? Well how about losing an entire feature film and a year’s worth of work? While this isn’t a new story, as it was featured in the special features on the DVD, the tale of the failed Toy Story 2 backup is fascinating. It just proves that even when you’re spending millions of dollars, a few mistakes can be disastrous.

Here’s the humorous animation of the potentially disastrous situation:

I know many of you might be thinking that just because something is deleted, doesn’t mean it’s gone forever, but I have to believe that the animation and the story was greatly simplified from the actual situation. It seems to me that whatever networked storage they were using would have made it very difficult to recover what was lost, or that the time it would have taken to recover that material would have delayed the movie by months. It’s certainly possible for a few IT people to screw up the backups, as you have to be diligent with large storage arrays to make sure that they are not corrupt.


While the story in this case had a happy ending, I’m sure many of you out there have been on the wrong end of file corruption or bad hard drives. I know I’ve had a few cases of bad hard drives or misplaced footage, so I think it’s worth bringing up the idea of having a good backup strategy when making a film. If you’re shooting 4K, you already know it’s going to be a headache, but for most of us, there’s no excuse not to have our footage in two, if not three, different places. Embarrassingly, the reason that the rest of the 5D Mark III/D800 review is not out yet is because I thought I’d had footage backed up in three places, but only had it in two, and it was accidentally deleted from one of my drives, leaving the only remaining copy on a completely different drive that belonged to a friend – who happened to be across the country. Good times.

It’s pretty easy to have an accident like that if you’re not careful, and being on set is one of the more stressful places to do that sort of work. So if you’re shooting a film (or any sort of project really), a couple things you should always keep in mind:

  1. Have a dedicated backup person on set, ideally someone who isn’t the Director or Director of Photography. If you can, make a separate space as far away from set as possible (within reason), so that the data wrangler has a quiet and stress-free place to back up your important footage.
  2. Keep the ENTIRE FILE STRUCTURE of the SSD drive or CF/SD cards. Don’t ever just pull individual clips from a card — always back up the entire drive or card. This can lead to issues in post if the file structure is changed.
  3. At a minimum, have the footage backed up to two separate drives, not just a RAID drive. The issue with just using a RAID is that if the RAID is corrupt in any way, it could possibly corrupt both copies — not good if that’s your only backup. If you can get it on three drives on set, you’ll be in excellent shape for any possible catastrophes.
  4. When you’re using solid state media, like SSDs or CF/SD cards, wait as long as humanly possible before needing to reformat those cards. Ideally you would use separate cards for all of the footage on one day, but if that’s not possible, rotating two or three cards will help keep your just-backed-up footage from being reformatted for at least a few hours.
  5. Develop a system with the data wrangler on set to ensure that cards are only formatted when the footage has been backed up. Using a color-coded tape system is a great way to do this, as detailed by Evan Luzi, who runs a great blog called The Black and Blue.
  6. If you use some sort of backup manager to automate the process on set, make sure that the program will check the integrity of the backups against the originals. I’ve often done backups manually, as time and resources has dictated it be done this way, but a checksum can certainly help prevent bad data on your backups.
  7. When your production is done, take that third backup to a completely different location. Fires and other accidents can, and do happen. While it is tragic, a good way to prevent your hard work from being lost forever is to have another copy in a completely separate location.
  8. If you’ve got the money or the time, LTO tape backups are a great option as an additional backup, but many times there isn’t the budget or the time to do them.
  9. If you’ve got hard drive backups and they aren’t SSD drives, DON’T let them sit on a shelf and do nothing. This is the easiest way for backups to go bad, because drives need to spin every so often to keep the integrity of the data.

While I’m not an expert on data backups, developing a strategy is important to keeping all that hard work safe. In my experience purchasing hard drives, it’s always best to read reviews about failure rates, and try to buy the cheapest one that seems to be working the longest. The best way to purchase drives is to not buy the latest and greatest, as reviews will most certainly be limited, but also because it usually takes a little bit for hardware manufacturers to work out the kinks for a particular model within a brand.

The other way you can minimize purchasing bad drives, is not buying two of the same drive from the same retailer. If a specific production run of drives has an issue, and you buy several from the same place at the same time, it’s possible to get multiple bad drives from that same run. That can spell disaster if you were planning on using all of those drives for the same backups.

As your budget level increases, so should the complexity and redundancy of your storage array. If you’ve got a fairly large project, or a series of projects, you should look into building a storage array that will not only speed up the access times for your data, but also give excellent redundancy. If you’re interested in building a SAN (storage area network), Rich over at Digital Cinema Demystified is currently building one and is detailing the process on his blog.

Something to always keep in mind: when it comes to backing up, data is never safe, so the more places you’ve got it, the better chance you’ll have against a complete catastrophe. You can always buy new hard drives, but trying to reshoot days or weeks is almost never an option.

Again, a lot of good tips have been detailed on this blog before, so you should check out this link, and also go to Evan’s blog as often as possible to read about his experiences being a camera assistant and data wrangler.

Links: The Black and Blue & Digital Cinema Demystified

[via FreshDV]

COMMENT POLICY

We’re all here for the same reason: to better ourselves as writers, directors, cinematographers, producers, photographers... whatever our creative pursuit. Criticism is valuable as long as it is constructive, but personal attacks are grounds for deletion; you don't have to agree with us to learn something. We’re all here to help each other, so thank you for adding to the conversation!

Description image 39 COMMENTS

  • john jeffreys on 05.13.12 @ 5:36PM

    i learned all of this the hard way. i accidentally reformatted my external hard drive one night (in which there was raw footage from a recent project I have not started editing), and it all vanished. i tried to get it restored, but the computer repair people wanted an exorbitant sum of money (and time) to get it back so I just sucked it up and told my client. i ended up getting sued in small claims court for losing the data, even though i refunded him for the services and gave him back what wasnt lost. we ended up settling for a small cash sum, but still, it taught me some great lessons

    • That’s always a difficult situation, and honestly it can happen to anyone, no matter how professional you think they are. I tend to keep buying new drives rather than erasing anything – because I’m so paranoid about losing any work.

  • It’s always good to tell people to back up their stuff. It’s really one of the best properties of digital media that you can even have an identical backup. Having grown up with shooting video and never really using physical film, the thought of having only one real original negative would pretty much scare the shit out of me :).

    BTW I have recently written two lenghty blog posts about the backup strategy I use for working with video and the tools I use to create 1:1 and differential backups of entire harddrives – maybe that is of interest to a few readers here as well: http://www.danielbachler.de/blog

  • Hey Joe – Thanks for the shout out and kind words about me and The Black and Blue! The tips your provided are great and provide a nice crash course for someone who is about to embark on a data heavy digital shoot. I couldn’t agree more that having some system for what’s been backed up and what hasn’t is essential. I once had an AC friend of mine tell me, “there’s two things we don’t joke about on set: the DP’s mom and erasing footage.”

    • Absolutely, your site is essential reading because it’s always presented from real-world, hands-on experience. That’s a great quote! It’s true, neither of those are joking matters. I would say especially the DP’s mother.

  • As a DIT myself on occasion, I agree with most of what is stated here. I use Shotput Pro to copy full cards to multiple locations with, if time allows, MD5 hash checks. My tower has multiple eSATA, USB3, and Firewire ports to accommodate drives of all type.

    In terms of managing cards, I try to personally pull all cards from the cameras on-set. If I’m working for a photographer, rather than a film shoot, this might not always be the case so my workstation also has two containers marked “To Offload” and “Clean”. If it is a film set, I also use a tape system of my own, with cards to offload being taped over the connectors with red tape, and clean drives with a square of green tape.

    The one thing I disagree with is “not buying drives from the same batch”. I think that is a bit of a misnomer, but I’ve heard pros and cons either way. You can get drives from the same supplier, but if you’re paranoid then ask them to check their stock and see if they have drives with manufacturing dates spread out by a few weeks.

    I’m surprised LTO backup wasn’t mentioned a bit further, but in essence it is a tape backup rather than hard drive. Tapes themselves are relatively inexpensive, but the drives to write them cost about $2,000. These are used on high budget productions where a master copy of original footage needs to go into archive. Honestly though, as a DIT, I’ve never had this requested as it seems that it’s taken care of by the post production supervisor.

    • Well what I would say about buying hard drives that way, is that you don’t HAVE to buy them that way, but it can’t hurt – and you’re right, you could always ask. From my experience this has worked well for me, that’s all. I would say if you get a bunch of drives from the same place, before you start putting important data, run them for a couple weeks, as most serious problems will occur in the first month of serious use. While not always true, many drives will fail immediately or in the first week if they are bad.

  • Frank Mareno on 05.13.12 @ 9:47PM

    This is my worst nightmare, I have been lucky for a long time but now I keep everything backed up on dvds and 3 different hard drives 1 of them turned off until back up time which is usually when I get an itch to do it. I even leave files on SD cards until after I’m done editing all my transferred footage because I’m so paranoid.

  • whats a good check sum software, preferably free that would help specifically for red?

    • nevermind

    • There isn’t really. I strongly recommend Shotput Pro, but their hashing is slow because they do the verification process at the end after all files are copied. I haven’t found anything that is as easy or jives well with offloading full cards as that piece of software. You get what you pay for.

  • I use a G-Raid for projects that I am actively working on… once its done, Its all back up on Blu-Ray disks! Love it!

  • DIT checking in, just a few points to consider.

    >Disc images are great, using mac’s disc utility it allows basic checksum verification (no MD5 however)
    >Shotput is great software, but it does not create DMG’s which many employers require like my current one (US NAVY) to comply with NDA’s.
    >Tape system (red for ready to dump, yellow for ready to format cards) is necessary
    >Syncing and transcoding on set with DSLR footage is becoming much more common place, invest in pluraleyes and use FCP7 -> Compressor, do not use Log & Transfer…
    >Stay away from RAID0 and RAID1, RAID50 is fine and RAID6 is great if you can afford it
    >MacbookPro is a must have (for now)
    >LEARN A LINUX DISTRO
    >LEARN A LUNUX DISTRO
    >Seriously, super important for advanced HDD recovery which you *will* need to do at some point as a DIT
    >Backups of EVERYTHING. Cables, drives, enclosures etc.
    >Learn how to solder, learn how to build a computer from scratch, learn how to use all Kona and Blackmagic products you can get your hand on, learn how to color correct, etc.
    >Buy a pelican or a similar case
    >Always use a UPS APC, never plug a cube tap or power strip into one.
    >If you erase footage, *you are not fucked*. Stop what you’re doing IMMEDIATELY, disconnect the device, boot into a linux distro, mount it and run your software of choice to recover the deleted files.
    >If you’re using an external recorder, always record internally at the same time for redundancy.
    >Stay up to date on everything, have all manufacturer contact information and hours at the ready when going on a new shoot, stay on stable firmware, read bug lists for that firmware etc.
    >Troll reduser for DIT advice.
    >Stay the hell away from RED cameras if you can help it, there’s a thousand reasons why but as a DIT, unless you own a RED yourself, it’s going to be more trouble than it’s worth.
    >Alexa’s have a simple workflow, learn it.
    >Find a mentor.

    /braindump

    • Beyond the UPS, you almost always need a power conditioner. I’ve been in motorhomes with spikes in power that constantly force it into battery mode, not good.

      I do agree with learning Linux, some of the live distros have amazing data recovery tools, especially when Mac formatted drives are really sensitive to power failures.

      In terms of hash verification, I’m working on some of my own technology to run MD5 and SHA1 during transfer, which saves a lot of time in comparison to running it at the end.

      I should mention as a RED DIT… the workflow is complicated and you’ll probably be required to have a RED Rocket to hang out with the big boys but it makes processing and transcoding dailies so so much faster.

      • I agree with all of this, especially the motor home quip. RV’s, trailers, motor homes, etc. can be a serious issue for DIT work, not to mention the actual production work that has to go on within if that’s the location.

    • Why Compressor and not Log and Tranfer Chris?

      • Personal preference first off, but after that you also have the many benefits of a separate media encoder. With that being said, you can then set up qmasterd and actually utilize your machine to the fullest when transcoding.

    • A few more things and a quick note. I just finished the first week of a hellish DIT gig that included a fried logic board on my macbook, food poisoning, a failed RAID 0, and a head injury, all on a Navy battleship. Thankfully I was able to recover the drives from the computer and the RAID drive using an external 2.5 and 3.5 adapter, and was able to transfer my software and licenses to my backup macbook. Oh, yea bring one of those too. This is pretty much the most stressful job on set if something fucks up, and it’s the only job on set where you can cost an entire production the whole job if you fuck up. This job is not for those with thin skin or gentle stomachs. It takes an iron constitution and a steel nerve to attempt data recovery from a bare drive ripped from the innards of an enclosure with the thought of a lost 10K day hanging over your head. Whew, I need a beer.

  • ByAnAmateur on 05.14.12 @ 2:59AM

    PC OR MAC CONNECTED TO NAS.
    NAS CONNECTED TO OTHER NAS.

    You run an image os (using something like VMWARE), running your favorite NLE (CS5/AVID/VEGAS).

    This image, we are going to call it ALPHA.
    ALPHA is being saved every 15 minutes on the NAS (synology 4 bay on newegg or a APU with freenas on it) while the O.S that runs your NLE is runned on a SSD.

    As you are editing with your NLE, I don’t even bother saving the project since the images of the live O.S that I am running in are actually state saves that I set in my VM. If the physical computer is damaged, I don’t care, I have the image of my vm somewhere. Even better, since your VM is accessible online, you can be 1000 miles away from ALPHA, be wherever you be,run a VPN download ALPHA, and BAM! any workstation that can run VM or any other Virtual Machine clients can run your project because it is a live state of your project! EVEN BETTER you can go back in time in the libraries of your images… and or ask your guys in color correction or sound to take states of alpha and make it their own run. The interesting point is when this team send back FILES of their stuff and their NAS is mirrored directly to your NAS. The director gets the choice of cuts / color / themes…. It’s an entire culture that has not been spread enough that makes us O.S. agnostic. It’s great.

    This further reinforce the workflow of each one of us (I have never seen two guys with the same workflow), because it forces us to be efficient (the smaller the ALPHA the faster the second team can turn data to products for clients, but it also gives greater quality of decision to the project manager. Everything is live recorded incrementally as you are editing, and only files and not images of the byproducts are share to the upper management of the pyramid.

    The only downside is when a live update of cs5 screws up on of your plug-in.

    However, this is what I am waiting for you guys. RED, SONY, CANON, PANASONIC, JVC, ALEXA, PHANTOM and so many others are throating us down to accepting the form factor of a camcorder.

    They are reinventing the wheel.

    Give me a webcam powered through USB3/Thunderbolt/e-sata whatever port or bus you can think of and give me my cam/laptop so that I can live cast/edit whatever I want to do.

    Think about logitech. think about samsung. think about android tablet with interchangeable camera and large sensor.

    Focus Groups of for pussies and I give credits to RED for that, but I would laugh my ass off if not blackmagic but logitech that could possibly make our dream come true…

    LOGITECH MCX 3200. 6K. 1080p Skype Mode. PL mount.

    Ahahhaa.

    sweet dreams guys.

    • Maybe I’m being picky here, but would you not run into performance issues editing through a VM?

  • ByAnAmateur on 05.14.12 @ 3:00AM

    Oh wow, my wife, pwned me saying that the Acronis way is cheaper and she’s right.

    • . . . how exactly do you pronounce this new web-slang term ‘pwned’ ?? it’s meaning, spelling and pseudo-hipster use has rendered me baffled for months . . .

  • Jonathan Turner on 05.14.12 @ 11:09AM

    I wish there had been more articles like this 5 years ago. I just thought I would pass along my backup approach (I’m only focusing on the data backup, not on-set backup).

    LTO Tape Backup:

    Initially I bought a LTO backup drive only to discover that it was incredibly slow and actually quite unreliable. Also, you need to use special software to build catalogs, if that catalog file corrupts then it is difficult to get your data back. The nail in the coffin for me is that if you want to get back just a specific bunch of files, it’s incredibly slow and painstaking as the medium isn’t random access. I would not recommend this solution unless you are very patient and interested in becoming a DIT.

    My current solution:

    I have a Mac Pro with 4 internal drives. Each drive is mirrored to a single Promise RAID box (in RAID 5 configuration), and I’ve set up scheduled backups using Chronosync (with verification turned on). This takes care of the day-to-day backup, but is only reliable in the short term.

    Concurrent to this I have a project archive system that involves backing up my data by project. When I finish a project I back it up to a bare drive (attached via a dual hard drive dock) and then that drive is mirrored to another bare drive (so I have 2 identical backups, ideally 1 of those is taken offsite). I also maintain an archive on my Mac Pro that consists of just the “Master” Quicktime file as well as a compressed version for posting on Vimeo etc. As I complete projects I maintain a log of each project: how big it is in terms of file size and where it’s backed up. I make sure I always have a version on Vimeo as a a last-ditch backup location.

    The only major concern with this solution is that I need to keep spinning up all of the bare drives because it’s not good for them to sit on the shelf for too long. My theory is that new storage options will keep appearing and the key is putting your data on something that you can easily get it back from. I assume in the next 2 years I’ll shift all my backups to SSD’s or some follow-up to BluRay data disks. With bare hard drives, it’s a really painless process to move that data to a new format.

    I don’t trust any of the cloud backup solutions, maybe just for temp files, but no way for long term.

    Hope this helps.

  • Get a Drobo… or two or three. It an intuitive and manageable RAID-like solution.

    http://www.drobo.com

  • OK: caveat – I work for a company supplying workflows for ‘larger’ post-production, film-makers, broadcasters for avoiding this type of problem at all stages of the workflow… great tips above, especially from Joe.

    Here’s what we constantly preach about:

    1. Any data that is captured on a camera should be wrangled on to two locations asap. Preferably in the field. Data delivered to storage locations should be checked that they are bitwise correct against the original data. There are lots of ways to do this for smaller operations: options include something like Shotput (http://library.creativecow.net/kobler_helmut/shotput-pro/1), which I would mix with something like a G-Tech drive solution from Hitachi.

    2. That’s the raw footage. Keep it somewhere safe. Somewhere safe does not include your SAN where you are editing your data! rm -r *. Viruses. Too much coffee. Too dangerous. The version on your SAN should always always be considered to be volatile. And besides, you want to keep your SAN lean and mean, not full of shots of peoples feet.

    3. Only once the film is edited should you even think about “what source data / daileys should I keep” for longevity – until then keep everything on your version of a “nearline” storage solution.

    A few other things:
    RAID6 is great (much better than RAID5 with today’s data sizes), but a bad controller and you lose the whole system. One copy on RAID6 is not enough.

    Tape, likewise. And if you are using tape then always always always verify the write!

    Of course, I’m not saying that’s all there is to think about when creating a sensible workflow!

    Jon (object-matrix.com)

    ps – is the Toy Story 2 youtube clip for real? Seems a little far fetched in places.

  • whilst on a month long trip I’d filled up an external hard drive with video and it went bad the week before we were to go home. Fortunately I downloaded some disc recovery software that saved it all! So whilst I had only 1 back up it was on a brand new hard drive which failed after very little use.. so that can happen but also trying to recover the data yourself and spending a little time and money on the process is well worth it!

  • We learned the hard way regarding Compact Flash cards… don’t use an external reader to connect to your computer! Who cares if it takes 30 more minutes to transfer, just copy directly from the camera :) Also, the article is correct in using multiple hard drives to copy it to (I have a NAS and 3 local hard drives), plus I don’t delete immediately from the flash card (SD or CF) like the article also states. But one thing I disagree is who does this… particularly for indie. We had one of those backup guys and it turned it out our stuff was stolen! So keep it with the DP and/or editor (in our case, I’m everything, so I keep it all!).

  • I’ve been burned on this as well. It was my first paying client gig. I was ecstatic to have it. The shoot went swimmingly, and everything was set to move into post. The second I dropped my first clip into a timeline in FCP, my drive failed. No big deal, I had backed up all the footage to the client’s external drive. Except not all of the data copied, so they were missing about half the footage. Still, I was ok, right? My DP had it on his macbook. That failed. Both my drive and his laptop failed within 24 hours of each other. Needless to say I lost the job. They still put something together but it was terrible and nowhere near as good as it could have been. I never even got to see the finished product, even though my crew’s footage was used.

    Moral of the story? Backup. Backup. Backup. And then Backup some more. Now I immediately make 3 copies of every card on set when it comes out of the camera. Those 3 drives go to 3 different locations. Then, when I get home, if I can do it affordably, I burn blu-rays of all the files. It’s not as good as LTO tape, but it’s better than a platter drive. Be forewarned, though, a blu-ray will get too hot if you write to it for a long period of time and stop functioning. Therefore you have to burn in increments of about 5 GB. Again, not a perfect solution, but it helps me sleep at night.

    All this data paranoia makes me wish I had been brought up to shoot film.

    • Film can have its own issues – I’ve been there and had them all. Film being loaded incorrectly – lost loops, etc. I’ve also had labs screw up footage and footage not run through the camera and then get exposed. Granted, these things are rarer the more experienced you are, but digital makes me stress just a little bit less even with the potential for losing footage.

  • Michael Locke on 05.19.12 @ 12:31AM

    All the hardware/software talk is moot it it doesn’t make it from the cards! Best investment: lots of media, at least your longest day’s shoot. Leave it there, cards are cheap these days and pretty hardy. Have more than the DIT look at the footage, if possible someone who’s seen what’s been shot (often different locations). If more than one camera is rolling, and you don’t get the last card from them, you may not know it. I didn’t. Lost a remote location shoot with guns, blood, and a cast that’s gone.
    It’s your job on set to make sure every recording is copied. If the sound guy says he’ll get it to you tomorrow, go to your AD or director, you’ll get your files. Demand a consistent location and access to your work station: never put it off! Don’t let loading out from a hellish shoot compromise your quality/review process. Be an ass, let everyone else carry gear, just sit and make sure everyone’s work is being preserved. Don’t find out later you didn’t prioritize the hat’s you were wearing. Then you’ll know Hell.

  • Regarding DIT’s and data wranglers.
    I would strongly agree about the need of having a software that is able to check and verify that your files are ok. (MD5 which is faster or SHA1more secure, more reliable)
    Shot put pro is quite good at what it does and is easy to use by anyone. The price is quite fair for what it can do.
    There is a free application (i can not recall the name). it is free but quite hard to handle at first.

    Currently i’am using Silverstack Pro from Pomfort, although somewhat pricey (599 euros for the set edition) you can get better price (45% off) if you are a student.
    It does what is supposed to and it does it very well, of course some minor details still require attention from the user.

    I had worked as DIT and Data Wrangler on 2 Feature Films, few commercials and other events and on dozen short films, only twice, in the beginning i was forced to do the backing up by hand (copy-paste). If you can, don’t do it. The errors in the files can be multiple and hard to find by hand.

    The responsibility is enormous but very few crew members are aware of that (that is until the moment when something goes wrong).
    The problem is that when the shooting is over, most of the crew members are going home, to rest but not you.
    You are going to your backup site and you are checking, backing up, checking again and generally overseeing the footage. Not to mention if you are tasked with primary color grading or masking the footage.

    Right now i’am on the feature film where i hold the DIT position “mixed” with 2nd AC and the production wanted to add to that, Assistant Gaffer, which of course was not possible even if you would be drinking red bulls mixed with espresso and fresh guarana(but it most probably would kill you, eventually).
    You need to explain to the people responsible that the work you do is crucial and if you are running all the time doing something else, you are asking for trouble. Check and re-check the footage then backup and backup some more.

  • Ok, I used to work in the Systems department for a big movie production house. What bonehead with root access would do something so completely retarded? Usually people with root on shot disks aren’t the type to rm -rf an entire box on accident. I cry fowl play.

  • What concerns me most is that many cameras can’t write to two cards at once.
    So the time from shooting a scene to when it’s copied to a computer you only have this one copy.
    I have never lost any shots or pictures, but it’s bugging me, because a flash card that fails is an entirely possible scenario at any time…

  • “At a minimum, have the footage backed up to two separate drives, not just a RAID drive. ”

    That is a very good point, because a RAID array is not a backup. A RAID array guarantees high availability, meaning that in the event of one drive failing, you can keep working while the array is restored on a new drive. However, it is not in any way a backup.

    A backup has to be physically seperate from the machine you are working on, so that in the event of files being deleted, or a fire, flood, or lightning stroke, the backup is in a safe place.

  • Reminds me of the cryptic lines: There are only two kinds of people. Those who have lost data, and those who will. Kind of like we are all mortal so will eventually die. Wonder how many millennium can our classic films can survive, and what format(s) will be used? I guess that’s more about archiving but is a related topic that Martin Scorcese is concerned with.