Most of us who are shooting on SD cards, CF cards, or some form of solid-state memory enjoy a number of benefits that being in a digital workflow provides. Some of the biggest Pros involve sheer convenience factors like the ease of making digital copies, and the ease of storage or backup. Some of the biggest Cons, however, involve the cost of storing those large amounts of data, the logistics, and the fact that hard drive media is a bit more fragile than tape-based backup. If you're looking for a piece of mind solution, and download speed isn't an issue, Amazon Glacier can provide what you need for an affordable cost. Hit the jump for more details, and an incredibly intricate graphic from yours truly:
...Hey, you figure out a way to visually represent a cloud server called Glacier, alright?
From the highlights page on Amazon Glacier:
Amazon Glacier is an extremely low-cost storage service that provides secure and durable storage for data archiving and backup...Amazon Glacier is optimized for data that is infrequently accessed and for which retrieval times of several hours are suitable...customers can reliably store large or small amounts of data for as little as $0.01 per gigabyte per month, a significant savings compared to on-premises solutions.
Archiving
So as you can imagine, you wouldn't want to use this service for a project currently in production -- it is purely for archival. I can certainly see the appeal of having my backup drives in the cloud. The accesibility to download them anywhere, even at a cost of "3 to 5 hours" per "archival job." I definitely have many redundant 1TB hard drives sitting in boxes that I have to start up/run/re-stock -- and it appears I'm in good company, as Emile Hanton at CineTechNews explains:
Old projects you’re likely never to touch again but still need to keep around incase a client asks to pull some old footage. Or perhaps it’s the original project files from your short film that you uploaded to Vimeo a number of years ago...What ever form it takes, we all have data sitting on hard drives that are only meant to have a lifespan of 3 to 5 years. So what do we do? Some just keep moving data from one hard drive to another every few years. Some backup to LTO tape, but even LTO Tape, which is suppose to have lifespan of 15-30 years needs to be rotated. I’ve even heard people say LTO needs to be rotated as frequently as every 5 to 10 years. If you’ve got a lot of data, those hard drives and LTO tapes can add up quickly and let’s not forget multiple copies for redundancy, plus offsite storage incase disaster strikes.
The hard drives I store locally contain independent film projects, stuff that I don't have the money to place into a safety deposit box as Hanton suggests in the above article. I have to differ with him on the LTO Tape front, as in the corporate and commercial world, I have had great success pulling back very old archived data from tape backup. Still, I agree with him that the whole rotating of drives and storing offsite (typically at another filmmaker's house that has a stake in the project, in my case) is a hassle that Amazon Glacier could help avoid.
Cost
So there is a convenience factor to Amazon Glacier, admittedly at the cost of being slower than pulling directly from a local drive. The next big question is of course monetary cost. Hanton does a great job summarizing this:
To store 1 TB of data in the least expensive Glacier region it would cost you $10 / month or $120 / year...Take that Glacier storage cost out to three years (the warranty period for most drives) and it has cost you $360 to store 1TB of data, not counting additional costs if you needed to retrieve anything. Where as you could have spent $80 on a hard drive, put it in your closet and had no additional costs if you need to retrieve the data...Now, let’s consider the cost to keep multiple backups. Three copies is my magic number, so $240 for three 1 TB drives. I need to store one of those copies off-site in a safe deposit box or other type of secure storage and that will probably cost about $10 month. That’s the cost to store 1TB on Glacier right there.
I also back up to three drives, though again I typically forego the safe deposit box for a trusted friend or filmmaker. Amazon is championing this service as economical, however in a few years time, you will be paying several times over for what you could handle yourself at home. This is the killer for my independent projects, however I must admit that if I were charging back to a client for storage fees, Glacier would make a lot of sense. And typically for corporate and commercial work I've done post in, there's already a built in storage fee and 7 year limit anyway, making storage the client's problem after that point. Speaking of which -- ever emptied out a closet of betas into the back of your 2005 Grand Am? ...Uh yeah, I mean I haven't either... I drive a Porche. (No I don't.) Anyway, transporting physical media is not fun.
The Cloud?
The other hesitation I have is just with cloud computing in general. Call me an old fogey, but personally it was a kind of "trust transition" getting used to solid state or spinning drive versus dv tape. I suppose it will be a similar transition going from local hard drive to the ominous cloud. But there's something different about not having any access whatsoever to the physical drives your media is stored on. I mean even with LTO Tape services you can get access to the media -- and we're not talking about your Grandma's cooking recipes on Google Drive... this is your blood, sweat, and gaff tape all enclosed in crucial .MOVs. I would never make an archival project backup to Google Drive or services like Dropbox. What's different about Amazon Glacier, other than it being a paid service? It's worth noting that the folks over at Wired seem to have a few hesitations as well.
I suppose I would just prefer Glacier to take the place of one or even two of those three backups I tend to make. And to Glacier's credit, you can mail them hard drives to be uploaded to or downloaded from, ala their AWS Import/Export service. But again, it's at a cost -- a "$80.00 per storage device handled" cost, plus about $3 an hour for transfer.
What are your thoughts on Amazon Glacier? Do you think you could use this in your independent, studio, corporate, or commercial workflow? Another question for you indies -- is this worth building into your post-production budget?
Link: Amazon Web Services - Amazon Glacier
[via CineTechNews]
[Cloud photo by Flickr user Karin Dalziel (CC), server photo by Flickr user Christian Haugen (CC), glacier photo by Flickr user Jay Galvin]
Your Comment
26 Comments
"The Cloud" is scary. I'm with you completely on the need to do multiple backups, and I just don't think backing up to something intangible is a good idea. What happens if someone or some group decides to hack these databases for the hell of it? It's not a matter of "if" a hard drive or cloud storage fails, it's "when".
I really wish someone on here would write an article about this: http://motion.kodak.com/motion/Products/Lab_And_Post_Production/Archival...
Backing up to film is a great point that was brought up in the "Side by Side" documentary because you don't have to keep making copies and thinking. I'm at least going to back up my demo reels and projects I'm extremely happy with to this.
September 21, 2012 at 7:04AM, Edited September 4, 7:54AM
Hey Brendan, great link and excellent point. I typically don't think film when backing up, but yes it does make a ton of sense for the final deliverable. However, it may not make as much sense to back up all your raw footage or alternate takes. Also, if you've ever got a tip or something, feel free to contact us with it - your idea here definitely has merit and value, for example.
September 21, 2012 at 7:42AM, Edited September 4, 7:54AM
Ok will do! And yes, backing up large amounts of raw footage to either film stock or even cloud storage online (rather than sending in the drive as they offer) would be insane.
September 21, 2012 at 8:34AM, Edited September 4, 7:54AM
I agree, I always thought the whole could idea was pain stupid choice for anything of value. Hackers can steal Millions of dollars from banks who have far better security than amazon, what's from making me believe that my footage that is sensitive won't get in the wrong hands?
September 21, 2012 at 8:07AM, Edited September 4, 7:54AM
Yeah, did you see one group's recent claim to shutting down GoDaddy for a day? It was to "test cyber security and other reasons"
September 21, 2012 at 8:30AM, Edited September 4, 7:54AM
No I didn't interesting, there is no way I'm ever using a cloud, unless it was to back up software CDs or something in that nature.
September 21, 2012 at 10:08AM, Edited September 4, 7:54AM
FYI, that was a lark...the registrar in question actually screwed up their own systems and after one random fool claimed responsibility the entire world went "oh noes" before fact checking.
September 21, 2012 at 5:16PM, Edited September 4, 7:54AM
Does Amazon Glacier have any advantages over other services, such as Backblaze?
September 21, 2012 at 7:48AM, Edited September 4, 7:54AM
David, good comment as this has lead me to Backblaze. I have to admit that it's looking pretty good compared to Glacier right now, though connection speed isn't mentioned in anything I'm reading on it. Check this article comparing Backblaze to Amazon S3, the more expensive/faster version of Glacier:
http://www.wired.com/cloudline/2011/10/backblazes-basic-cloud/
September 21, 2012 at 8:14AM, Edited September 4, 7:54AM
+1 for backblaze...I use it to back up all of my files, including nearly 1TB of video projects. It's not massively fast uploading from the UK, topping out at about 0.4MB/s but it gets there in the end, a useful addition to my 2 onsite backups. I've never had to do a full disaster recovery but I've needed various files before when I've been away from home and just pull them down. Very easy, and stupid cheap :-)
If you're concerned about privacy (and don't trust BBs encryption) you can always pull down truecrypt and put your files in disk images!
September 27, 2012 at 3:52PM, Edited September 4, 7:54AM
I work at Backblaze and I've been reading NoFilmSchool for a while and love seeing this topic here. David, Backblaze doesn't have any bandwidth caps or throttle your upload speed. It's $5/month for unlimited storage. If you are shooting a lot of video and can't upload as much as you shoot, then it might not be a good solution. I recommend to my friends that they always store at least 3 copies of their files; original, local backup and offsite backup. And to be extra paranoid a fourth copy with a friend.
Amazon Glacier is a great tape replacement. It's designed for archiving a big chunk of data - but not so great for incremental backups.
September 21, 2012 at 10:48AM, Edited September 4, 7:54AM
Hi Casey. So does Backblaze have any sort of a limit, then? If not, why wouldn't I just use it to back up a few terabytes of data?
September 21, 2012 at 9:50PM, Edited September 4, 7:54AM
Backblaze doesn't have any limits in the product. We have some users backing up over 7 and 8 Terabytes. However, there is only so much data that can be uploaded given people's bandwidth. And restoring that much data would also take a really long time.
September 22, 2012 at 11:24AM, Edited September 4, 7:54AM
Will not use. $10 a year maybe. $10 a month? No.
September 21, 2012 at 8:08AM, Edited September 4, 7:54AM
Funny, Frys has an ad out today for 2GB for $90. This has been a headscratcher for me for years. Harddrives will constantly outperform the cloud... but Harddrives fail. The last thing you ever want to do is go buy 3, put your labor of love or your clients work on them only to find out next year, "It was a bad batch'
Then there's restoration time. How long does it take to download a terabyte?! Probably half as many days as it takes you to upload it. The constant upside of local storage is 'speed of the restore' (Oh, how I long for the days when I only had 1 TB of archive). LTO is the way to go. High threshold of cost to get in, but for $40, you get 800gb/1600gb and backups are much more stable. So that's almost $120 for that triplicate backup scenario, (at less than a TB or time in a half, depending on your LTO version) but if your house burns down, Hope you had all of that equipment insured.
BluRay is a very cheap compromise - 23gb of storage for $0.25 after a $60-$80 dollar outlay for the burner drive. Yes, it takes 44 disks to back up a TB, ($15) and you have to cataloge... but it's by far the cheapest storage solution that's still effective, if you're only working on commercials, Music videos and shorts. Working on a feature? BluRay is not a real option. The only feature I've worked on exported to 90gb at full HD... oh well.
I say LTO is a 'studio' grade investment, Cloud is good for the hobbyist, and Harddrive or BluRay for the hustler in the middle.
September 21, 2012 at 8:42AM, Edited September 4, 7:54AM
The only caveat I'd place on all that is that whether you're the hustler or the hobbyist in that scenario, for the love of all that's nerdy PLEASE retain at least two copies of everything. It only sounds expensive until you look at the cost/benefit of simply having one-more-copy. I look at something like Glacier (which is going to promptly decimate Backblaze, by the way) as a great, inexpensive way to get that critical "third copy." The one that's there in case of the bad batch of hard drives, the bad spindle of optical media, or the Very Bad Day that something happened to your living space or office.
September 21, 2012 at 9:11AM, Edited September 4, 7:54AM
This is EXACTLY the cloud solution I have always wanted, and I've overjoyed that a proven company is finally offering it. I've been clamoring for years now to have someone offer me offsite storage that's cheap, slow, but reliable...to serve as the "third copy" for all the things that I keep duplicated at least twice locally (everything).
I'd never propose that anyone should entirely trust some random distant service as the ONLY copy of anything. That's patently insane. But in my opinion, any user/consumer who uses "the cloud" that way is just Doing It Wrong. When you put stuff in one place, it's not backed up! Backups are duplicate copies of things.
If you don't keep duplicate copies of something you claim is important to you, the reality is...it's not important to you. That's the long and the short of it. Every geek has had to deal with data loss horror stories from muggle friends, and that's the most useful response you can give. Hold their hand and teach them what backup really means.
This is a great, affordable way for people to get a third/fourth/etc copy maintained of their valuable digital content, that's not endangered every time a power spike hits that array of multiple drives they leave plugged into one power source at one apartment in one building...because they think that's "redundant."
I can't wait to hard sell all my clamoring tech support needy friends on this, it's going to save us all so much pain...and cost so very little for doing so.
September 21, 2012 at 8:50AM, Edited September 4, 7:54AM
I'm surprised at many for being reluctant to cloud backup as a viable offsite storage solution. I use to use Mozy, but it got expensive, and I'm currently looking for another storage provider. Most are encrypted, but you can also encrypt files yourself. I mostly backup photos and would like to archive video.
Currently I keep two or three copies, but they don't get done daily. I have a laptop, backup drive, and thumb drive for transferring personal files and download to and from work.
I have a NAS, but it is slow and clunky. I used to swap backup drives and keep a copy at work, but I don't do it as often as I should. Ideally I would like a cheap cloud backup to use as a last resort in case my home gets robbed or burns down. If someone wants to hack into my photos I'm not really concerned.
September 21, 2012 at 9:23AM, Edited September 4, 7:54AM
For me the killer feature (and the reason I wanted this story on this site!) is the AWS Import/Export service. With HD video, backing up a terabyte of data over a typical broadband connection would take... a month? A year? Sending them the drive and having them upload it is exactly what I'm looking for. And no, this shouldn't be your primary backup -- but if you're making three, I think it looks like a good option for one of them.
September 21, 2012 at 9:31AM, Edited September 4, 7:54AM
Ditto all of this! I just read of this here this morning and I'm already pouring through years of AVCHD getting it better broken down and organized, and I'm looking over at one of my bare drives sitting in a dock just waiting to be filled up and mailed off. The speed at which "Glacier" is going to make me feel better about my data is immediate. ;)
September 21, 2012 at 10:04AM, Edited September 4, 7:54AM
Thanks for the article; I kind of share your reservations about the cloud. I think the cloud providers have work to do to document, publish, and have externally audited their policies for data security (against both accidental loss and deliberate breach).
However in your summary of the cost/benefits, I think there's one point you've missed -- obsolescence. I'm picturing you in a few years' time, holding a drive in one hand with a USB port, and laptop in the other with a Thunderbolt (or some other future thing). People are rightly worried about stored media in closets (disk, tape) becoming unreadable -- but there's a huge difference when the data is on a live server, powered on 24/7. Basically what you're buying is that Amazon will take care of mirroring the data over to newer drives as the old ones are unplugged and tossed. That takes care of a large part of the obsolescence problem (though not all of it).
September 21, 2012 at 1:21PM, Edited September 4, 7:54AM
To me it makes more sense offering the client a one time fee of a $100-$200 data storage option (for x years) and taking care of it on my own by having several projects on two 1TB drives (i.e. two copies but you can do three if you really want to.) As drives become cheaper, you buy new ones, consolidate the projects and you can charge less and less.
September 22, 2012 at 3:13AM, Edited September 4, 7:54AM
So you put irreplaceable stuff on two hard drives, which may even have been manufactured at the same time, and you expect it to still be there several years later? Even doing three copies that way would be considered insane by a system administrator.
September 22, 2012 at 11:57AM, Edited September 4, 7:54AM
I don't mean to sound harsh, my point is who's *maintaining* those hard drives? Who is plugging them in, spinning them up, and performing thorough checksum comparisons to make sure neither one of them has degraded...that every byte of every frame is still there and still identical? How often is that being done? That's the sort of extra overhead that goes into real data archival, and that's the sort of thing you'd expect from a quality cloud service. ZFS at home plus Glacier in the cloud sounds pretty great for an affordable personal or small business solution to me.
September 22, 2012 at 12:34PM, Edited September 4, 7:54AM
When comparing archiving systems, a lot of the focus tends to be on the cost of the media rather than the durability or reliability of the system.
Hard drives have relatively high failure rates. Archiving media on 3 drives sounds like it would be reliable, but it's not going to get you the annual durability of 99.999999999% and self healing archives that Amazon promises.
I suspect that LTO tape would provide a level of durability that approaches Amazon's if you had 3 copies in different physical locations. The downside with tape is that while the cost of media is lower than using Glacier, there are operational costs associated with rotating tapes, checking them, transporting them to multiple secure locations. These costs are easily overlooked and hard to calculate.
I think Amazon wins in terms of durability and long-term simplicity, and may match tape in terms of cost when you factor in the total cost of ownership. That said, Amazon has some downsides as well. The first major one is the pain of ingesting data. Uploads take forever when you are dealing with terabytes. At $80 per drive handled, the import service that Ryan mentioned is expensive. The second major downside of Glacier is the cost of retrievals. If you download more than your free daily allotment of data retrievals, the costs can be very steep.
This post has a bunch of cost scenarios for using Amazon Glacier for archiving video. http://www.screenlight.tv/blog/2012/09/11/video-backup-and-archiving-for...
I guess the question here is whether the material that is being archived is
September 22, 2012 at 1:25PM, Edited September 4, 7:54AM
I use virgin media unlimited backup and storage. £5 a month here in the uk. Brilliant for longterm offsite cloud storage.
September 27, 2012 at 2:31PM, Edited September 4, 7:54AM