March 31, 2016 at 2:29PM

0

external RAID solutions for 4k

Hey everyone. First time, medium time.

I've seen a few forum topics similar to this but I wanted to put forward my hypothetical setup and see what others thought. I'm currently on a cheese grater Mac with JBOD as my storage setup. I'm going to be buying a maxed out 5k iMac soon, which means I'll finally be forced to start from scratch with storage instead of juggling disks all the time and archiving constantly.

My clients are mostly non-profits, with a few corporate clients here and there. None of them have asked for 4k delivery yet, but I know it will happen someday, and editing in 4k is just around the corner for me. I'm currently shooting with a C100 and GH4 and editing in Premiere. Projects are generally short documentary in style and I'm not doing any 3D fx or compositing. Some motion graphics in AE and color grading, but that's about it.

My goal is to have at least 6 TB available for active projects on a RAID 5 for redundancy. I want a RAID 0 setup for my backup drive with more space than that so that Time Machine can keep versions of my project folders. I've deleted an entire interview before by accident and my auto-backup deleted it as well since it wasn't versioned. I'll archive on SATA disks using a dual bay enclosure (already owned).

I'm pretty confident about the above criteria (although open to other suggestions), it's just the specific products and features that I'm unsure of. Below is what I'm currently considering, specifically:

Active projects in RAID 5: Promise Technology 8TB Pegasus2 R4 Thunderbolt 2 RAID Storage Array (http://www.bhphotovideo.com/c/product/1013099-REG/promise_technology_p2r...) - This would give me 6TB to work with in a hardware-controlled (I hear that's important?) RAID 5 and 4 bays which should improve speed. My main question is, forgetting about the importance of redundancy, will I notice a speed difference between a Thunderbolt 2 4-bay hardware-controlled RAID 5 and a USB 3 2-bay Raid 0 working at the level I am right now? Again, I recognize the other benefits to the Pegasus R4, but it's so pricey for such a small amount of space, especially if I'm not going to notice a speed difference working at my level.

Time Machine backup drive in RAID 0: WD My Book Duo 12TB (2 x 6TB) Two-Bay USB 3.0 RAID Array (http://www.bhphotovideo.com/c/product/1053138-REG/wd_wdblwe0120jch_nesn_...) - If I get a larger active projects drive, this will also have to grow. But if I stick with 6 TB of usable storage, is 12TB a good size to be happy with using Time Machine for several years? Also, I'm assuming there is no benefit to having a Thunderbolt 2 RAID as my backup drive.

With all that being said, the things I most want to avoid are defective enclosures and setups that have a tendency to disconnect the drives at random. For this reason I have not looked very hard at OWC's solutions, although their pricing is much more appealing for larger amounts of storage.

Any thoughts are welcome. Thanks!

23 Comments

RAID 0 for a backup does not make any sense.
RAID 5 seems overkill.

Make two backups of the raw data and have a working disk, that should be more than enough.

It seems to me you are going to spend a lot of money for something you do not need.

March 31, 2016 at 2:49PM

8
Reply
avatar
Cary Knoop
Member
2223

Hi Cary, Just to make sure you're understanding my question:

I'm looking at RAID 5 for my active projects which I've read several times is ideal for editors. the RAID 0 would be where I'm backing up that RAID 5 with Time Machine. I'm trying to avoid human error and the slog of manually backing up data every time I add anything to a project folder. So I would have an active projects drive with redundancy and then an automatically backing up 2-bay RAID 0 that just acts as a big single drive so that Time Machine will not bug me about the single drive getting full.

If you understood that the first time, are you telling me I should avoid RAID setups altogether and just get a bunch of beefy drives like I have been all this time? If I do that then Time Machine (with its versioning ability) won't be able to help me much since it needs 1 target location for everything. I should have mentioned that automation is another priority for me here.

March 31, 2016 at 3:04PM

9
Reply
David Ells
DP
88

I edit 4k all the time and disk speed is seldom the bottle neck.

And if it is use an SSD or, even faster, use a ram disk (in case your NLE does not allocate enough memory).

In my opinion you be better off spending your money on a fast muti-core machine with lots of memory (minimum 48GB).

March 31, 2016 at 3:13PM, Edited March 31, 3:30PM

6
Reply
avatar
Cary Knoop
Member
2223

Thanks Cary. That's helpful to hear about the source of bottle necks. The problem with going Mac Pro (I'm not willing to switch to Windows again) for a multi-core setup is I won't get a new display with it, which I desperately need.

I'd never considered an iMac until I saw this video: https://www.youtube.com/watch?v=NJt3av99e8k . Apologies if everyone here has seen it thousands of times. This was compelling, especially considering that the iMac he tested there is already a generation old.

I would be spending significantly more getting a new decent display and and buying an 8-core Mac Pro (as opposed to the 6-core which under-performs compared to the iMac on most tasks) then I would buying this setup with a RAID.

March 31, 2016 at 3:33PM, Edited March 31, 3:33PM

1
Reply
David Ells
DP
88

In terms of speed the RAID 5 should be as fast as your USB 3 RAID 0 set-up.

RAID 5 is pretty safe storage. One drive can die and then you just swap in a replacement and let the RAID 5 box automatically rebuild the RAID back to it's previous state. The only way you can lose data with a RAID 5 set-up is if two drives both die at the same time, which doesn't happen very often. ( it's never happened to me in 10+ years of using a RAID 5 box )

If you can afford it you might consider putting in larger SATA drives into your RAID box, as most new RAID enclosures can take up to 6 TB drives, which would give you a 18 TB of very fast protected work-space with a four drive RAID.

I only archive projects when they are completely finished, and then I do it by manually copying project folders from the RAID to a bare drive in a SATA dock. It's slow to copy for large projects, but then I don't archive more than once per week.

March 31, 2016 at 3:14PM, Edited March 31, 3:18PM

0
Reply
Guy McLoughlin
Video Producer
32889

Thanks Guy. It sounds like you are already using the workflow that I'm envisioning. I assume you also have a backup system in addition to your RAID 5 redundancy before you archive? Can you tell me about that?

March 31, 2016 at 3:52PM

25
Reply
David Ells
DP
88

Actually I don't. RAID 5 is pretty much bullet-proof. ( just keep a spare drive around in case one drive actually dies )

For larger projects I keep an external SATA drive that I use to back-up all camera footage and raw audio for the project, so if I accidentally delete something from the RAID I can go back to the original camera footage. I normally don't delete anything from the RAID 5 until the entire project file has been backed up to a SATA drive.

I also use a great little Windows app called SuperCat ( http://no-nonsense-software.com ) that instantly catalogs all of my external SATA archive drives and makes it very easy to figure out which SATA drive has the archived file I want to restore. ( sometimes you just want one little file, but you can't remember where it's been stored )

The only drawback to using external SATA drives to archive work is that you have to run each drive for a few minutes every year or they will eventually lose their magnetic storage. At the beginning of every year I do a generic file search on every archive drive which takes a few minutes, and ensures that the drive is good.

Overall I've been pretty lucky and have only lost one hard-drive over ten years, and have never lost data so far. ( rebuilt the RAID 5 and all my data was still there )

March 31, 2016 at 4:21PM, Edited March 31, 4:27PM

0
Reply
Guy McLoughlin
Video Producer
32889

You wrote: "The only drawback to using external SATA drives to archive work is that you have to run each drive for a few minutes every year or they will eventually lose their magnetic storage."

Really? Never heard of that!

March 31, 2016 at 5:17PM

22
Reply
avatar
Cary Knoop
Member
2223

I think you may be playing with fire not backing up your RAID 5. If that enclosure every tanked magnificently then even a RAID 6 would be lost. But I guess there's always something that could render your backup solution useless no matter how many we have.

What enclosure have you been using for RAID 5? Is it hardware or software controlled?

April 1, 2016 at 9:23AM

0
Reply
David Ells
DP
88

Offloading to an external SATA drive for archiving is playing with fire. Spinning it up every so often helps, but it's not exactly bulletproof. Drives fail—all the time. SSDs aren't much safer.

Of course, something is better than nothing. But don't just throw stuff on drives and feel like it's safe for ever.

Data retention on HDDs, SSDs, and even optical media is not very good. Not if you're talking in terms of years vs. months.

April 1, 2016 at 11:55AM, Edited April 1, 11:55AM

0
Reply
avatar
David S.
3260

As far as backing up to RAID 0 goes, please understand this means you're doubling your chances of data loss because if one drive goes, you lose everything.
You might consider Editing on SSDs (one per project) and using Carbon Copy Cloner to back up to RAID 1 (say, automatically every 30 minutes). Over time this will save you lots of money (RAID 1 is way cheaper than 5, and you can format and reuse the SSDs when you finish a project), and will be faster for editing.
If money is no concern, RAID 5 is good (safe, good read speeds, not-so-good write speeds), and RAID 1+0 would be ideal (very fast and very safe).

March 31, 2016 at 3:26PM, Edited March 31, 3:33PM

21
Reply

Thanks Ethan. I get the risks with RAID 0. I also have an off-site backup that I upload to via Crashplan. I don't have the fastest upload speed so sometimes it lags behind my projects a bit, but after a few days it's completely caught up. So I do backup twice as a rule. I should have mentioned this in my top post but I was worried that it was getting too long.

I don't think I can do the SSD route. I am juggling up to 6 projects simultaneously at times and with the relatively few ports on the iMac I think I'd go crazy. Clients are also always dragging old projects back up so I like to keep projects that are up to a year old in my active working drive.

The RAID 1 solution is problematic for me because it limits my automation of backups. I think I could only get up to 6TB at once with a mirrored setup and Time Machine would fill that up very quickly if I'm working off of 6TB with versioning.

March 31, 2016 at 3:48PM

0
Reply
David Ells
DP
88

I’m sorry to hijack this conversation, but I am currently building my own budget work and storage solution, and I have one question: I’m planning on working off of two 1TB hard drives in a RAID 0, backing up to one 2TB hard drive. Is there any specific format that I should use? Obviously FAT32 has the 4GB limitation, so that’s not an option. But I’m unsure whether it matters or not which one I use other than that (I’m currently working on a 2013 Macbook Pro).

Again, sorry for hijacking this post.

March 31, 2016 at 6:03PM, Edited March 31, 6:03PM

3
Reply
José Pedro Pinto
Wannabe
751

Hi Jose,

No problem! I would either use exFat or Mac Journaled as formats. ExFat just if you think you might share that drive with a Windows user at some point.

April 1, 2016 at 9:12AM

1
Reply
David Ells
DP
88

Thanks a lot for the reply! That's what I was thinking, just wanted to make sure to avoid making a dumb mistake.

April 1, 2016 at 2:36PM

2
Reply
José Pedro Pinto
Wannabe
751

I think you have it backwards—RAID-0 doesn't make any sense at all for archiving. You're doubling your chances for data loss with zero benefits. If you need the space, hard drives are cheap.

If it helps, here's my system—take it or leave it:

Active projects live on a Raid-0 drive. A cheap USB drive lives next to it and Carbon Copy backs up the active drive to the cheap drive for safety. Whenever my Raid-0 loses a drive (happens once or twice per year on average) I pop in a new drive and restore from the backup USB drive.

When projects are done, I move them to our NAS (I believe it's set to Raid-10, but the principle still applies) since it has internal redundancy to protect against data lost.

At all times, Backblaze is running and keeps emergency back ups in the cloud. Never had to use it, hope I never have to.

Additionally—Time Machine sucks. Seriously. Buy Carbon Copy, it's way better and easier to use.

April 1, 2016 at 11:51AM

4
Reply
avatar
David S.
3260

A little bit of information from the IT side of the world.

RAID5 used to be really popular because if/when a drive failed, you could just swap it, wait a couple hours for the array to rebuild and continue. However, rebuilding a RAID5 requires immense amounts of new parity calculations and substantial read and writes to the whole array. With disks less than 1TB, this is hours. With the bigger disks we use these days, this is sometimes more than a day of work for the array.

(Not-so) Hypothetical situation: If you have four drives in an array, and one dies from old age, how much can you trust the rest of the disks? Do you want to find out by subjecting them to 24 hours of stress testing? If you have another drive fail during this process, you will lose some data, likely all data (depending on how sophisticated the RAID controller is).

One of the reason myself and a lot of IT people use RAID 10 for anything performance and business critical is if/when a disk fails, the restore operation is simply reading from one disk and writing to the new one. Much less prone to failure and much faster. Yes, you do get only 50% total capacity, but you can potentially have 2 drives fail (assuming they aren't the mirrors of each other) and still rebuild your data. Plus, you have the same speed as a RAID 0, which is faster than a RAID 5.

Now, for my personal rig, I use an SSD boot drive with a small (4TB usable) RAID 10 inside my workstation. When a project is completely finished it goes onto an external RAID 1 array for long term storage. I use RAID 1 for cold storage because read/write performance isn't required.

Just my two cents!

April 1, 2016 at 1:45PM, Edited April 1, 1:45PM

5
Reply
avatar
Mike Racine
Filmmaker
359

Almost all of the data disasters I've known of from friends that work in IT have involved very old drives ( 5-10 years old ) that were not being maintained properly.

I personally think hardware based RAID 5 boxes are a good solution if you are willing to replace all of your drives every 4 years or less. ( I've been buying the Western Digital RED NAS drives for the past couple of years on the assumption that they are more reliable than the standard WD GREEN or BLACK drives )

Yes RAID 10 is even better but you have to be willing to sacrifice quite a bit of drive space for this.

April 1, 2016 at 2:46PM

0
Reply
Guy McLoughlin
Video Producer
32889

I agree that RAID-5 has lots of problems in the >1TB drive world, as do the filesystems both windows and OS X use. For critical systems, we prefer software RAID (we gain speed as our systems improve) and ideally better filesystems (we prefer ZFS and use either RAID Z2 or RAID Z3). We do not consider RAID as a back up, but just to improve reliability while working (to prevent a failure during editing, rather than to prevent data loss in general).

We back up our data to some combination of: other systems, stand alone drives, Crashplan, and/or dropbox.

We expect drives to last 3 years and try to cycle them every 2.

For all our ZFS file systems, we scrub our data twice a month at least.

April 3, 2016 at 12:01AM

0
Reply

>>>re: Hard drives losing their data over time

I haven't researched this in quite a while, but the basic premise is that all magnetic hard drives will eventually lose their stored magnetic field over time, so the drive data will need to be periodically "refreshed" in order to retain the data on the drive. Last time I checked the consensus was to "refresh" a drive every year to be 100 percent certain your data was still there and readable.

Though I'm starting to think that my method of "refreshing" my drives may not be reliable. In digging around for more details today, I stumbled across a Windows app that is designed to refresh drive data, so I think I'm going to give this a whirl...

DiskFresh : hard drive maintenance tool
http://www.fact-reviews.com/info/diskfresh.aspx

...I know that hard-drives are generally not used for permanent archival storage, but so far I haven't needed data storage for more than 5 years. There is a product called M-DISC which is a writable DVD or Blu-ray optical disc that claims 1,000 years of archive life. It contains a single inorganic recording layer instead of optical dyes.

https://en.wikipedia.org/wiki/M-DISC

April 1, 2016 at 1:58PM, Edited April 1, 2:32PM

0
Reply
Guy McLoughlin
Video Producer
32889

>>>I think you may be playing with fire not backing up your RAID 5. If that enclosure every tanked magnificently then even a RAID 6 would be lost.

I've used more than 40 drives over the past 10 years and so far I've only lost one hard-drive, but never lost any data because I was able to rebuild the RAID 5 by swapping in a new drive. ( I replace all my drives every 3-4 years because I don't trust drives older than 4 years old )

Yes, there is a small risk of losing data if two drives fail at the same time, but it's a risk I'm willing to take. As I mentioned previously I do keep back-ups of camera and audio files on external SATA drives, so if my RAID 5 was somehow destroyed I would still have the raw unprocessed media to start-over with.

>>>What enclosure have you been using for RAID 5? Is it hardware or software controlled?

I avoid using software RAIDs because I want a completely stand-alone RAID that can rebuild itself automatically with no external dependencies.

My first hardware RAID was a low-cost OWC box that cost about $400, but I'm not sure if they are still making hardware RAID boxes. ( i.e. they might be software only )

My current RAID is a Promise box that I bought used from a friend, it's a hardware RAID but it uses a dedicated RAID card for data transfer. At some point I will upgrade everything to a Thunderbolt 2 RAID, but that means buying a new computer which is not in the budget right now.

April 1, 2016 at 2:28PM, Edited April 1, 2:29PM

3
Reply
Guy McLoughlin
Video Producer
32889

From our experience, the biggest problem with hardware RAID is the if the box dies, you are likely to lose everything because it is common that the new controller will not work with the old controller's data. On the other hand, one can always build a new system and run the old drives with software RAID.

April 3, 2016 at 12:17AM

20
Reply

RAID is not backup: http://www.petemarovichimages.com/2013/11/24/never-use-a-raid-as-your-ba...

There are a variety of great reasons to use or avoid RAID 0, RAID 5, and/or RAID 6 as external drives for editing. None of them substitute for a proper backup strategy. And none of them guarantee that they will meet your 4K edit requirements until you measure them end-to-end with your actual hardware. An SSD can solve a lot of performance problems that RAIDs don't, provided they have enough storage to hold your clips and provided you have the right high-performance interface slot available on your computer.

But to summarize: RAID fundamentally solves a few problems (performance, resilience), SSD solves one problem (performance), and backup solves many problems (but not performance). You need a comprehensive, multi-layered approach to build a robust, high-performance system that works well over time.

April 3, 2016 at 11:19AM

0
Reply

Your Comment