September 1, 2011

Some Good Tips for Data Wrangling On a Budget

Evan Luzi, who runs the camera assistant blog The Black and Blue (and who also wrote a guest post here on NFS), takes a look at the crucial but unheralded position of data wrangler. The data wrangler is the person on set who is responsible for making sure that what gets recorded on camera makes it back to the editor without any data loss or corruption. This is not an action-packed video, but if you might find yourself doing work in this position it's a nice and informative behind-the-scenes commentary. There, I warned you: I don't want to hear any comments of "this was boring!" Hmm, that wasn't a very good front page teaser. Well, check it out anyway:

Oh, and Evan doesn't typically work in this capacity, but the right job came along so he took it. In my own case, I've never been hired as a data wrangler, but there was one Sony SRW-9000 shoot on which I was supposed to be the Camera A operator -- but then the DP decided he wanted to operate and, not wanting to displace the B or C camera guys, I hopped over to the vacant position of data wrangler (someone had forgot to hire one -- oops). Here's Evan's take on the gig:

If you think this was boring, keep in mind we're all here to learn. Right? If not, I'll have Evan throw a sword:

Also check out Julio's notes in the comments, which are an excellent summation of Evan's tips.

Personally, one of the ways I'm planning on saving some money on the data wrangling for Man-child is by putting to use the new Thunderbolt interface. It should speed up transfers and remove the need for complex cabling (and a huge cart). Of course, I bought a laptop right before the interface started appearing on Mac laptops. Ah well.

From a comment on the video (the first one, I mean), I found out about a $99 data wrangler app called ShotPut Pro. There's also a Windows version; both offer wranglers and wranglettes the ability to "verify file-by-file sizes, MD5 or CRC checksums, or byte-to-byte comparisons of your media copies to ensure they exactly match the originals." When you want to be absolutely certain that you can format your camera's shooting media without losing anything, that sounds like a good thing. Anyone have any experience with ShotPut Pro?

[via Wide Open Camera]

Your Comment

30 Comments

For FREE, I love Terra Copy for Windows. It can replace the typical Windows copy command to include CRC checksums. http://www.codesector.com/teracopy.php

September 1, 2011

0
Reply

That was boring!! : D

September 1, 2011

-2
Reply
RevBenjamin

Watch out! I'll throw my sword at you! ;-)

September 1, 2011

-2
Reply

You know what's exciting though? Losing all of your dailies!

September 1, 2011

-2
Reply
Nathan

Man, what a rush!!! There's nothing like that sinking feeling in your stomach!

September 1, 2011

-2
Reply
Ryan Hall

5 thumbs up.

July 30, 2012

0
Reply
Drew Lahat

Koo -- thanks for the write up! Was pleasantly surprised to find myself on NoFilmSchool this morning :-) It's true the job is pretty boring -- and that's exactly why I filled my time by shooting the video!

To give you an idea of just how action packed it can be: the day before I recorded this video, I was hired to do data wrangling for a National Guard instructional shoot being shot on a couple DSLRs. So I showed up, got the hard drives ready, got the computers setup, and went over my process with the DP and cam ops.

Then I sat there for 4 1/2 hours watching them shoot. My first card dump didn't come until we broke for lunch!

With higher data rate cameras like RED, the pace moves a bit faster, but it's typical when data loading to spend a lot of time "chillin"

Of course the trade off is all the thousands of dollars they're spending on the production is dependent on you doing your job correctly -- they're paying you to shoulder that responsibility.

Thanks again Koo! Let me know if you have any workflow questions for Man-child

September 1, 2011

0
Reply

Not boring at all. The more you learn the better prepared you will be. Great vid!

September 1, 2011

0
Reply
Doug

Some good tips. The stress of losing data is extreme - very bad, don't let it happen to you out there... be thorough and go through checklists. Develop good habits.

It was cool of you to share your knowledge.

September 1, 2011

0
Reply
Julio

NOTES

http://nofilmschool.com/2011/09/data-wrangling/

ExpressCard34 for CompactFlash and SDHC cards

2 external drives (FW800)

Always have 2 copies of everything before you willingly erase or move data.

2 rolls of camera tape (red and green)
Wrap cards that are OK to format in green tape, wrap enough tape that the cameraman has to unwrap it to use it again.

UPS power supply for power interrupts.

=================

Folder Structure
DAY 1 - date
CamA
RawA001 Folder (sequential num continues uninterrupted over days) <<<<dump absolutely everything on the card into this folder.
RawA002 Folder (2nd dump of the day)
CamB
CamC
Sound
SND001 Folder (1st dump of the day)
PLEASE READ ME - LOG NOTES.txt

DAY 2
DAY 3

==================

Check total data size on CF or SD card against the transferred files on hard drives (number of GBs). See that they're equal.

5 spot checks (per 100 files).
Watch beginning, scrub through and watch end.

Don't bow to pressure to turn over cards to be formatted too quickly before checking data.

Tell the person to which you're handing the card to be formatted that this card (w green tape) is OK to format. Verbal confirmation. Don't be ambiguous about it ("this is OK"). Tell them explicitly: "This is OK to format".

September 1, 2011

-1
Reply
Julio

Nicely done Julio, very attentive class! Adding a link to the post.

September 1, 2011

1
Reply
avatar
Ryan Koo
Founder
Writer/Director

I always use Toast Titanium to checksums. There is a little app that comes with Toast and it´s hidden in the utilities. It`s call Compare :)

September 1, 2011

-1
Reply

Absolutely great video Evan. I do some DAM for my job at a major outdoor company and this is totally legit, comprehensive information. Nicely done!

September 1, 2011

-1
Reply
Nate

I myself am a very different species of data wrangler... a VFX data wrangler to be exact, so I was very interested to see another job being done under the same name, presented in a very easy to understand and very "listenable/watchable" way. Great post!

September 2, 2011

1
Reply

For your Feature Film you might want to check out Silverstack. http://pomfort.com/silverstack/setedition.html
It handles RED, Alexa and H.264 files and looks a little more professional than ShotPut Pro.

September 3, 2011

-1
Reply
Crazy_King

I played with Silverstack, and it has one major shortcoming. It works with ONLY those formats. You can't import XDCAM, AVCHD, or any other format =\

September 3, 2011

0
Reply

I have used shotput pro a on a few productions, worked well and was simple to use. I've also used the RED software on RED shots. Never used anything else while working with the RED.

September 4, 2011

0
Reply
Nic

thanks for the info, not boring at all..
It's not everyday that you get told how to dump properly ;)

September 5, 2011

0
Reply

Well done, this real world stuff is always helpful. Just curious what kind of UPS you use for that setup?

September 5, 2011

-2
Reply
sjc

Thanks very much for sharing your process. Well done lad ! Like the green tape/ red tape idea.
I use ShotputPro verification software to confirm bit by bit confirmation of transfer of all files, works with most data card formats, including Red One.

September 7, 2011

1
Reply
Mike M

Hi there,

As a data wrangler/manager/DIT myself this was not a bad overview of data managing on set. I would NEVER trust a Finder copy, use the ditto command in Mac or use Shotput Pro for Mac or PC.

The file naming scheme is also a bit messy.

September 13, 2011

0
Reply

To add to this, I do all my QC with broadcast spec monitors and make triplicate backups and transcode dailies on set to be ready for ingest by the assistant editor. If I have time I will write a blog on what I consider the most robust way of managing data.

September 13, 2011

0
Reply

Hey Christopher,

My goal in this video was to show a simple way to do a basic data transfer. I've never lost any footage through Finder, I'm proud to say.

With that said, how would you change the file naming scheme? Am curious on your perspective.

September 14, 2011

1
Reply

I agree with the others, it's a little more robust to have a file copy program that does a checksum after copying.

I detest the name "data wrangler", but I guess in LA we've standardized it to DIT... irks me for some odd reason. With that in mind, and this video didn't seem to mention, is that as DIT your job is not only to make data safe and sound but you're also bridging the workflows of the camera department, director/scripty, and the editor; this may mean tagging clips with metadata from the scripty or, in the case of special effects, notes from the 2nd AC on camera stats.

I would also suggest keeping logs in paper form instead of a text file. Why? You can tape the logs to the outside of the hard drive boxes, or you can tape a plastic sleeve (half page size, fold your log) to the hard drive itself so the editor knows what is on the drive without plugging it in. The last feature I worked on used 8 of the 2tb g-drives pictured here.... I can't think of an editor who wants to plug in 8 drives to find one scene.

September 15, 2011

-1
Reply
Angelo

Hi Angelo,

I feel like I should clarify some things about the video.

First, I understand why you don't like the name data wrangler, but I was in no way doing DIT work. I wasn't doing anything except literally backing up data. So to say I was the DIT on the shoot would be misleading since I had no hand in doing looks or anything to do with post. That's why I didn't mention anything about that -- it was out of the scope of what I was hired to do.

Perhaps we can say I was the data loader? Memory Card Dude? Kid with the Computer? :-P

Also, that's a good idea about the log sheets. In the video, what I was referring to more was little notes about certain clips or files. If you only have one note about a specific clip, it would seem unnecessary to tape it to the HD or the box.

Certainly paper log sheets make sense if you are keeping notes for every single shot, but, as I mentioned above, I was really only backing stuff up.

DIT work is a far more robust job with a lot more considerations -- as you have rightly brought forth -- than simply transferring and making sure the footage is there.

Hope this helps clear some things up!

- Evan

September 22, 2011

1
Reply

i WANT TO ASK YOU ABOUT THE TERM " DIT"? I WANT TO KNOW THE SPECIFICS FUNCTIONS ABOUT A DIT PERSON? THANKS FOR THE ANSWERS

PAZZO
MEDIA MANAGER
POST PRODUCER
AND EDITOR
AND SMILEY GUY

March 3, 2012

-1
Reply

This is fab and just what I was looking for.

I have an interview tomorrow at a place where, if successful, will involve alot of traveling with a great company.

Thank you, much appreciated

June 13, 2012

-3
Reply
J

just wanted to add a basic rule of mine: NEVER HAVE DRINKS ON THE SAME TABLE AS YOUR GEAR!!! NEVER! NEVER!! NEVER!!! ;) you just never know who will bump into the table spilling liquid all over your gear, killing everything.. I actually experienced this! Good video!

January 23, 2014

-1
Reply
water dude

Is shotput Pro still the preferred application for offloading in CF cards in 2014? Im on windows so I cant use Silverstack. I could use Prelude or Bulletproof, but Im mainly working with Magic Lantern MLV files and I dont believe either sees those. Any advice?

June 3, 2014

1
Reply

Just transfer whole file structure with Teracopy and enable Test Checksum in the program.

July 1, 2014

1
Reply
Robert