The issue of where you put your stuff

Small camera connected to tape archive

Raw camera formats and 4K video result in mountains of data. Wayne Norton explores the issue of where you put all your stuff.

The relatively short lifetime of digital imaging has been a fascinating time indeed. Digital images have now become the standard - despite the better quality potential of film-based imaging. Digital media handling is certainly more attractive in the short term - footage can be reviewed immediately after shooting and no film processing steps are required. Digital footage can be acquired, ingested into an editing suite and output to BluRay in an end-to-end digital workflow - and all done quickly. The same media can be easily converted into any number of formats, for use from broadcast television to web publication. Plus there are no noxious chemicals being used in a darkroom. It is clear that digital imaging has many advantages. Image quality however has not been one of them.

Demand For Film Quality...

Perhaps spurred on by the growing popularity of firmware-hacked DSLR and Micro 4/3 cameras as tools for filmmakers as well as demand for more color space and resolution, users are driving camera makers toward high bitrate 4:2:2 formats and 4:4:4 RAW image video formats in order to more closely resemble the quality of film - and the market is responding pretty rapidly. Soon enough will RAW images be pouring out of future generations of cameras - and the distinctions between still and video cameras will most likely fade away. Image quality will actually be equal to or better than film. We will have all the fast interchangeable lenses we want and there will be plenty of sliders, dollies, cranes, jibs, and Steadicams to mount our 4K rigs on. New media cards will hold 1T of RAW image data and SSD drives will hold many Terabytes. New CPUs, GPUs and MICs will help accelerate the processing and viewing of the RAW image sequences. LED TV units will handle 3D@ 4K and colorists will become Magi. Amateur footage will look better than reality and YouTube will look like IMAX. So where’s the rub?

Digital Shelf Space

Once you have acquired your footage the fun begins. If you have say, 100G of RAW image data on your in-camera SSD drive (easily obtained in a day’s shooting), you first must get it into your editing computer before you can do anything with it - and it takes time to copy! If you plan to edit your footage you will need to park it on some fast discs so it doesn’t choke your editing software as you scrub the timeline. A multi-disc array (RAID0 is still the fastest) of high-speed SAS or SSD disc drives will provide the performance you need to handle RAW images up-to-speed, but can get expensive quickly - and to do any large project edits you will need at least a few Terabytes of fast disc to work with. RAID0 arrays do not have built-in redundancy or protection however and should not be thought of as reliable long-term storage for your original images! For dependable and longer-term holding, a RAID5 (or better) array with built-in redundancy is safer for your original files. This type of array can be built with slower commodity disks and will be cheaper to grow than the faster disks. It will also keep your files relatively safe. Obviously this will add to the expense - and may need to be expanded over time (if you keep filming!)

So now, to work with our new 2-4K RAW camera footage we will need a hot computer with GPU card, LOTS-O-RAM for the editing suites, a fast disc array for editing and a redundant disc array for intermediate storage - just to get going up to reasonable speed. Throw in some copy time (from disc-to-disc) and a whole bunch of meaningless filenames - possibly from several (camera) sources - and possibly with identical names! Now it’s starting to get interesting...

Where’s The Film Reel?

One not-so-obvious advantage of film over digital is in archiving and storage. Film stores nicely in a can on a shelf for many years and can be labeled for easy identification. Home movies made on 8mm equipment are still around after many years (albeit discolored or otherwise decayed) because there was never a real storage issue. Every consumer could deal with 8mm film without the need to purchase anything more than a basic projector and screen. RAW video will be an entirely different story on all fronts.

Regardless, imaging enthusiasts will spend the money to obtain the equipment they need to deal with the requirements of RAW editing and will adapt. The results will be worth it.

But there is no film can with an original film reel in it to put on a shelf with a label. This may be a bigger issue than is immediately apparent.

The Tape Loop

Similarly to film, magnetic tape never had much of a storage issue either. As long as there is some storage space available you can maintain (and grow) an archive. Advances in digital imaging technology itself spawned a series of tape iterations as a byproduct - from early Beta and VHS on. From a technical or performance perspective, tape was never a real limitation - only a mechanical device prone to damage. In this regard it is great to not have to rely on tape any more, but a somewhat critical component has fallen out of the loop.

There is no longer an archive generated by a tapeless workflow - which means that long-term archiving must happen somewhere else. No longer will there be a tape copy with original footage that can be stashed for safe-keeping - and no shelving scheme or ledger to help keep track of the footage over time.

But we wanted to get rid of tape in the first place! Why would this be a problem?

Sheer Weight Of Numbers

Consider that even a modest 100G per day of RAW image acquisition will yield about 1/2T per 5 day week of data - in a year you will have over 26T of original RAW image data to manage and store! If you are working with multiple cameras on multiple projects the numbers will grow accordingly. How will you store and track those images indefinitely? Once they are deleted from hard disc, they are gone forever!

Computer backup tape units may still be one of the more reasonable solutions. At about $50 US per LTO-5 tape (for example - capable of up to 3T compressed), you could cover your archive needs in the example above for less than $500 per year - plus the investment in the tape drive and interface. This can eliminate the need for a large amount of RAID5 disc space. You will also need tape-capable backup software as it is no longer included as a base operating system component (with the exception of Linux / Unix systems). Despite any initial expense and handling however, compared to disc storage, tape is much easier (and less expensive!) to grow - and to store over time. Simply stated, without a tape copy from the camera, if you want to hang on to your original video files you will need some reliable and expandable storage to do it with.

Perhaps the best storage systems for production use are large NAS or SAN based systems - fully redundant with high-speed interconnects, hot spare disks and intelligent disc controllers as used by large corporations, but can cost well into the millions of dollars. Notwithstanding, even large corporations with huge NAS or SAN storage infrastructure may still use tape as the last tier of long-term storage as it is by far the most economical.

Unfortunately tape has pretty much dropped out of the consumer market - and has also dropped out of the camera (with exception of many pro video cameras). Even tape drives for home use are all-but-vanished and most manufacturers are touting tapeless backup systems. Backup software is now mostly geared towards Internet and remote backups - but pushing 100G of image data across your DSL connection may prove less-than-timely! And what about restoring that data when you need it? I believe that this will be one of the greatest challenges for data of all types in the very near future - where do we park it all - and how do we get at it when we need it?

Management Of Originals

To keep with the analogy of film, the next rub is in media management. With film reels in metal cans you can add labels and a shelving scheme to help keep track of your footage over many years. The addition of a ledger makes the archiving of film academic. This is not the case with digital media. There is no simple method of keeping track of a myriad of files - and in the case of video files, filenames may not remain unique throughout acquisitions. Directory naming structures may help, but it is difficult to create meaningful names for each and every directory that will make sense when referred to in a year or two. Even within an editing project the lack of unique filenames can cause considerable confusion - and multiple cameras will only add to the disarray.

It’s All In A Name

A good first step in establishing a management solution would be to give your video filenames some meaning. By adding a unique prepended identifier and file creation timestamp to the filename you can more readily keep up with project, scene, camera, take, and other information that along with the timestamp will make it a lot easier to locate your originals. Add a ledger (database, notepad, pen and paper?) to keep better entry details and you have a rudimentary management system. For larger projects and professional applications there are video management services and systems available that are designed to the task and should have everything you need to help keep track of your footage over time.

Odd Turn Of Events

Editing and compositing RAW video footage will require a sizeable expense in computer equipment and disc drives. Long-term storage will be expensive and hard-to-manage without new solutions - or better connectivity to the home / business so that large datasets can be transferred to remote locations within reasonable timeframes. Support equipment (video monitors, waveform and vectorscopes, etc.) for 2K to 4K will initially be quite expensive and for normal HD, output will need to be resized to 1080P or 720P for BluRay distribution or broadcast. RAW in-camera storage media will also be expensive and transfer times to an editing computer will be a factor.

In a way the digital imaging age has spawned an odd turn of events. We have progressed from sending film out to the lab for chemical processing to processing the footage ourselves with our computers. Likewise, film editing can now be done in computer rather than by special movie editing systems and actual film splicing and handling - not to mention special effects and compositing - previously impossible for the budget-challenged filmmaker. In this respect the power is now in the hands of the camera / computer owner - as is the burden of expense! You may not need to outfit a darkroom, but you will definitely need to outfit your computer. After all, the computer has become the darkroom!

Certainly one of the biggest advantages of digital imaging is in saving time - it effectively eliminates some very time-consuming events from your workflow - film processing and splicing...and as I’m sure you have heard, time is money. In this regard, digital imaging is here to stay - and the long-awaited quality of RAW footage will be well-received and appreciated.

Now all we need are affordable new ways to store our digital films and keep up with them over the years. Personally, I would push for a (high-quality but tiny) backup tape internal to future RAW imaging cameras - I still have plenty of shelf space...