[Beowulf] Big storage

[Robert G. Brown]

On Thu, 30 Aug 2007, Jakob Oestergaard wrote:

> I find it interesting (and surprising) how little people like tape :)

It isn't that difficult to understand.  People WOULD like tape and USED
to like tape back when a single tape on a single tape drive would back
up your whole system.  I even remember those days.  But what, maybe
seven or eight years ago the hard disk curve (which has an even shorter
capacity doubling time than Moore's Law does for the rest of the system)
crossed over the tape curve (which has a MUCH longer one) and life has
sucked ever since.  Tape backups even back when they still "worked",
kind of, had gotten madly expensive compared to the disk they were
backing up -- you could pay $1000 for disk and $3000 for the tape backup
unit really easily.  Then suddenly you needed an entire tape library and
were looking at $5000 in hardware plus a pile of $100 tapes (that had to
be replaced once a year or thereabouts) to back up your $1000 worth of
disk.  At the same time, the bandwidth issue has steadily grown -- if
you're backing up (say) 20 TB to a $7000 tape library holding $2400
worth of tapes over a 1 Gbps channel, that's 8 seconds per GB
(idealized) times 20,000 GB, or 160,000 seconds -- about two >>days<< to
do a full backup at 86,400 seconds per day.  And of course if people are
using the disks being backed during that time one has the usual problems
with possible inconsistency.

So we're still in that pickle -- tape libraries with TB capacity are
expensive, bottlenecked, finicky (bad tapes happen, often silently until
you need them for a restore), and did I mention expensive?

And necessary, alas.  Even maintaining a full mirror of RAID 5 systems
for disk-to-disk backup doesn't give one the warm fuzzies that a good
old tape library with period full backups archived and stored in a
fireproof box offsite.  The only alternative that comes close is to rent
space on an offsite RAID in a for-rent server farm (lots of them out
there these days) and pump a periodic dump out to it from your primary
servers, but that typically only works for relatively small sets of
critical data -- things USB backpack disks can probably manage just as
well -- or you run into the significantly greater network bottleneck of
Internet WANs.

So for backing up my home md RAID (which has a downed disk and is about
to be rebuilt with twice the capacity intead of adding back another 80
GB drive) I use a cheap 160 GB backpack disk, ditto for my laptop
(although I only back up data from either one as distros are infinitely
reinstallable).  For professional/work backup, a tape library is the
ONLY way to go, however painful and expensive it might be.  In fact, the
current capacity of our tape backup system is the size limiting factor
to our provision of disk -- adding disk is cheap cheap cheap as I can
build a perfectly usable 2 TB RAID 5 (5 500 GB disks) out of OTC parts
for what, $1500 these days?  Or put it into a commercial hot-swap
chassis and gussie it up as a pro-grade RAID that is even larger for
maybe twice that.  RAID 5 is less than $1/GB out to the 2 TB level, in
other words.  But tape drives for standalone systems that might back
such a RAID up are pretty much nonexistant these days at the consumer
level.  It's been years since I saw a PC with a built in tape, because
the tapes one could build in cost more than the PC.

I don't know, ultimately, if this is really a good or a bad thing.  I've
lived through many changes in tape format at this point, and seen
firsthand that tapes actually do suck as a long term archival storage
mechanism.  I've got an unreadable QIC tape from IBM 5100's in my office
(with my archaic copy of APL mastermind on it:-) and an ALMOST
unreadable 9 track reel tape with some ancient fortran for quantum
chemistry on it, just for luck.  I used to have 5.25" floppies around
until a few years ago when I finally sighed and pitched them.  Tape is
professionally necessary, but the reality of long term data storage is a
continuous process of revolving upgrade and migration, generally disk to
disk, facilitated by the ever growing capacity of those disks.  Data on
the shelf, any shelf, will almost certainly be unrecoverable on a
decadal timescale between information degradation on most media (those
pesky very high energy cosmic rays with their showers of secondary
particles on an in-medium collision, thermal degradation, dust, coffee,
tape bleed, etc) and the disappearance of players for the obsolete media
-- does ANYBODY on this list still have a floppy drive capable of
reading 360 KB floppies?  How about 9-track reel tapes?

I think we're seeing CD-ROM entering its end-stage right now.  DVD
drives are replacing CDs across the board, fortunately backwards read
compatible -- so far.  Music CDs seem to be about the only thing
sustaining the format's existence.  As flash continues to drop in price
and its capacity continues to grow, though, I'm expecting to see OTC
music migrate.  We're already at the stage where if somebody was really
clever they could build a nifty solid-state flash backup unit (or active
disk) of 50+ GB capacity for maybe $1000 dollars, as earlier discussion
indicated.  In a year or two that will be doubled and halved, and PCs
may actually start coming with a standard 10-50 GB flash as its basic
OS disk, add on hard disk strictly optional.

So tapes ARE necessary, but sure, nobody likes them.  Who would?  What
we WANT is a 20 TB capacity uber-reliable radiation hard redundant box
the size of a standard 5.25" drive enclosure that stores the data on a
removable unit the size of a zip drive from the old days, that costs
$200 or so OTC from any of twenty vendors (media costing $10 a unit) and
with a 10 Gbps out-of-band channel between it and our hard disk array
with complete file locking support and hot and cold running software to
automate real-time, reliable backups with.  :-)

What we've got is tape.:-p

    rgb

>
>> But something that you have to be prepared for when going to that
>> storage volume is that you *will* suffer data corruption at some
>> point, and you need to plan for it. See for example
>>
>>   http://cern.ch/Peter.Kelemen/talk/2007/kelemen-2007-C5-Silent_Corruptions.pdf
>>
>
> Thanks!
>
>> It's quite possible (though unlikely) for a hard disk to suddenly
>> return corrupted data without signalling a read error, and this is a
>> possibility that raid controllers typically just ignores. And then you
>> have the usual crop of software, firmware and hardware errors that can
>> trash your data more or less silently.
>
> It seems ZFS end-to-end checksums is not such a bad idea after all :)
>
>> At a minimum, make sure you keep checksums of all files so you can
>> verify their integrity.
>
> Thank you for the feedback!
>
>

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu