[Beowulf] Big storage

[Greg Lindahl]

On Tue, Apr 15, 2008 at 06:02:51PM -0500, Bruce Allen wrote:

> So far we have not seen any signs of data corruption on Linux+Areca 
> systems (and our data files carry both internal and external checksums, so 
> we would be sensitive to this).

Check out these papers from Fast08, they're based on NetApp data for
more than a million disks:

http://www.usenix.org/event/fast08/tech/full_papers/krioukov/krioukov.pdf 
http://www.usenix.org/event/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf
http://www.usenix.org/event/fast08/tech/full_papers/jiang/jiang.pdf 

And, finally, this one makes you wonder how Linux storage works at all:

http://www.usenix.org/event/fast08/tech/full_papers/gunawi/gunawi.pdf

Internal checksums for small files can be defeated for lost writes. I
take it that your external checksums are stored in a totally different
place? In which case you ought to be pretty well covered.

-- greg