[Beowulf] distributed file storage solution?

[Mark Hahn]

>> have you had success with this approach? 
>
>Absolutely it does.  They promise they can do something better than anyone 
>else can, let them prove it.

what I'm skeptical about is the cost of doing such a demo.

>> I'm pretty sure vendors would have 
>> laughed at us if we asked for it, though for something small (few TB, 
>> <100 clients, etc), I'm sure it would work.  then again, anyone could do 
>> that sort of small project on a weekend.  if you have hundreds of clients,
>> expensive (non-Gb) interconnect, multiple racks, power,cooling,floor needs,
>> I don't think you'll get a lot of people offering to do full demos for you.
>
>Why would they laugh?  We manage our cluster efficiently and get a lot of real
>work done.  If you tell me you can improve that (otherwise why change anything),
>shouldn't you be willing to put your money where your mouth is?

how much money?  my local SFS cluster is 70 TB, not really that large,
and consumes 3.5 racks.  (storage doesn't consume much power, so the 
racks are probably <10KW total, but still require dual 20x220 plugs for 
each rack.

I'd guess that a vendor would have to spend perhaps $20K to get such 
a cluster in place for demo purposes, and that's ignoring the interconnect
entirely (which is quadrics in this case, so non-negligable in price ;)

>When I tell my masters we should go with something, I make sure I have all the
>facts stacked in my favor so there are no surprises.  When my masters fail to listen
>and impulse buy I wind up with a file system with infiniband (which has been talked 
>about here) that a year later when I asked my co-sysadmin here if it was  finally 
>running, he smiles and answers, "That depends on how what you mean by running...

interesting; I'd enjoy hearing more about it.  obviously, IB _is_ capable of 
working, so what appear to be the sticking points?

>>in short, if you do this, aren't you effectively pre-selecting high-margin vendors?
>
>Who cares what the margins are?  If they have something that will increase the
>ability of our cluster to complete more jobs in a shorter amount of time-is that not
>why we are here?

if my 70TB costs $100/GB, it's quite a different proposition than a
lower-margin solution that costs $1/GB.  even though the latter might 
be a better solution, such a vendor would obviously never be able to 
afford to demo it the way you describe.

I think you mentioned EMC in an earlier message, and I suspect that pretty
much equates to what I'd call high-margin.  this is NOT to criticize your 
decision or EMC, just that it's probably a lot more than $1/GB.

in fact, I'd love to hear your comments about EMC as well.

FWIW,
the 70 TB cluster I have here is similar to three others we have at our 
4 largest sites.  36 shelves of 11x 250G SATA, connected by dual-U320
to 12 servers on the cluster's interconnect (along with metadata-pair).
performance is generally good, and I'm not sure whether we have any real
issues with metadata slowness (this is lustre/sfs.).