[Beowulf] Using beowulf to unify or consolidate storage

[Geoff Jacobs]

Lux, Jim (337C) wrote:
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Didier Caamano
> Sent: Wednesday, May 05, 2010 11:44 AM
> To: beowulf at beowulf.org
> Subject: [Beowulf] Using beowulf to unify or consolidate storage
> 
> Hello to everyone,  
> 
> I apologize if this email is out of place, but I have the following personal project and I have been searching over the internet to try to find the best possible solution.  We are not a big company and I am trying to implement some sort of SAN or NAS to consolidate my storage.  I've been reading through Beowulf Book, I have to admit I am just in the first pages, but before I continue on reading I wanted to ask the question so not to waste my time reading in case it is not possible.
> 
> I have a whole bunch of PCs that are not longer in use and are just collecting dust, I'm trying to (in case it is possible) to somehow put them all to work as a single unit and use their hard drives or add more hard drives, to create a Storage Area Network.  Is it possible, using Beowulf to achieve this goal.  Are there any recommendations as to where to start with this?  I am eager to learn new things, I have experience using BSD and GNU/Linux, I just want to know which direction to go in order to achieve my goal.
> 
> Thanks and have a good day.

New HDDs would be optimal, but it's possible to use older equipment for
clustered storage, especially if it's good quality or if you're just
testing the software. You could take a hybrid approach. For example, if
you have older server or workstation boards, you could equip them with a
few new "dumb" 4 port SATA cards (PCIe, PCI-X and PCI in order of
preference), 4X SATA drives per card, Intel GigE adaptors, your UNIX of
choice and use any one of a number of SAN or cluster filesystem schemes
(Lustre, Gluster, GFS, XFS over iSCSI, etc) to connect it all together.
You can go use port multipliers if you want to trade speed for size, but
I doubt is worth considering in terms of price. Do some reading on
different SAN configurations and try to think of what would be most
appropriate for what you're doing.

It's going to be slow compared to more modern, purpose built hardware,
and you will have to devote a lot of time to testing and tweaking until
you have confidence in it's performance and reliability. On the other
hand, you'll probably learn a lot regarding the technology.

One thing you must not do when going down this path is rush the hardware
into production. You're undertaking all the risk if your project doesn't
pan out, so don't blow too much economic and political capital. Treat it
as an R&D project, not as production, and assume that at some point you
will move to newer hardware, an outside SAN/NAS solution, or something
else as needs change. Size and spend accordingly.

If you need production hardware as of yesterday, there are people on
this list from industry (including reps from Penguin Computing, our
gracious hosts) who would be happy to discuss options with you.

-- 
Geoffrey D. Jacobs