[Beowulf] RAID for home beowulf

[Tomislav Maric]

Mark Hahn wrote:
>>>> So, maybe the bold question to ask would be: what would be the best RAID
>>>> config for 3 HDDS and a max 6 node HPC cluster? Should I just use RAID 1
>>> do you mean for each node?
>> No, the nodes are diskless. I plan to scale the cluster and 1TB of
>> storage is quite enough, even if I use 6 nodes, or 2x6 nodes. That's
>> actually what I know from my small experience in running CFD codes on 96
>> cores cluster. That's the reason for thinking about RAID in the first
>> place: create stable and good performing centralized storage for the
>> future number of the nodes (i.e. 12 nodes with 4 cores and 16 GB of RAM
>> each).
> 
> perhaps I'm confusing multiple threads, but didn't you say earlier
> that your workload was disk-intensive?  from that I'd assume you'd 
> want the speedup you'd get from having each node do IO to local disks.
> 

No, I've said that I have centralized storage and I want to make it fast
and avaliable. :) And I've said that this is the first time I'm doing
this and that didn't benchmark it yet, it's a two node machnine, that
will be scaled after benchmarking if it shows promise. :)

>>>> for the system partitions on one disk,  and RAID 0 for the simulation
>>>> data placed on the same partitions on other two disks: after
>>>> post-processing, the data is gone anyway... and with a good backup
>>>> strategy, I don't have to worry about RAID0 not recovering from a disk
>>>> fail...
>>> you're going to back up a raid0?
>>> From your question, I sense it's a bad idea... :) I have no clue, this
>> is the first time I'm doing this.
> 
> well, using raid0 is a declaration that the files are purely transient
> (have no meaning or value when the job or job-step is over.)  so backing 
> them up is somewhat strange.  or from the other direction: backing up 
> means that the files are valuable, and you don't want to use raid0 for 
> that.  (unless, I suppose, the files are read-only...)
> 

Thank you for the info, I only know about RAID what I was able to read
from the short and out of date how-tos. I'm only thinking about it
through the information I've got so far: RAID0 for speed, RAID1 for
mirroring, RAID5 a hybrid that causes much discussion. That's it, so
far. What does it mean that they don't have any value? They are at the
disk and I'm the sole user for now, and after I've done with them, I can
dispose of them and leave the processed simulation data on a safe place.
I admit I'm getting lost here.

>>> in any case, I think you should consider net-booting and using the node
>>> disks as a 3x raid0.  if the local files are really transient, then
>>> your startup script can just reinitialize the local disks every boot.
>>> (which would leave you with a working node even after a disk failure or two!)
>>> that's assuming you need or can benefit from the capacity or bandwidth.
>>>
>> OK, I want a net boot because the nodes are diskless, the remaining
>> question is how to use 3 HDDs with RAID to get a performance boost where
> 
> since you're talking about IO over the net, which is gigabit, right,
> it hardly matters.  gigabit is 100 MB/s at best, and that's less than 
> one disk's worth of throughput.  3-disk raid10 will give you 50% space
> efficiency but fast writes; raid5 will give you 66% space but relatively
> slow writes.

OK, that's correct. I guess I'll have to see how the writing is going to
take place when a bunch of nodes (maybe 6 of them) send a lot of data
(even over GigEth) to the master node. I was thinking ahead, with the
RAID or whatever other option for expanding the storage, making it fast
(parallel file system?) and reliable. I guess now that you've explained
it, I'll have no problems with a few nodes, since the eth is the
bottleneck, as it will probably always be. I just wanted to make sure
that the nodes don't wait for the master to write down the data and that
my OS and other installation is safe from disk failure. That's the
strategy, but obviously my knowledge can't follow it at all. :)
> 
>> I need it (like the /home where the data is written) and HA for the /
>> dir., in case of disk fail. Is this the right way of thinking?
> 
> you should always strive for simpler systems.  don't introduce differences
> in config unless they're clearly necessary.  don't partition disks unless 
> necessary.
> 
> 

How do you mean differences in config? I'm configuring the master, and
the other nodes are to be diskless.. I have separated these partitions:
  /swap /boot /  /var and /home. Is this ok?

Thank you very much for the advice.

Best regards,
Tomislav

>>>>> or better yet, don't bother booting of the local disk.  simply make your
>>>>> head/admin/master server reliable and net-boot.  it's likley that nodes
>>>>> won't be functional without the master server anyway, and net-booting
>>>>> doesn't mean you can't use the local disk for swap/scratch/...
>>>> Well, I want to configure the net boot for all diskless nodes and use
>>>> the master node and it's RAID for a performance gains with writing CFD
>>>> simulation data against network communication and to be able to scale
>>>> more easily.
>>> I'm not sure I parse that.  net booting is orthogonal to whether or not
>>> you store data locally or over the net.  but yes, gigabit is somewhat
>>> slower than a single modern disk, so local IO will win.
>>>
>> Thanks.
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>>
>