[Beowulf] Computation on the head node

[Joe Landman]

Vlad Manea wrote:
> Thanks Perry,
> 
> I would probably go with a separate file server.
> I have seen a lot of NAS on the market recently (i.e. Netgear ReadyNAS 
> NV+ 4Tb X-RAID)
> and I am wandering if they might be a good/bad choice (the price is 
> appealing...).

You get what you pay for ...  c.f. 
http://reviews.cnet.com/external-hard-drives/netgear-readynas-nv-network/4505-3190_7-32578766.html 



> On the other hand I do not want to see all may cluster idle waiting for 
> the file server.
> The w/r rate transfer for these NAS is ~30-40 MB/sec which probably is a 
> modest one.

Its all about the design and implementation.  These units are rate 
limited internally to about 80 MB/s (speed of a single drive).  Now 
share that among 10-20 nodes.

We approach it from a different view.  Start out with a very fast RAID 
and attach good networking to it.  Here is a RAID6 across 12 disks.

This is writing an 84 GB file

root at crunch:~# dd if=/dev/zero of=/big/local.file bs=8M count=10000 
oflag=direct
  10000+0 records in
10000+0 records out
83886080000 bytes (84 GB) copied, 124.942 s, 671 MB/s

This is reading an 84 GB file

root at crunch:~# dd if=/big/local.file of=/dev/null bs=8M count=10000 
iflag=direct
10000+0 records in
10000+0 records out
83886080000 bytes (84 GB) copied, 123.236 s, 681 MB/s

Coupling the network to the RAID so that you can expose most of the 
performance out the net is an issue that the ReadyNAS doesn't seem to 
have addressed.  They are more focused upon bulk storage for small sites.



> Do you have any recommendations for the file server? I think I need lots 
> of RAM and fast
> disks. Since SCSI are expensive, are SATA disks OK? Also, can I take the 
> advantage of multiple
> LAN ports on the file server? (I see the Dell PowerEdge SC1435 can 
> handle 4 NICs for example.)

One of our customers has a unit with 10 NICs in it they are using to 
feed several clusters, and an office.  Another has IB and 6 NICs for a 
similar arrangement.  Others have 10 GbE cards.  If your cluster network 
can handle the heavy networking, I/O is often a rate limiting system for 
performance, so try to get as much network bandwidth in and out as you can.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615