[Beowulf] High Performance for Large Database

[Laurence Liew]

Hi,

 From what I understand it has to do with "locking" on the SAN devices 
by the GFS drivers.

Yes.. you are right.. most implementations will have separate IO and 
compute nodes... in fact that is the recommended way.... what I had 
meant in my earlier statement was that "I prefer data to be distributed 
amongst nodes - IO nodes - rather than have them centralised in a single 
SAN backend.

GFS + NFS is painful and slow as you have experienced it... hopefully 
RHEL V4 will bring about better performance and new features to address 
HPC by GFS (unlikely but just hoping).

Laurence

Craig Tierney wrote:
> On Mon, 2004-11-15 at 06:26, Laurence Liew wrote:
> 
>>Hi
>>
>>The current version of GFS have a 64 node limit.. something to do with 
>>maximum number of connections thru a SAN switch.
> 
> 
> I would suspect the problem is that GFS doesn't scale past
> 64 nodes.  There is no inherent limitation in Linux on the
> size of a SAN (well, if there is, it is much larger than 64 nodes).
> Other shared filesystems, like StorNEXT and CXFS, are limited 
> to 128 nodes due to scalability reasons.
> 
> 
>>I believe the limit could be removed in RHEL v4.
>>
>>BTW, GFS was built for enterprise and not specifically for HPC... the 
>>use of SAN (all nodes need to be connected to a single SAN storage).. 
>>may be a bottleneck...
>>
>>I would still prefer the model of PVFS1/2 and Lustre where the data is 
>>distributed amongst the compute nodes
> 
> 
> You can do this, but does anyone do it?  I suspect that most
> implementations are setup where the servers are not on the compute
> nodes.  This provides for more consistent performance across
> the cluster.  Also, are you going to install redundant storage in
> all of your compute nodes so that you can build a FS across the
> compute nodes?  Unless the FS is for scratch only, I don't want
> to have to explain to the users why the system keeps losing their data. 
> Even if you use some raid1 or raid5 ATA controllers in a 
> few storage servers, you are going to be able to build a fast and
> fault-tolerant system that just using disk in the compute nodes.
> 
> 
>>I suspect GFS could prove useful however for enterprise clusters say 32 
>>- 128 nodes where the number of IO nodes (GFS nodes with exported NFS) 
>>can be small (less than 8 nodes)... it could work well
> 
> 
> I had some experience with a NFS exported GFS system about 12
> months ago and it wasn't very pleasant.  I could feel the latency
> in the meta-data operations when accessing the front ends of the
> cluster interactively.  It didn't surprise me because other experience
> I have had with shared file-systems have been similar.
> 
> Craig
> 
> 
>>Cheers!
>>Laurence
>>
>>Chris Samuel wrote:
>>
>>>On Wed, 10 Nov 2004 12:08 pm, Laurence Liew wrote:
>>>
>>>
>>>
>>>>You may wish to try GFS (open sourced by Red Hat after buying
>>>>Sistina)... it may give better performance.
>>>
>>>
>>>Anyone here using the GPL'd version of GFS on large clusters ?
>>>
>>>Be really interested to hear how folks find that..
>>>
>>>
>>>
>>>------------------------------------------------------------------------
>>>
>>>_______________________________________________
>>>Beowulf mailing list, Beowulf at beowulf.org
>>>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: laurenceliew.vcf
Type: text/x-vcard
Size: 150 bytes
Desc: not available
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20041116/c78ad27a/attachment.vcf>