[Beowulf] Need recommendation for a new 512 core Linux cluster

[Steven Truong]

Hi, all.  I would like to know for that many cores, what kind of file
system should we go with?  Currently we have a couple of clusters with
around 100 cores and NFS seems to be ok but not great.  We definitely
need to put in place a parallel file system for this new cluster and I
do not know which one I should go with?  Lustre, GFS, PVFS2 or what
else?  Could you share your experiences regarding this aspect?

I also would like to know how many head nodes should I need to manage
jobs and queues.  And what else should I have to worry about?

Thank you very much for sharing any experiences.