diskless nodes? (was Re: Xbox clusters?)

[W Bauske]

Velocet wrote:
> 
> On Fri, Dec 07, 2001 at 09:47:10AM -0500, Troy Baer's all...
> 
> > Well, there's nothing keeping you from keeping the root filesystem on NFS
> > and using local disk for swap, /tmp, and /var.  We do that on our Pentium
> > III and Itanium clusters, and it seems to work pretty well.  The biggest
> > problem is user education ("No, 'cp file /tmp' does not copy it out to all
> > the nodes in your job, use pbsdcp instead").
> 
> With large scratch disk bandwidth required I'd definitely go this way.
> But even with $100-$150 a node on $300 diskless nodes, thats 35-50% the
> cost of a node. If the parameters of the requirements for the
> cluster's disk usage dont dictate more than a few Mbps of scratch
> bandwidth being required, then you can get a much higher price
> performance out of diskless machines. Of course this reduces
> flexibility of the cluster, but if you can ensure its only used for
> certain types of jobs that fit into a range of parameters, then you
> can get a fair increase in performance for the money.
> 

Something is missing in your analysis.

Keep in mind you spend the money on disk you need, whether you put it
on a server or put it local. The difference is only if you can't buy
the correct size disk to match your applications. If you can only buy
much larger disks, then maybe putting it on a server makes sense so
you can share it among several nodes. I don't know your scratch 
comsumption per node so I can't say if you saved money or not.

I do know my scratch disk consumption per node. And I really have
a use for that local disk. Even if I had put it on the server, I still 
have to pay for it so I don't lose much by connecting it locally and I 
gain I/O bandwidth without the restriction of running across a network.

Then again, I have an app that would run very well on a diskless 
cluster but because of that other app, I need lots of disk space.

Local node install puts the bandwidth where that particular app needs it.


Wes