[Beowulf] Can one Infiniband net support MPI and a parallel filesystem?

[Gus Correa]

Hello Mark and list

The measurement was based on walltime.
It just refers to the user occupancy of the cluster, versus what was 
left idle
(for all reasons, e.g. lack of resources to serve large queued jobs, 
lack of enough jobs to fill all nodes, etc).
The number is simply the utilized resources divided by the available 
resources.
This gives a coarse measure of machine utilization.
Take the walltime of all jobs multiplied by the number of nodes (or 
CPUs) each job used,
sum them,
and divide by the duration of this period (say, one year) times the 
number of nodes (or CPUs) in the cluster.

Maybe 70% utilization is low compared to airplane seats, subway 
occupancy, hotel rooms, restaurant tables,
Internet, telephone networks, and perhaps to other clusters.
I don't know, I am not an operations research person.
The only other number I could find for a (well used) large cluster in 
our science field was below 70%,
and now Chris mentioned 77%.

Are there published numbers of resource utilization for other machines,
say, public clusters in the US, Canada, Europe, world?

Yes, our cluster is dedicated to a small group of earth scientists and 
students (20-40 users)
and it is small (32 nodes, 64 cpus). 
Cluster size and user population size most likely make a difference,
but in any case, I would be interested in seeing any other numbers
for any kind of cluster.

Regards,
Gus Correa

-- 
---------------------------------------------------------------------
Gustavo J. Ponce Correa, PhD - Email: gus at ldeo.columbia.edu
Lamont-Doherty Earth Observatory - Columbia University
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------


Mark Hahn wrote:

>>> It appears we've averaged almost 77% utilisation
>>> since the beginning of 2004 (when our current usage
>>> system records begin).
>>>
>> Thank you very much for the data point!
>>
>> I've insisted here that above 70% utilization is very good,
>> given the random nature of demand and jobs on queues in the academia, 
>> etc.
>
>
> that sounds very strange to me.  do you really mean that 30% of your 
> cpu time is idle?  I wonder whether there could be a big
> difference in methodology.  for instance, if you're using an MPI library
> (probably based on tcp) that doesn't spin-wait but blocks as for disk IO
> say 20% of the time, then you might consider this to be 80% utilization.
> an MPI that spin-waits might show 100% with the same perf/throughput.
>
> 70% utilization is terrible if you really mean "fraction of 
> allocatable cpu
> time occupied by jobs".  that is at the job scheduler level, not at 
> the kernel scheduler level.
>
>> However, some folks would want more than 90% efficiency to get happy.
>
>
> I would be embarassed to have less than 90%.  perhaps 70% would make 
> sense
> for a cluster dedicated to a small or narrowly-defined group.  I find 
> that a sufficient userbase means you _always_ have something to run, 
> of any size/resource available.