[Beowulf] SC13 wrapup, please post your own

[Prentice Bisbal]

On 11/25/2013 07:11 PM, Adam DeConinck wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
>> 4. I went to a BoF on ROI on HPC investment. All the presentations in
>> the BoF frustrated me. Not because they were poorly done, but because
>> they tried to measure the value of a cluster by number of papers
>> published that used that HPC resource. I think that's a crappy, crappy
>> metric, but haven't been able to come up with a better one myself yet. I
>> was very vocal with my comments and criticisms of the presentations, so
>> if any of the presenters are reading this now, I apologize for
>> hi-jacking your BoF. Getting good ROI on a cluster is close to my heart,
>> but is also difficult to quantify and measure. I hope I can be part of
>> the discussion next year.
>>
> Do you have any thoughts you can share on what alternative metrics
> might look like, even if you can't think of one that's clearly better?

Unfortunately, I have no better suggestions at this time. The best 
metric will vary depending on what your priorities and business goals 
are. I've had some discussions on different metrics for different goals, 
but nothing fruitful/concrete has come out of them. Too nebulous.

Many people just want to know if their cluster is being fully utilized 
and how full the queue is, under the assumption that if the existing 
hardware is at capacity, that's enough justification for spending more 
money on more hardware, or proves that the cluster was needed. However,  
I think even utilization of an HPC resource can be misleading, since 
someone could be doing something stupid in their codes that might be 
keeping the processors busy, but isn't really helping them get a 
solution any quicker. Also, this doesn't measure the opportunity cost of 
researcher swho want to use the resource but can't because they don't 
know how, or don't even know it's available to them.

Right now, the only answer I can think of regarding system utilization 
is "ask your cluster system admin - he sees everything that goes on, and 
talks to the users needing support, so he probably has a good feel for 
what's going on on your systems." Unfortunately, that doesn't make for 
pretty Powerpoint slides.

>
> I have no horse in this race as I've been doing industry HPC for the
> past few years, but I'm curious what good metrics for ROI on an academic
> or lab cluster might be. Total number of papers? Number of
> citations after an N-year time window? [shrug]
There are two metrics that are often discussed in academia: The number 
of papers published based on that research, or the amount of grant money 
brought in by researchers using that resource. There is a paper Amy Apon 
and other on this topic, titled "High Performance Computing 
Instrumentation and Research Productivity in U.S.
Universities":

http://www.jiti.net/v10/jiti.v10n2.087-098.pdf



>
> ROI measurement can sometimes be difficult even in an industrial or
> commercial setting, especially if the HPC resource is used for R&D or
> "engineering support" as opposed to something that feeds directly into
> the product.

It can be difficult, but I think it's still much easier than in an 
academic setting, where many consider the labor of grad students to be 
free, and it's difficult to put a price on the value of research results.
>
> Cheers,
> Adam
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (Darwin)
>
> iQEcBAEBCgAGBQJSk+cxAAoJEGqxms5cZfz0vEcH+gLqL/LmLZgDUcASJGn1U+M0
> SbHbwFTbA6DOeMym4gAfvfXrKueabFYA4qHPmAaCoC2XVhCr6vORMLIn27hoA0S7
> 0dI+hN0H4OUC9u9Xgp9HZtPa2YD1nlYSby6A3D751VDW5m1r4Sn3aafoODyZGkcp
> 7FyAPhCnK+EgUinICgnfaXVMvCsUJ6cMMi2OicKjLPw85KIkRMfyDbAgh3myE7p2
> S6SHKBb3PHzWqIAUZ2iEU+C1XJbwH7bw83ciOJN49T0PPX5+I1Lced76RWcIZC40
> YqYdF1kUhBBD7VMonByUBAcCRq7fl9cCLDnEh78z3pZ7+/4dqJcmmMjqyyavrmA=
> =xgE9
> -----END PGP SIGNATURE-----
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20131127/a77f8eee/attachment.html>