[Beowulf] SC13 wrapup, please post your own

[Prentice Bisbal]

On 11/26/2013 09:15 AM, Lux, Jim (337C) wrote:
> On 11/25/13 4:11 PM, "Adam DeConinck" <ajdecon at ajdecon.org> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>>> 4. I went to a BoF on ROI on HPC investment. All the presentations in
>>> the BoF frustrated me. Not because they were poorly done, but because
>>> they tried to measure the value of a cluster by number of papers
>>> published that used that HPC resource. I think that's a crappy, crappy
>>> metric, but haven't been able to come up with a better one myself yet.
>>> I
>>> was very vocal with my comments and criticisms of the presentations, so
>>> if any of the presenters are reading this now, I apologize for
>>> hi-jacking your BoF. Getting good ROI on a cluster is close to my
>>> heart,
>>> but is also difficult to quantify and measure. I hope I can be part of
>>> the discussion next year.
>>>
>> Do you have any thoughts you can share on what alternative metrics
>> might look like, even if you can't think of one that's clearly better?
>>
>> I have no horse in this race as I've been doing industry HPC for the
>> past few years, but I'm curious what good metrics for ROI on an academic
>> or lab cluster might be. Total number of papers? Number of
>> citations after an N-year time window? [shrug]
>>
>> ROI measurement can sometimes be difficult even in an industrial or
>> commercial setting, especially if the HPC resource is used for R&D or
>> "engineering support" as opposed to something that feeds directly into
>> the product.
>>
>> Cheers,
>> Adam
> Definitely a challenge.
>
> Maybe we have webcams that look at all the users and we calculate
> percentage of time smiling while interacting with the cluster?
>
> ROI for "technology development" is a tough thing to calculate.  ROI, by
> it's nature is a "money returned for money spent", and the return is
> somewhat intangible.
>
> All of these metrics require having a baseline so you can do a
> before/after comparison.  And realistically, there needs to be a fairly
> long averaging time on the metric.  Here's the annual paper output of a
> noted physicist.
>
> 1901  1
> 1902  2
> 1903  1
> 1904  1
> 1905 25
> 1906  6
> 1907  8
> 1908  4
> 1909  5
> 1910  6
> 1911  8
>
> How would you evaluate the ROI of feeding him?  Started kind of slow, had
> a really good year, and likely received a "exceeds expectations" annual
> review. But that set a new bar, and now his supervisor is going to be
> hammering him.. Dude, your output is slacking off, I think we need to put
> you on a performance improvement plan, and this year, you're going to be
> "does not meet" in your review.
>
>
> The other problem is that paper publishing (and the schedule thereof) is
> influenced by things other than availability of computational resources,
> so you need a very large sample so those influences average out.  For
> instance, lack of funds or permission to travel to a conference, combined
> with the recent fad of "you must present in person" will have an effect.
> The sequester and/or furlough will almost certainly manifest itself in any
> sort of time series counting publications.
>
> The other thing is that there is a long gestation period for some work.
> You might not have something "publishable", especially with the bias
> against publishing null or negative results. That doesn't mean that the
> HPC work wasn't useful, if it found a bunch of "ways not to go".
>
> There might also be a availability of workforce to grind out the papers
> issue.  At least at JPL, relatively few people work on a single job or
> task.  A more typical scenario is having 2 or 3 projects you work on
> simultaneously, along with half a dozen things you support. There is a
> tendency to spend one's time on the latest thing to go wrong, and in a "do
> more with less" environment, there's not a lot of down time in which to
> catch up.
>
> In an environment where short term results are more important (or, at
> least have more "gain" in the control loop) it's tough to push "getting
> published" higher up the priority list, since the personal ill effects of
> not publishing may be years down the road, compared to immediate ill
> effects of "the project will be cancelled if we don't make the deliverable
> this month".
>
> At JPL, it is easy to tell in which organizations, the "papers published"
> metric is important in annual ranking and review: they're the ones with
> lots of papers.  That's not to say that other organizations don't do lots
> of publishable work, but if your annual review depends on something
> *other* than the metric, you're not going to spend your time doing it.
>
> Export controls also rear their ugly head.  A lot of interesting problems
> that can be attacked by HPC are the practical ones. But in a number of
> industries, once you move beyond pure research and theory (TRL 3), you get
> into an area where it is either competition sensitive or export
> controlled.    It is true that you could write a paper that is suitably
> expurgated and sanitized, but you still have to go through the export
> control/public release review process. And that's time consuming too.
>
>
> The whole competition sensitive/proprietary rights/ export controls aspect
> might be why some of you have commented on the gulf between what's
> presented on the show floor and what's presented in the talks.
>
>
Thanks for help underscoring why # of publications or grant money is a 
lousy metric for HPC ROI. I'm saving this one for future reference.

One concern with publications as a metric, in addition to measuring the 
quantity, what is the quality? For example, I think it's safe to say 
that the benefits are Einsteins paper outside the scientific community 
were hard to imagine right away, but over a hundred years, almost all of 
humanity is still benefiting from them. For example, I'm sure no one 
thought of the GPS system in 1905, but today, just about everyone in the 
first world, and even many people in the second and third world either 
have a device with GPS capabilities, or somehow benefit from GPS 
systems, which wouldn't be possible with his theory of relativity. How 
could you calculate for that return on investment in 1905? And then 
there's Alan Turing's paper on computers, which created a new industry 
that is now a large percentage of the global economy... There's just no 
way you can accurately predict the "return" on academic research.

--
Prentice