[Beowulf] Definition of HPC

[Ellis H. Wilson III]

On 04/16/2013 03:50 PM, Max R. Dechantsreiter wrote:
> ...Perhaps HPC should be defined problem-down, rather than hardware-up.
>
> Regarding your "Issue #1:"
>
> There are some _nearly_ EP problems that require very minimal, but crucial,
> communication (perhaps at long intervals).

Fair enough, but as I alluded to, you can stage that kind of thing. 
Let's say you have 100 tasks that all have to sync up half-way through 
their run, and have low I/O requirements.  Rather than making sure you 
can get 100 cores to run on, just get 20 cores, and split the execution 
into two phases, phase A prior to the barrier sync and phase B 
afterwards.  Then just issue all the 100 runs for phase A in batches of 
20, until 5 rounds have completed, and safe the intermediate output you 
need for syncing.  Then, following completion of phase A for all 100 
tasks startup phase B, pointing the new tasks at the respective outputs 
they need from A.  Again, another 5 batches, and your complete.

Assuming short run times of the tasks, having your own equipment, albeit 
1/5th of what you could get for that price in the cloud, will still be 
faster due to the startup time of the cloud-based solution and will beat 
the price as well if you can find someone to share the costs with you.

> As to your "Issue #2:"
>
> "Owned compute" has some advantages over "rented compute."  In general, the
> control one has over one's owned resources enables applications to run with
> greater performance.  Some optimizations just demand root access!

Although I hear those who have responded to this, this is particularly 
true in my case as a systems researcher.  Not only is my research 
impossible to optimize without root access, it's impossible to perform 
whatsoever.  Because of that I am constantly at odds with my IT dept at 
PSU.  Hence the NAS server and small beo-cluster in my home...

Are there any people out there that provide rented, but more 
traditionally "HPC" compute (as in, it has Infiniband, huge memory, no 
virtualization, fast NAS storage, etc)?  I'm not sure there is a huge 
market for it, but it would be an interesting and differentiated offering.

Best,

ellis