[Beowulf] Gentoo in the HPC environment

[Joe Landman]

On 06/25/2014 03:42 PM, Gavin W. Burris wrote:
> On Wed 06/25/14 05:29PM +0000, Andrew M.A. Cater wrote:
>> RHEL doesn't cut it for these people: they know that they want later
>> GCC / different commercial compilers / hand written assembly
>
> Any OS flavor will allow for all of these, including RHEL.  I'm not sure
> how RHEL prevents any of this.

How much of the rest of your environment do you need to recompile to 
make sure the ABI/API changes don't introduce mismatched impedance bugs?

We've largely given up on trying to get Red Hat to use a Perl that was 
released this decade.  They have a history of breaking dev tools (gcc 
2.96 anyone?  Perl 5.8.x with the broken patches causing allocations to 
take 100x the time they normally required.  Python 2.x, for x<7, causing 
massive breakage in other stacks).

So we build our own tools in our own tree, and its on every node.  Not 
impossible in Red Hat/CentOS, but we have to work around some of these 
ABI/API and library changes.

>
>> -  a later kernel with a smarter scheduler ...
>
> I'd really like to be sold on the latest 3.x kernel scheduler, but I am
> not sure that it would provide a significant performance improvement for
> a loaded compute node.

You are aware that the good folks at Red Hat backport some useful things 
from later kernels ...

>
>> At this point, you might as well get your local team to support Debian
>> on this - and you will have all the extra packages that Debian may
>> provide over RHEL :)
>
> I like having a local team.  But I like backing them up, too.  When they
> hit a weird hardware error or performance problem, they should be able
> to call an expert systems engineer that has the aggregate experience
> from many deployments.

... which, as often as not, leads to some profoundly interesting 
failures and performance regressions.  Think the THP fiasco.  6.2 and 
prior releases.  The tuned monstrosity. Etc.

No one is trying to tell you that you should look at other distros.  But 
I might suggest that others have perfectly valid reasons for 
consideration of alternative distros, and the arguments you are using to 
justify this one distro aren't quite as strong as you might think.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
twtr : @scalableinfo
phone: +1 734 786 8423 x121
cell : +1 734 612 4615