[Beowulf] Gentoo in the HPC environment

[Prentice Bisbal]

On 06/25/2014 07:14 PM, Ellis H. Wilson III wrote:
> Catching up:
>
> 1. My comment on loving Gentoo but not suggesting it for HPC 
> deployment assumed old-style installation across the entire cluster as 
> the sole OS.  Obviously netboot/VM/Docker beats the hell out of such a 
> static implementation, I just wasn't interpreting the OP's question 
> that way, which was my bad.  Joe nailed it in this regard.
>
> 2. My new place of work (Panasas) supports RHEL 6 (maybe 7, I'm green 
> here), so it's there's one datapoint where RHEL is still supported.  
> We also support a bunch of other distros though, so it's certainly not 
> exclusive.  I /hope/ to someday push for Gentoo weak-module support. 
> We'll see about that.
>
> 3. I cannot count on my hands and toes the number of times restrictive 
> distros like RHEL bit me in the ass when I tried to run any number of 
> HPC applications to push my new/shiny/academic file system.  Holy 
> smokes what I would have given for a flexible environment then.
>
> 3.a continues below:
>
> On 06/25/2014 04:50 PM, Prentice Bisbal wrote:
>> On 06/25/2014 03:08 PM, Kilian Cavalotti wrote:
>>> On Wed, Jun 25, 2014 at 10:29 AM, Andrew M.A. Cater
>>> <amacater at galactic.demon.co.uk> wrote:
>>>> RHEL doesn't cut it for these people: they know that they want later
>>>> GCC / different commercial compilers / hand written assembly -  a 
>>>> later
>>>> kernel with a smarter scheduler ...
>>>>
>>>> SCL really doesn't work - it's stil not up to it.
>>> One way to deal with this is to separate user applications from the
>>> OS, as much as possible. And compilers could be considered as user
>>> applications.
>>> You can just use a very minimal OS on your compute nodes, then compile
>>> and install all the user facing bits in a shared location. You hand an
>>> environment modules system to the users and off they go. Systems such
>>> as EasyBuild (https://hpcugent.github.io/easybuild/) aim to facilitate
>>> this by allowing easy compilation and installation of scientific
>>> software (based on descriptive specification files, à la Gentoo
>>> ebuilds), including dependencies, and by automatically generating
>>> environment modules.
>>>
>>> This way, you don't really care what the underlying OS is. You can
>>> have as many versions of GCC, Python, R, Perl, Ruby or anything
>>> installed alongside each other with no side effect, as long as you
>>> load the right module before running your job. It's like a
>>> distro-agnostic ebuild system.
>>>
>>> You can keep the distro the hardware vendor recommends to retain
>>> support (for interconnect drivers, parallel filesystems and such)
>>> while making your users happy with the newest versions of the software
>>> they need^Wwant.
>>>
>> I agree with this approach. I've been doing this for years, and it's
>> really not has hard as people make it out to be. There's the occasional
>> 'dependency Hell' situation, but that's not usually that bad unless you
>> are building a GUI application. Fortunately, GUI users aren't too common
>> in HPC. Overall, I find compiling Perl, R, etc. from source and
>> installing each version in it's own installation directory much easier
>> then learning how to get package managers to allow you to install
>> different versions of the same packages in a sane way.
>
> 3.a: Agree with some of the above from Kilian and Prentice.  The 
> absolute best scenario (IMHO) for users and administrators is to 
> provide an environment where users who want full control and want to 
> take the responsibility for support onto themselves (e.g., me and all 
> my fellow systems researchers in my PhD) can do so, while users who 
> want all of that "nonsense" to get out of the way so they can run 
> their application X to do "real" science without having to recode it 
> for bleeding-edge (or super-old) version of GCC/Perl/Python/etc can 
> also do so.  The ONLY place this is possible is in a flexible 
> (VM/Docker/etc) environment. Administrators just need to tell the 
> latter group "choose from ImageA, ImageB, and ImageC to run your 
> application." Counter-argument will be "I have to manage those 
> images." Counter-counter argument is "Is that really harder than 
> rolling special environments to cope with those users (which you'll 
> have to do anyhow)?"

I disagree with needed separate images in a VM/Docker/etc. environment, 
maybe if your doing system-level research, but most HPC users work 
exclusively in the user-space,and just want to run their 
MATLAB/NAMD/LAMMPS, whatever job. In this case, just installing 
user-space applications and libraries in a different path from the 
distro-supplied versions is adequate. Modifying PATH and a few other 
environment variables that most users can handle is all you need 
modules, lmod, softenv and other utilities make that even easier for users.

If you're doing lower-level research involving kernel modules or kernel 
tuning, yes, you will need VMs or something. But this is usually 
'research', not 'production' HPC.

>
> I ended up doing very crazy root-stealing, chroot-establishing things 
> to get my science done in my PhD.  If you prevent intelligent people 
> from doing their work, they are going to be your worst nightmare.  
> Don't kid yourselves if you think you are doing anyone favors by 
> providing super-static OS environments like RHEL for your users.  You 
> are just being lazy (and not the good kind of programmer lazy).

This is so true. If you are a roadblock to your users, they will find a 
way around you.
>
> Best,
>
> ellis
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf