[Beowulf] software for activating one of many programs but not the others?

[Gus Correa]

Hi David

As Ryan and other said,
you don't need to install everything in /usr/local or similar.
You can make a separate location to install the packages,
install them (using configure --prefix) in directories named after the
package name, version,
and perhaps the compiler and version used, (say,
/sw/openmpi/4.0.1/gnu/4.8.5/).

You can then use the traditional
Environment Modules (Tcl based) http://modules.sourceforge.net/
or the newer Lmod (Lua based) https://lmod.readthedocs.io/en/latest/
to set the environment properly
(i.e., prepending or appending to PATH, LD_LIBRARY_PATH, INCLUDE, MANPATH,
etc),
pointing to the bin, lib, include, etc, subdirectories of your software
installation directory
(say, /sw/openmpi/4.0.1/gnu/4.8.5/{bin,lib,include,man, ...}
There are prototype modules available out there,
to help you get started writing them for your installation.
On CentOS 7 environment-modules is available from the base repository, and
Lmod is available from epel.
Or you can install the latest from the links above.

Users can even create their own modules, to load the software of their
choice.
Modules support requisites and conflicts.
Properly written modules produce a clean unique envirnoment for each
different purpose.
Switching between different environments is also easy.

As Chris and others said, you can also use EasyBuild to streamline the
configure/make/install procedure
https://easybuild.readthedocs.io/en/latest/index.html ,
but manual installs are perfectly fine.

I hope this helps,
Gus Correa




On Tue, Aug 20, 2019 at 2:50 PM Ryan Novosielski <novosirj at rutgers.edu>
wrote:

> Really sounds like you should be using environment modules. What I’d
> recommend to anyone starting out today would be Lmod:
> https://lmod.readthedocs.io/en/latest/
>
> Most of the software building/installation packages interface with it.
>
> Generally the software installs are done into a place that’s unique for
> each package and version, and maybe even for what compiler it was built
> with (see hierarchical).
>
> --
> ____
> || \\UTGERS,     |---------------------------*O*---------------------------
> ||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
>      `'
>
> > On Aug 20, 2019, at 1:11 PM, David Mathog <mathog at caltech.edu> wrote:
> >
> > On a system I am setting up there are a very large number of different
> software packages available.  The sources live in /usr/local/src and a
> small number of the most commonly used ones are installed in
> /usr/local/bin, /usr/local/lib and so forth.  The issue is that any of the
> target end users will only want a couple of these.  If they were all fully
> installed into /usr/local there would be some name conflicts.  They may
> also be bringing some of their own versions of these, and while $PATH order
> can help there, it would be best to avoid those possible conflicts too.
> Users don't have priv's to modify /usr/local, so they cannot
> install/uninstall there themselves.
> >
> > So I'm looking for something like
> >
> >  setup software_name install
> >  setup software_name remove
> >
> > which would install/uninstall the packages (perhaps by symlinks) from
> >
> >  /usr/local/src/software_name
> >
> > under the user's home directory.  The goal is that the setup scripts NOT
> be constructed by hand.  It would have a
> >
> >  setup software_name install
> >
> > which would emulate a:
> >
> >  make install
> >
> > and automatically translate it into the appropriate setup commands.
> Some of these packages have hundreds of programs, so anything manual is
> going to be very
> > painful.
> >
> > Anybody seen a piece of software like this?
> >
> > I don't expect this to work in all cases.  Some of these packages hard
> code paths into the binaries and/or scripts.  The only hope for them is for
> the user to do some variant of:
> >
> >    cd $HOMEDIR
> >    (cd /usr/local/src; tar -cf - software_name) | tar -xf -
> >    cd software_name
> >    make clean  #pray that it gets everything!!!
> >    ./configure --prefix=$HOMEDIR
> >    make
> >    make install
> >
> > There is a file which documents how to build each package, although it
> is nowhere near complete at this time.
> >
> > Docker is already available if the user wants to go that route, which
> avoids this whole issue, but at the cost of moving big images around.
> >
> > Thanks,
> >
> > David Mathog
> > mathog at caltech.edu
> > Manager, Sequence Analysis Facility, Biology Division, Caltech
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20190820/712c2d65/attachment.html>