[Beowulf] User notification of new software on the cluster

[John Hanks]

Acknowledging that this is an area where personal preference and existing
culture play a very big role in what works for any given group/environment,
I'll attempt an answer noting how this works for us. Warning: I'm prone to
ranting...

We use the paid version (with .edu discount) and users must have a
university email address to sign up or must get an explicit invite from us.
We don't integrate any kind of auth between slack and our campus AD though.
The separation gives us a communication channel that is semi-independent of
local foobar'ing of networks and services. Our main open-to-all channels
are #general (where all announcements, problems, questions, etc., originate
or get reported), #python (questions/answers about python, which gets used
fairly heavily) and #sysadmin (where config change commit logs go and
sysadmin stuff gets discussed). We then have a number of channels for
individuals, lab groups, specific projects, etc., where more
private/focused discussions take place. Last we use free single channel
guest invites to have vendor channels where we can invite a specific vendor
during troubleshooting sessions. We are also seeing an increase in
single-channel invite requests for people who are doing short term
collaboration with others from outside the university. Announcements that
affect everyone go to @everyone in #general and our MOTD directs people
there to stay up to date allowing opt-in and easy individual control over
how much info they get from us.

<rant>I hate writing documentation. Absolutely hate it. We have the
obligatory wiki page, but it is almost immediately out of date once written
and updating it is boring and seems like a waste of time especially
considering the tiny number of views it gets. But I love answering
questions and solving puzzles (how do you do X in SLURM? how to script Y in
python?...) By posting these answers to slack using snippets and pinned
posts, we get cheap, permanent documentation that is time-stamped,
searchable and syntax highlighted. And as a bonus the community becomes
self supporting. At first I worried about organization, but the reality is
that we are living in a search based world so it just seems to work (been
to stackoverflow/stackexchange lately?). And as a sysadmin (over 25 years
now, my how time flies) I have never seen or worked in an IT environment
where change management and change control provided more benefit than the
damage it causes to the culture, productivity and workflow of a group. In
general ITIL, ISOxxxx, lean six sigma,etc., exist to provide pseudo-work to
justify the employment of non-technical people who are otherwise useless,
but that is another rant altogether... Adopting the habit of keeping ALL
configs in a subversion repo (feel free to use git or whatever works) then
having a commit-hook to post the commit logs to a public slack channel is a
self enforcing way to keep all interested parties up to date on
configuration changes which requires very little additional effort. It also
creates a history, enables reverting easily and a host of other great side
effects. That it doesn't require a project manager, long meetings, a
strategic plan, etc., irritates those who need those things to justify
their white-collar welfare check, err, I mean paycheck. But when you just
need to communicate and stay as close to the bleeding edge as possible this
approach distills out the basic necessities. Having full transparency on
your commit logs, even though users can't access the actual repo unless
explicitly added (which we are happy to do upon request on a case by case
basis) keeps an open and trust based relationship between sysadmins and
users. You cannot put a price on how that positively impacts the overall
environment.</rant>

I've tried blogs, wikis, mailing lists and periodic user meetings. You can
meet the needs of some small portion of users with each of those but none
works for everyone and all of those approaches require moderate to large
investment in organizing and updating. Slack (or similar tools) allow
everything to grow organically and dynamically respond to changes in
environment, user requirements, or anything else that comes along. Since we
manage nodes, storage and software in an equally organic and dynamic way,
the entire approach allows us to do this in a self-documenting way with
very little staff effort (1 admin and 1 app person for 20,000 cores, just
under 2 PB storage. several dozen workstations and 200+ users with a large
range of applications and services supported).

But, definitely imagine YMMV in a giant font here. The level of freedom,
openness and transparency that makes this work for us isn't always an
option and this has evolved and thrived here in no small part because the
two of us are not subject to intrusions or meddling from a manager who
needs to justify their existence (we are fortunate enough to have an
awesome boss who lets us do our jobs). The culture also extends into other
areas, for example, there is no account provisioning process or form, if
you are in the university Active Directory, you can log in and use the
resources in the default fairshare pool, no permission needed. Slack is
just part of an overall cluster culture built around eliminating busywork,
pseudo-work and process so we can focus on enabling computation and
science. If you have an existing group of pseudo-workers involved, this
approach won't get much traction. Unless perhaps you can re-title them all
"community and social media managers".

jbh

On Wed, Sep 28, 2016 at 8:10 PM Christopher Samuel <samuel at unimelb.edu.au>
wrote:

> Hi John,
>
> On 28/09/16 09:55, John Hanks wrote:
>
> > We take the approach that our cluster is "community managed" and discuss
> > all aspects of managing it, software installs, problems, usage,
> > scheduling, etc., in a dedicated slack.com <http://slack.com> instance
> > for our center.
>
> A really interesting summary and I think that's a nice way of doing
> things - building a community around it is something we tried in the
> early days here with forums but we had little interest from users and
> the systems admins never had time to check them anyway.
>
> I'm curious whether it is an open channel or if you have some way for
> users to auth themselves to access it?
>
> Also do you use the free or paid version?  The uni uses the free version
> for its involvement in the national cloud programme and the message
> limit it has can be annoying.
>
> When I was my previous job I ran a blog for a while (called syslog,
> forgive me!) about updates and changes, but again ran into the problem
> of too much work and too little time to keep updated.
>
> All the best,
> Chris
> --
>  Christopher Samuel        Senior Systems Administrator
>  VLSCI - Victorian Life Sciences Computation Initiative
>  Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
>  http://www.vlsci.org.au/      http://twitter.com/vlsci
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-- 
‘[A] talent for following the ways of yesterday, is not sufficient to
improve the world of today.’
 - King Wu-Ling, ruler of the Zhao state in northern China, 307 BC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20160929/cb5f1d5d/attachment.html>