[Beowulf] Fault tolerance & scaling up clusters (was Re: Bright Cluster Manager)

Thu May 17 06:00:39 PDT 2018

>>>>> "J" == Lux, Jim (337K) <james.p.lux at jpl.nasa.gov> writes:

    J> The reason I hadn't looked at "diskless boot from a
    J> server" is the size of the image - assume you don't have a high
    J> bandwidth or reliable link.

This is not something to worry about with Qlustar. A (compressed)
Qlustar 10.0 image containing e.g. the core OS + slurm + OFED + Lustre is
just a mere 165MB to be transferred (eating 420MB of RAM when unpacked
as the OS  on the node) from the head to a node. Qlustar (and its
non-public ancestors) were never using anything but RAMDisks (with real
disks for scratch), the first cluster running this at the end of 2001 was on
Athlons ... and eaten-up RAM in the range of 100MB still mattered a lot
at that time :)

So over the years, we perfected our image build mechanism to achieve a
close to minimal (size-wise) OS, minimal in the sense of: Given required
functionality (wanted kernel modules, services, binaries/scripts, libs),
generate an image (module) of minimal size providing it. That is maximum
light-weight by definition.

Yes, I know, you'll probably say "well, but it's just Ubuntu ...". Not for
much longer though: CentOS support (incl. OpenHPC integration) coming
very soon ... And all Open-Source and free.

Best,

Roland

-------
https://www.q-leap.com / https://qlustar.com
          --- HPC / Storage / Cloud Linux Cluster OS ---

    J> On 5/12/18, 12:33 AM, "Beowulf on behalf of Chris Samuel"
    J> <beowulf-bounces at beowulf.org on behalf of chris at csamuel.org>
    J> wrote:

    J>     On Wednesday, 9 May 2018 2:34:11 AM AEST Lux, Jim (337K)
    J>     wrote:

    >> While I’d never claim my pack of beagles is HPC, it does share
    >> some aspects – there’s parallel work going on, the nodes need to
    >> be aware of each other and synchronize their behavior (that is,
    >> it’s not an embarrassingly parallel task that’s farmed out from a
    >> queue), and most importantly, the management has to be scalable.
    >> While I might have 4 beagles on the bench right now – the idea is
    >> to scale the approach to hundreds.  Typing “sudo apt-get install
    >> tbd-package” on 4 nodes sequentially might be ok (although pdsh
    >> and csshx help a lot) it’s not viable for 100 nodes.

    J>     At ${JOB-1) we moved to diskless nodes and booting RAMdisk
    J>     images from the management node back in 2013 and it worked
    J>     really well for us.  You no longer have the issue about nodes
    J>     getting out of step because one of them was down when you ran
    J>     your install of a package across the cluster, removed HDD
    J>     failures from the picture (though that's likely less an issue
    J>     with SSDs these days) and did I mention the peace of mind of
    J>     knowing everything is the same?  :-)

    J>     It's not new, the Blue Gene systems we had (BG/P 2010-2012
    J>     and BG/Q 2012-2016) booted RAMdisks as they were designed to
    J>     scale up to huge systems from the beginning and to try and
    J>     remove as many points of failure as possible - no moving
    J>     parts on the node cards, no local storage, no local state,

    J>     Where I am now we're pretty much the same, except instead of
    J>     booting a pure RAM disk we boot an initrd that pivots onto an
    J>     image stored on our Lustre filesystem instead.  These nodes
    J>     do have local SSDs for local scratch, but again no real local
    J>     state.

    J>     I think the place where this is going to get hard is on the
    J>     application side of things, there were things like
    J>     Fault-Tolerant MPI (which got subsumed into Open-MPI) but it
    J>     still relies on the applications being written to use and
    J>     cope with that.  Slurm includes fault tolerance support too,
    J>     in that you can request an allocation and should a node fail
    J>     you can have "hot-spare" nodes replace the dead node but
    J>     again your application needs to be able to cope with it!

    J>     It's a fascinating subject, and the exascale folks have been
    J>     talking about it for a while - LLNL's Dona Crawford keynote
    J>     was about it at the Slurm User Group in 2013 and is well
    J>     worth a read.

    J>     https://slurm.schedmd.com/SUG13/keynote.pdf

    J>     Slide 21 talks about the reliability/recovery side of things:

    J>     # Mean time between failures of minutes or seconds for
    J>     # exascale
    J>     [...]
    J>     # Need 100X improvement in MTTI so that applications can run
    J>     # for many hours. Goal is 10X improvement in hardware
    J>     # reliability. Local recovery and migration may yield another
    J>     # 10X. However, for exascale, applications will need to be
    J>     # fault resilient

    J>     She also made the point that checkpoint/restart doesn't
    J>     scale, you will likely end up spending all your compute time
    J>     doing C/R at exascale due to failures and never actually
    J>     getting any work done.