[Beowulf] experience with HPC running on OpenStack [EXT]

[Tim Cutts]

I think that’s certainly true.  As with all things, it depends on your workloads.  The vast majority of our genomics codes are either single threaded, or multi-threaded on a single node.  There's relatively little MPI, and we maintain a "parallel" queue on bare metal for precisely that set of workloads.  That still doesn't preclude us using OpenStack to deploy the hardware in future (we don't really want to have multiple cluster management tools if we can avoid it)

Tim

On 09/07/2020, 08:42, "Beowulf on behalf of m.somers at chem.leidenuniv.nl" <beowulf-bounces at beowulf.org on behalf of m.somers at chem.leidenuniv.nl> wrote:

    Hi,
    
    i suggest you not only look at the flexibility / complexity regarding
    administrating a cluster with OpenStack [there are also many other tools
    for that] but also *actually benchmark* with a parallel (threaded) code
    you know well and check the strong scaling via Amdhal's law by making some
    speedup graphs on VMs and baremetal. You might actually be throwing away a
    lot of raw cpu power for *real* HPC codes by doing things on VMs. Serial
    on VMs != parallel on VMs: It all depends on your codes and the VM details
    now.
    
    https://urldefense.proofpoint.com/v2/url?u=https-3A__www.researchgate.net_publication_221565869-5FVirtual-5FMachine-5FScalability-5Fon-5FMulti-2DCore-5FProcessors-5FBased-5FServers-5Ffor-5FCloud-5FComputing-5FWorkloads&d=DwIGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwAw4Q50&m=_yJYkZKYQiXZvQ0j93qojccbPBsK9sa7KPzuY0SqMfo&s=SyYiPPEcMoIVkDM4r1He3iem6VpBeJEAONvs4L1ro0E&e= 
    
    P.S. in the paper above, the mentioned floating point micro benchmarks are
    'trivially parallel': all cores work independently without any
    communication and do not share a single bigger workload. This is certainly
    not the case for actual HPC codes and these will not show such linearly
    perfect scaling neither on VMs, nor on bare metal.
    
    Mark Somers
    
    -- 
    mark somers
    tel: +31715274437
    mail: m.somers at chem.leidenuniv.nl
    web:  https://urldefense.proofpoint.com/v2/url?u=https-3A__www.universiteitleiden.nl_en_staffmembers_mark-2Dsomers&d=DwIGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwAw4Q50&m=_yJYkZKYQiXZvQ0j93qojccbPBsK9sa7KPzuY0SqMfo&s=cx4F7pwhUGVO2qwdVF9GvprpBMnzgdmfh2phS6w-k2M&e= 
    
    _______________________________________________
    Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
    To change your subscription (digest mode or unsubscribe) visit https://urldefense.proofpoint.com/v2/url?u=https-3A__beowulf.org_cgi-2Dbin_mailman_listinfo_beowulf&d=DwIGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=gSesY1AbeTURZwExR_OGFZlp9YUzrLWyYpGmwAw4Q50&m=_yJYkZKYQiXZvQ0j93qojccbPBsK9sa7KPzuY0SqMfo&s=aXHejS8q514AnQnGZaZL6TdH5xa84wUMhx-OknF5Zwo&e= 
    




-- 
 The Wellcome Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.