<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 11/20/20 4:43 PM, David Mathog
wrote:<br>
</div>
<p>[...]<br>
</p>
<blockquote type="cite"
cite="mid:alpine.LRH.2.21.2011201321480.14545@newsaf.bio.caltech.edu">
<br>
Also, searching turned up very little information on using MPI
with perl.
<br>
(Lots on using MPI with other languages of course.)
<br>
The Parallel::MPI::Simple module is itself almost a decade old.
<br>
We have a batch manager but I would prefer not to use it in this
case.
<br>
Is there some library/method other than MPI which people typically
use these days for this sort of compute cluster process control
with Perl from the head node?
<br>
</blockquote>
<p><br>
</p>
<p>I can't say I've ever used Perl and MPI. I suppose it is doable,
but if you were doing it, I'd recommend encapsulating it with
FFI::Platypus (<a class="moz-txt-link-freetext" href="https://metacpan.org/pod/FFI::Platypus">https://metacpan.org/pod/FFI::Platypus</a>).<br>
</p>
<p>This however, doesn't seem tp be your problem per se. Your
problem sounds like "how do I launch a script on N compute nodes
at once, and wait for it to complete".<br>
<br>
If I have that correct, then you want to learn about pdsh
(<a class="moz-txt-link-freetext" href="https://github.com/chaos/pdsh">https://github.com/chaos/pdsh</a> and info here:
<a class="moz-txt-link-freetext" href="https://www.rittmanmead.com/blog/2014/12/linux-cluster-sysadmin-parallel-command-execution-with-pdsh/">https://www.rittmanmead.com/blog/2014/12/linux-cluster-sysadmin-parallel-command-execution-with-pdsh/</a>
).</p>
<p>I write most of my admin scripts in perl, and you can use pdsh as
a function within them. <br>
</p>
<p>However ...</p>
<p>MCE::Loop is your friend.</p>
<p>Combine that with something like this:</p>
<p> $mounts=`ssh -o ConnectTimeout=20 $node grep o2ib
/proc/mounts`;</p>
<p>and you can get pdsh-like control directly in Perl without
invoking pdsh.<br>
</p>
<p>The general template looks like this:</p>
<blockquote>
<p>#!/usr/bin/env perl<br>
<br>
use strict;<br>
use MCE::Loop;<br>
<br>
MCE::Loop->init(<br>
max_workers => 25, chunk_size => 1<br>
);<br>
<br>
my $nfile=shift; <br>
<br>
# grab file contents into @nodes array<br>
my @nodes;<br>
chomp(@nodes = split(/\n/,`cat $nfile`));<br>
<br>
# looping over nodes, max_workers at a time<br>
mce_loop {<br>
my ($mce, $chunk_ref, $chunk_id) = @_;<br>
# do stuff to node $_<br>
} @nodes;<br>
</p>
</blockquote>
<p><br>
</p>
<p>This will run 25 copies (max_workers) of the loop body over the
@nodes array. Incorporate the ssh bit above in the #do stuff
area, and you get basically what I think you are asking for.</p>
<p>FWIW, I've been using this pattern for a few years, most recently
on large supers over the past few months.<br>
</p>
<p><br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:alpine.LRH.2.21.2011201321480.14545@newsaf.bio.caltech.edu">
<br>
Thanks,
<br>
<br>
David Mathog
<br>
<br>
<br>
<br>
_______________________________________________
<br>
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin
Computing
<br>
To change your subscription (digest mode or unsubscribe) visit
<a class="moz-txt-link-freetext" href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a>
<br>
</blockquote>
<pre class="moz-signature" cols="72">--
Joe Landman
e: <a class="moz-txt-link-abbreviated" href="mailto:joe.landman@gmail.com">joe.landman@gmail.com</a>
t: @hpcjoe
w: <a class="moz-txt-link-freetext" href="https://scalability.org">https://scalability.org</a>
g: <a class="moz-txt-link-freetext" href="https://github.com/joelandman">https://github.com/joelandman</a>
l: <a class="moz-txt-link-freetext" href="https://www.linkedin.com/in/joelandman">https://www.linkedin.com/in/joelandman</a></pre>
</body>
</html>