[Beowulf] Jupyter and EP HPC

[Lux, Jim (337K)]

Modeling? I would think you’re doing empirical observation and data taking….



From: Beowulf <beowulf-bounces at beowulf.org> on behalf of "beowulf at beowulf.org" <beowulf at beowulf.org>
Reply-To: John Hearns <hearnsj at googlemail.com>
Date: Sunday, July 29, 2018 at 11:05 AM
To: "beowulf at beowulf.org" <beowulf at beowulf.org>
Subject: Re: [Beowulf] Jupyter and EP HPC

https://github.com/JuliaParallel/ClusterManagers.jl

Sorry for the terse reply. Warm evening sitting beside the Maschsee in Hannover.  Modelling beer evaporation.

On Fri, 27 Jul 2018 8:47 pm Lux, Jim (337K), <james.p.lux at jpl.nasa.gov<mailto:james.p.lux at jpl.nasa.gov>> wrote:
I’ve just started using Jupyter to organize my Pythonic ramblings..

What would be kind of cool is to have a high level way to do some embarrassingly parallel python stuff, and I’m sure it’s been done, but my google skills appear to be lacking (for all I know there’s someone at JPL who is doing this, among the 6000 people doing stuff here).

What I’m thinking is this:
I have a high level python script that iterates through a set of data values for some model parameter, and farms out running the model to nodes on a cluster, but then gathers the results back.

So, I’d have N copies of the python model script on the nodes.
Almost like a pythonic version of pdsh.

Yeah, I’m sure I could use lots of subprocess() and execute() stuff (heck, I could shell pdsh), but like with all things python, someone has probably already done it before and has all the nice hooks into the Ipython kernel.



James Lux
Task Manager, DARPA High Frequency Research (DHFR) Space Testbed
Jet Propulsion Laboratory  (Mail Stop 161-213)
4800 Oak Grove Drive
Pasadena CA 91109
(818)354-2075 (office)
(818)395-2714 (cell)

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20180730/ce24eeb6/attachment.html>