[Beowulf] Jupyter and EP HPC

[Lux, Jim (337K)]

I've just started using Jupyter to organize my Pythonic ramblings..

What would be kind of cool is to have a high level way to do some embarrassingly parallel python stuff, and I'm sure it's been done, but my google skills appear to be lacking (for all I know there's someone at JPL who is doing this, among the 6000 people doing stuff here).

What I'm thinking is this:
I have a high level python script that iterates through a set of data values for some model parameter, and farms out running the model to nodes on a cluster, but then gathers the results back.

So, I'd have N copies of the python model script on the nodes.
Almost like a pythonic version of pdsh.

Yeah, I'm sure I could use lots of subprocess() and execute() stuff (heck, I could shell pdsh), but like with all things python, someone has probably already done it before and has all the nice hooks into the Ipython kernel.



James Lux
Task Manager, DARPA High Frequency Research (DHFR) Space Testbed
Jet Propulsion Laboratory  (Mail Stop 161-213)
4800 Oak Grove Drive
Pasadena CA 91109
(818)354-2075 (office)
(818)395-2714 (cell)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20180727/1bcd5c76/attachment.html>