[Beowulf] Jupyter and EP HPC

[John Pellman]

You might find some enlightening tidbits at around the 26:55 mark here re
Ipyparallel and Kubernetes:

http://www.rce-cast.com/Podcast/rce-116-jupyter.html



On Sat, Jul 28, 2018 at 1:01 PM, Andrew Holway <andrew.holway at gmail.com>
wrote:

> Hi Jim,
>
> There is a group at JPL doing Kubernetes.  It might be interesting to ask
> them if you can execute Jobs on their clusters.
>
> Cheers,
>
> Andrew
>
> On 27 July 2018 at 20:47, Lux, Jim (337K) <james.p.lux at jpl.nasa.gov>
> wrote:
>
>> I’ve just started using Jupyter to organize my Pythonic ramblings..
>>
>>
>>
>> What would be kind of cool is to have a high level way to do some
>> embarrassingly parallel python stuff, and I’m sure it’s been done, but my
>> google skills appear to be lacking (for all I know there’s someone at JPL
>> who is doing this, among the 6000 people doing stuff here).
>>
>>
>>
>> What I’m thinking is this:
>>
>> I have a high level python script that iterates through a set of data
>> values for some model parameter, and farms out running the model to nodes
>> on a cluster, but then gathers the results back.
>>
>>
>>
>> So, I’d have N copies of the python model script on the nodes.
>>
>> Almost like a pythonic version of pdsh.
>>
>>
>>
>> Yeah, I’m sure I could use lots of subprocess() and execute() stuff
>> (heck, I could shell pdsh), but like with all things python, someone has
>> probably already done it before and has all the nice hooks into the Ipython
>> kernel.
>>
>>
>>
>>
>>
>>
>>
>> James Lux
>>
>> Task Manager, DARPA High Frequency Research (DHFR) Space Testbed
>>
>> Jet Propulsion Laboratory  (Mail Stop 161-213)
>>
>> 4800 Oak Grove Drive
>>
>> Pasadena CA 91109
>>
>> (818)354-2075 (office)
>>
>> (818)395-2714 (cell)
>>
>>
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20180728/afe9f743/attachment-0001.html>