[Beowulf] Torrents for HPC

[Skylar Thompson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/12/2012 03:42 PM, Bill Broadley wrote:
> Using MPI does make quite a bit of sense for clusters with high
> speed interconnects.  Although I suspect that being network bound
> for IO is less of a problem.  I'd consider it though, I do have
> sdr/ddr/qdr clusters around, but so far (knock on wood) not IO
> limited.  I've done a fair bit of MPI programming, but I'm not sure
> it's easy/possible to have nodes dynamically join/leave.  Worst
> case I guess you could launch a thread/process for each pair of
> peers that wanted to trade blocks and still use TCP for swapping
> metadata about what peers to connect to and block to trade.

We manage this by having users run this in the same Grid Engine
parallel environment they run their job in. This means they're
guaranteed to run the sync job on the same nodes their actual job runs
on. The copied files change so slowly that even on 1GbE network is
rarely a bottleneck, since we only transfer files that are changed.

Skylar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/XxvAACgkQsc4yyULgN4b6dACfb5KIcql9wAbcudIKiO+IMrHX
xS4An1caTjSp0MOCgb4Ach6h8ynQe7CF
=LE07
-----END PGP SIGNATURE-----