[Beowulf] Intel MPI 2.0 mpdboot and large clusters, slow to start up, sometimes not at all

[Mark Hahn]

> Does anyone have any experience running intel mpi over 1000 nodes and do you have any tips to speed up task execution? Any tips to solve this issue?

it's not uncommon for someone to write naive select() code that fails
when the number of open file descriptors hits 1024...  yes, even in 
the internals of major MPI implementations.