> It's actually an MPI job (HPL using OpenMPI) which is
> reporting the
> problem.
>
> The head scratching continues...
>
I had a similar problem earlier in year with some blades. It was pretty ugly for a while. Most of it was related to firmware on the blades and IB.