[Beowulf] recommendations for parallel IO

[Paulo Afonso Lopes]

Oi, Ricardo.

>
>   Hi all
>
>   We have a small cluster but some users need to use MPI-IO. We have a
> NFS3
> shared partition but you would need to mount it with special options who
> would hurt performance.

Yes... options include all the available ways to enforce "no client
caching" and that is (usually) very bad for performance :-)

There's also NFS4.1 but I can't speak about it other than the last time (>
6 months) I looked, it was VERY OS dependent (you had to run kernel
2.6.x.y.z); furthermore, I haven't looked at the MPI-IO support status on
4.1.

> We are looking into a nice parallel file system to
> deploy in this context. We got 4 boxes with a 500Gb disk in each, for the

Are the 4 boxes just for the filesystem service, or are they "the small
cluster" ?

> moment, connected with Gb. We have another Gb connection dedicated to the
> MPI traffic.
>
>   We need an open source solution, we are looking into PVFS

I am using it and I have very good experiences with PVFS: easy
installation, support -- !excellent! -- and good performance; the only
minuses are 1) CPU use when you used GbE "dumb" cards (those usually
integrated in the mobo) and 2) some limitations on the POSIX interface
(which should not hurt you, as you're going the MPI-IO way).

If the 4 nodes are both the compute and I/O nodes, then (1) above will
hurt your applications iff they overlap I/O and computation.


>  and Gluster
> (but from what we see, Gluster doesn't quit fit the bill? It's more a
> distributed filesystem than a parallel filesystem... or are we taking the
> wrong turn on our reasoning, somewhere about this?)
>
Yes, you're right, me thinks :-) However, I have no experience

You could also look at Lustre. Back in the days (of CFS, Inc.) where there
were 2 versions, the free and the not-free, the free was a nightmare to
install (been there), had quite a few bugs, and was always months behind
the non-free, but I am told that when Sun picked it they changed that, and
there is only one version (altough there are some mumbles about Oracle's
doing this and that)

Abraço

paulo


-- 
Paulo Afonso Lopes                        | Tel: +351- 21 294 8536
Departamento de Informática               | 294 8300 ext.10702
Faculdade de Ciências e Tecnologia        | Fax: +351- 21 294 8541
Universidade Nova de Lisboa               | e-mail: poral at fct.unl.pt
2829-516 Caparica, PORTUGAL