[Beowulf] Network Filesystems performance

[Joe Landman]

Ashley Pittman wrote:
> On Sat, 2007-11-10 at 18:33 +1100, Chris Samuel wrote:
>> On Fri, 24 Aug 2007, Michael Will wrote:
>>> The first advise is to stay away from redhat for file servers since
>>> they have some bursty I/O bugs and don't support XFS.
>> We've run through Fedora (OK, but release cycle too quick), Ubuntu 
>> (Dapper has longer support, but a bit flakey) and now run Debian Etch 
>> (very happy).

We have used SuSE (all flavors), RHEL/Centos (many flavors), Ubuntu, and 
a few others.  We have generally found that the RHEL requires the most 
modifications to get to a reasonable base level (the Rocks folks are 
dealing with the issues of new hardware/old kernel right now, and it 
isn't pretty).  SuSE requires some tweaking (and sometimes it is 
annoying to deal with all the added "extra bonus [mis]features" like 
zenworks/zmd).  Ubuntu has been by far the easiest to make "just work" 
and work well at that.  I know it is Debian under the hood.  I know some 
folks like Fedora as well.

Most distro discussions devolve into pitched emotional battles at some 
point, full of an overabundance of heat, and an underabundance of light.

> 
> This is an interesting proposition, you seem to imply it's easier to
> change distribution that it is to customise the one you are already
> using, I'm wondering what your reasoning for this is.

I can't speak for Chris or others.  For us, there are just too many 
broken packages in some distros, that render it difficult to fix.  If 
you stick with the "officially supported" bits, you have to live with 
broken packages.  If you install your own, the "helpful" package manager 
will happily overwrite your updated package if you didn't use the 
default packaging mechanism for the update.  And if you did, it will 
report a conflict/dependency purgatory for you to unwind.

This is why I tend to favor installing as minimal systems as possible if 
you install them.  Most distros like creating huge dependency radii for 
their packages.  Trying to remove some unneeded sound libraries recently 
under SuSE caused the package manager to eventually wish to remove 
glibc.  Yeah, this could be a broken dependency list.  Or it could be a 
real dependency.  Somehow.

Some distros don't try to get in your way so much, and actually let you 
replace bits at a sub-system level.  I have been pretty impressed with 
Ubuntu(Debian) in this regard.

> Granted in theory it *should* be easy to change distro and it has
> getting easier as they have become more and more homogeneous over time
> but equally replacing the RedHat Kernel with one that supports XFS isn't
> the end of the world either.

Agreed, though all the other broken bits (Perl, et al), just make you 
want to chuck the whole distro in favor of something more modern.

Usually we deliver our software in its own distinct namespace tree. 
This allows us to not depend upon the distro issues.  This way we fight 
bugs, not distro problems.  Well, most of the time.

> Are you choosing a distro which you can run un-modified and hence buy
> support for or do you genuinely believe it's easier to learn and support

Hmmm...  I am not sure if Chris wants to "buy" support for the distro. 
Last I checked the only two distros I knew of with any sort of HPC focus 
are Scyld and Caos.  Everything else is focused upon LAMP service and 
related.  Buying support from a distro provider who doesn't grasp how 
the system is being used might not fly.

> a new distro than it is to install a new kernel on the current one?
> Personally I've always been of the opinion that you should pick a distro
> based on what you have experience of in-house and then modify that
> distro to meet your needs rather than the other way around.

If you spend weeks and weeks fighting broken bits after broken bits, it 
gets a little tiring.  I would say that a quick cost benefit analysis is 
in order.  If you can do it more quickly with less effort on another 
distro, frankly, there is little reason why you shouldn't.

> 
> Ashley,
> 

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615