Fedora cluster project? (was Re: [Beowulf] Opteron/Athlon Clustering)

[Jeffrey B. Layton]

Mitchell Skinner wrote:

>On Wed, 2004-06-09 at 18:14, Jeffrey B. Layton wrote:
>  
>
>>    I've been using FC1 since 2 days after it came out. It's OK,
>>but I've had a few problems with it (it locks up about once a
>>week but I'm not sure that's the fault of FC1). I've found it to
>>be decent overall. However, I think it's a bit too cutting edge
>>for production clusters. The target for FC is more toward the
>>desktop IMHO. That's the wrong target for clusters. At the
>>ClusterWorld conference the panel on cluster distros was pretty
>>much in agreement that a cluster distro needs to be lighter
>>weight than a typical distro (I'll mention a few things below
>>on that).
>>    
>>
>
>My experience with fedora has been that it's pretty solid, although I
>haven't used it on a cluster.  I seem to recall seeing somewhere that
>RGB has, and I'd be curious to know about that experience.
>
>The main objection that I buy so far to Fedora on clusters relates to
>ISV distro support; obviously if you're in that boat then Fedora is
>probably more cutting edge than your ISV would like to deal with.  Linux
>in general is getting more of a desktop focus lately and Fedora is right
>there because it's closer to what's happening, but I don't get the
>feeling that the distro itself is de-emphasizing non-desktop use.
>  
>

Another important objection is the limited patch time. Many companies
don't want their admins creating updates for security holes for bug fixes.
That's what we pay vendor(s) to do. Of course a vendor could pick up
FC and support it as long as it wants, but I'm not sure how many will
do that.

>In the absence of the ISV app thing, there's some sentiment that the
>software versions in Fedora are not fully tested enough for "production"
>use.  Obviously, Fedora is not as well tested as RHEL; however, I'm not
>convinced that for clusters the extra functionality you get with newer
>software isn't worth trading away some testing.
>  
>

Perhaps you are right. However, I know from experience you can get
too aggressive with functionality and destabilize the cluster :) I think
there is definitely a trade-off in there, I just can't put my finger on
the exact trade-off.

I don't like RHEL because it's Oracle driving the requirements on it.
I don't run Oracle, not do I have a need to. However, Redhat is one
of the big Linux distro players. Consequently, people and companies
will look to them for guidance, which means you will get the Oracle
driven distro whether you need it or not (at highly inflated prices I
might add).

>I don't see how the "lighter weight" issue relates to Fedora; obviously
>your compute nodes are not going to have the full complement of packages
>regardless.
>  
>

My point is that Redhat installations, FC included, tend to load extra stuff
that isn't needed in clusters. I don't need XMMS, Samba, Mozilla, etc.
on the compute nodes. You can control this installation of extra
stuff, but why bother when other distributions give you more control
up front? I don't need extra junk on the head node either. It's just more
stuff I need to track for security problems including security patches
and for open ports, etc. Auditing requires that we track all packages
on the head node and the compute nodes at all times.

BTW, there should be an article in an upcoming ClusterWorld
Magazine issue that presents the panel discussion from the Conference.
I'm not sure it will be a direct transcription of the panel or a mix of
transcription and commentary.

>>   Since I use cAos quite a bit let me make a few comments about
>>it. cAos-1 is something like a "proof of concept" in that the developers
>>wanted to get the build process down before moving on. cAos-2, which
>>should be out very soon will be much more advanced. cAos is not a
>>straight RHEL rebuild. I like to think of it as a RHEL rebuild as a
>>starting point, but with more thought into the other components. Also,
>>many of the cAos developers are cluster people so everything they do
>>is focused on clusters to some degree. So it's a well thought out distro
>>that works great on the desktop and works very well on clusters.
>>    
>>
>
>I certainly respect the effort that people are making with cAos.  On the
>other hand, following the fedora list and peeking at the cAos lists
>gives me the impression that fedora, as a release, gets more testing
>from end users than cAos will for a while.  Obviously, cAos benefits
>from the testing that the packages in RHEL have gone through; however,
>the fact that it isn't a straight rebuild means that you can't rely on
>that quite as much.
>  
>

Most cAos testing and development is done using IRC. I'm not sure if
anyone is logging the IRC, but there's lots of very good people in there.
I also think that judging a distro by the number of posts to mailing list
is not the best idea. But I understand that in the absence of anything
else, it was something to look at - just not the best idea.

Also, I would look closely at some of the people involved in cAos.
Do a little google on the names and see what they have done. There
are some amazingly accomplished people there, some of which are
from the cluster community. Also take a close look at what people
are saying about it and how they are using. I wouldn't be suprised
to see a number of companies supporting it now :) Real live
commercial support with secuity patches and email and telephone
support - the works.

On the other hand, I think, and correct me if I'm wrong, that there is
still a big debate raging on FC because basically it's still controlled
by Redhat. For a community distro that sounds funny to me. If it's
a Redhat distro, call it a Redhat distro. If it's not, let the community
control it.

>As an example, just the other day on the rocks list someone posted that
>samba on 3.2.0 is broken because there's some version skew between the
>different samba packages.  I know cAos is not Rocks, however I think
>people underestimate the amount of work involved in coordinating a
>release.
>  
>

I'm not sure I follow you here. Can you explain this a bit more?

>>Up to date is not always good (I've been there before :)
>>    
>>
>
>Agreed; on the other hand, if cAos-2 will be "much more advanced", then
>aren't you/they moving a little in the more up-to-date direction as
>well?
>  
>

Yes, but not like FC. FC is moving on the bleeding edge (but perhaps
not as much as Gentoo). I like the concept of having a distro "out there",
but not for production clusters. cAos-2 will have some "advanced"
features, but not nearly as agressive as FC2, but definitely more
aggressive that straight RHEL. I don't know where one might put
it on the distro "spectrum" from RHEL on one end and FC2 on the
other end.

>>I really don't like anaconda. Have you looked at the code - yuck! The
>>installed in cAos (cinch) is a much better idea. It's not GUI, but that
>>is a good thing IMHO. You can even hack it since it's just a bash script!
>>    
>>
>
>IMHO, anaconda can be as ugly as it wants, as long as red hat keeps
>maintaining it and making sure it runs on a ridiculous variety of
>hardware, which they appear to be doing.  Was running clean code the
>principal reason for developing cinch, or was there functionality that
>didn't exist in anaconda that the cAos developers wanted?
>  
>

Not sure, maybe Greg will pick up on this one.

However, I would hazard to guess that it was because of clean
code and scriptability (is that a word?).

>I think there's a lot to like about the warewulf approach, if you have a
>homogenous hardware environment; I seem to recall seeing heterogenous
>support being one of the goals for ww2 and I'm looking forward to that. 
>We're not heterogenous but I think there's a decent chance that we'll
>end up that way.  
>

I'm doing heterogenuous clusters now. It's not difficult. I don't
know about mixing I2's, Opterons, PIII, etc. But you can
easily mix 32-bit processors of various models and makes. I'll
have to bug Greg about that one, but I don't think mixing
such a wide range of architectures is a wise idea (depending upon
your problem). So, you can easily do it today but not on a weird
"franken-cluster" (even though we all probably have one of those
in our basement).

>In any case, if I do end up participating in a fedora
>clustering project I'd certainly want to package up the warewulf vnfs
>and tools as an option there.
>  
>

Good Luck to you!

Jeff

>Mitch
>
>
>  
>