[Beowulf] OS for 64 bit AMD

[David Kewley]

Oh goody, I get to play devil's advocate.  Wait, did I just say RH is the 
devil? ;)

No, I don't work with them, just have been a RHL, FC, and now RHEL admin for a 
few years, and have been reading mailing lists for a couple of years.

Guy Coates wrote on Friday 01 April 2005 04:16:
> On Thu, 31 Mar 2005, David Kewley wrote:
> > If customers show RH that there are real-life needs for xfs that are not
> > satisfied by ext3, then RH may well be willing to invest in in-house xfs
> > expertise.
>
> XFS has dump/restore programs that actually work,

What exactly does this mean?

> and allows for online filesystem expansion;

ext3 now has online expansion as well.

> It is also faster than ext3 (google for the benchmarks).

They've now added htrees for faster lookups in large directories, and have 
implemented reservations (no, I don't understand this latter, but I'd guess 
it speeds up filespace allocation?).


Joe Landman wrote on Friday 01 April 2005 05:49:
> In real world applications (large data set analysis codes, and related)
> it is reasonably well known that you get better performance out of ext3
> by turning off journaling, effectively making it ext2.  Ext3 has a
> number of serialization and blocking issues associate with its
> journaliing code, not to mention other issues (directories with many
> files comes to mind).

I think ext3 doesn't lose much to ext2 in some benchmark types, in fact, I 
think it may be faster in a few.  But whatever.

And again, they've now added htrees.

> The I/O dominated/intensive applications that I have seen or worked with
> using xfs vs ext3 have demonstrated significant performance advantages
> to xfs.  Due to NDA's I cannot talk about specific details though the
> performance differences were significant.

OK.  Have any of these taken place within the last 2-3 months with RHEL4? :)  
I suspect you'll see speedups on some applications now.

> > My understanding is that RH choose to support ext3 but not xfs because:
> > 1) they have in-house expertise for ext3 but not for xfs, and 2) they
> > believe that xfs has no real advantages over ext3.
>
> They have many customers that disagree with 2,

Yes.
> and 1 is a tautology, as they have invested heavily in ext3, and thus they
> have people to work on it.

It is not meant as a tautology, but as a practical acknowledgement that if 
they are to support customers' problems with xfs at the level they'd wish, 
then they'd have to hire several people, or spend extensive time to train 
several people.

As I recall, the remark about difficulty supporting xfs came from Arjan van ee 
Ven, one of their kernel developers.

> Some inherent disadvantages of ext3 show up when you start looking at
> large file systems and large files.  Xfs has much higher limits.  If you
> want to build a 30TB file system across a huge disk array attached to a
> sizeable SMP machine, can you do it with ext3?  (no as of RHEL3).  If
> you want to work with a 2.5 TB file (part of a recent benchmark we ran),
> can you do it with ext3?  (no as of RHEL3).  Xfs doesn't have a problem
> with either of these.

I don't know what the current limits are, but I'd bet they're relieved.

> > If customers show RH that there are real-life needs for xfs that are not
> > satisfied by ext3, then RH may well be willing to invest in in-house xfs
> > expertise.
>
> Unlikely.  Customers have been showing a clear need for this for a while
> (Sloan sky survey, and many others with huge and high speed data
> requirements).  Redhat prefers to use the excuse that it is a large and
> complex package.  Hmmm.  So are Xorg, Openoffice, ....
>
> I do not expect Redhat to do this.  SuSE has, as have most of the rest
> of the major distributions (including the 1 man distribution shops), so
> the excuses that one hears are ... well ... probably not the real
> reasons.  Redhat does not want to promote a competitor to technology it
> supports.  That seems to be a simpler explanation, and I believe is
> better supported by observing their actions.

That may well be the case; certainly RH has market self-interests to look 
after.  And I know that folks have voiced similar suspicions about RH many 
times regarding many details of technology choices they've made.  May be, may 
not be, we'll see.

> Note:  FC-x has xfs enabled, one only needs to use "linux xfs" on the
> boot line during installation.

Yup.  They don't have to support it, which is likely why they don't bother 
disabling it. :)

David