[Beowulf] OS for 64 bit AMD

[Joe Landman]

David Kewley wrote:
> Oh goody, I get to play devil's advocate.  Wait, did I just say RH is the 
> devil? ;)

???

[...]

>>XFS has dump/restore programs that actually work,
> 
> 
> What exactly does this mean?

XFS' dump and restore programs work.

[...]

>>It is also faster than ext3 (google for the benchmarks).
> 
> 
> They've now added htrees for faster lookups in large directories, and have 
> implemented reservations (no, I don't understand this latter, but I'd guess 
> it speeds up filespace allocation?).

xfs does things by extents and b*trees.  Has for a long time (almost a 
decade).  ext3 just got a tree implementation.

> Joe Landman wrote on Friday 01 April 2005 05:49:
> 
>>In real world applications (large data set analysis codes, and related)
>>it is reasonably well known that you get better performance out of ext3
>>by turning off journaling, effectively making it ext2.  Ext3 has a
>>number of serialization and blocking issues associate with its
>>journaliing code, not to mention other issues (directories with many
>>files comes to mind).
> 
> 
> I think ext3 doesn't lose much to ext2 in some benchmark types, in fact, I 
> think it may be faster in a few.  But whatever.

Not benchmark tests, real world application tests.  I remember tests 
with NASTRAN and others where the journaling was a major issue for ext*. 
    So much so that the MSC.Linux distribution (made in part to run 
MSC.NASTRAN) made ext2 its default.

I have run other code recently which, again, appears to lend support to 
what I am indicating.

> And again, they've now added htrees.

which means .... what?

The only things that matters in the end are the performance, the 
stability, and the utility.  Anyone can construct benchmarks that stress 
only the areas they are interested in highlighting, or game benchmarks 
meant to expose features by tuning.

In the end xfs has been around for the better part of a decade.  It is a 
well respected, and highly tuned beast.  It is known to scale to sizes 
and performance that is still out of reach for ext3 and a few others. 
Real world applications and use cases bear this out.

> 
> 
>>The I/O dominated/intensive applications that I have seen or worked with
>>using xfs vs ext3 have demonstrated significant performance advantages
>>to xfs.  Due to NDA's I cannot talk about specific details though the
>>performance differences were significant.
> 
> 
> OK.  Have any of these taken place within the last 2-3 months with RHEL4? :)  
> I suspect you'll see speedups on some applications now.

RHEL4 came out in March.  I am unaware, unless we are in a time warp, of 
2-3 months of RHEL4.  Our work was done within the last 6 months.

>>>My understanding is that RH choose to support ext3 but not xfs because:
>>>1) they have in-house expertise for ext3 but not for xfs, and 2) they
>>>believe that xfs has no real advantages over ext3.
>>
>>They have many customers that disagree with 2,
> 
> 
> Yes.

Go search for Sloan Sky Survey.  And FNAL, and ...

> 
>>and 1 is a tautology, as they have invested heavily in ext3, and thus they
>>have people to work on it.
> 
> 
> It is not meant as a tautology, but as a practical acknowledgement that if 
> they are to support customers' problems with xfs at the level they'd wish, 
> then they'd have to hire several people, or spend extensive time to train 
> several people.

I disagree.  Redhat, like many other distro vendors is a packager.  They 
add some value in the packaging and configuration, though I will leave 
that to others to debate.  Some of their staff work on kernels, X, and 
other things.  I do not believe that they actively contribute to the 
development of everything that is in their packages.  They pick some 
things and work from there.

They know for example, that IBM actively works on and contributes JFS, 
and they have a vested interest to make that operate correctly.  SGI has 
  an active on-going vested interest in xfs, and they are continuing to 
support that.

> As I recall, the remark about difficulty supporting xfs came from Arjan van ee 
> Ven, one of their kernel developers.

So ... and I am not sure how you might answer this, how come Klaus 
Knopper, the Ubuntu team, SuSE, Mandrake, Debian, (most of the rest of 
the "major" distros) can support xfs, but Redhat cannot?  It is fairly 
obviously not a manpower issue, but a business decision.  Remember, 
Klaus is effectively a one man shop.

The argument about hiring additional people to handle that makes very 
little sense, especially as they bought Sistina, and are offering this 
as their high end file system for clustered servers.  It looks very much 
like a business decision, which I and a fair number of their customers 
disagree, and have voiced to them repeatedly.  Some of us have voted 
with our wallets, and are using other distributions.  SuSE is quite nice 
apart from some minor annoyances with autodetection.


>>Some inherent disadvantages of ext3 show up when you start looking at
>>large file systems and large files.  Xfs has much higher limits.  If you
>>want to build a 30TB file system across a huge disk array attached to a
>>sizeable SMP machine, can you do it with ext3?  (no as of RHEL3).  If
>>you want to work with a 2.5 TB file (part of a recent benchmark we ran),
>>can you do it with ext3?  (no as of RHEL3).  Xfs doesn't have a problem
>>with either of these.
> 
> 
> I don't know what the current limits are, but I'd bet they're relieved.

Theoretically you can do a 32 TB partition on RHEL4.  You can do a 2 TB 
file on RHEL 4.  You wont have a problem using xfs with a 32TB (or 32PB) 
file system, and you can go somewhat higher than 2TB in individual file 
size.  On a 2.6 kernel with CONFIG_LBD enabled, you can get up to a 9 
million TB file on the same sized file system, though I would challenge 
you to gather, power, and connect sufficient number of disks to test this.



> 
> 
>>>If customers show RH that there are real-life needs for xfs that are not
>>>satisfied by ext3, then RH may well be willing to invest in in-house xfs
>>>expertise.
>>
>>Unlikely.  Customers have been showing a clear need for this for a while
>>(Sloan sky survey, and many others with huge and high speed data
>>requirements).  Redhat prefers to use the excuse that it is a large and
>>complex package.  Hmmm.  So are Xorg, Openoffice, ....
>>
>>I do not expect Redhat to do this.  SuSE has, as have most of the rest
>>of the major distributions (including the 1 man distribution shops), so
>>the excuses that one hears are ... well ... probably not the real
>>reasons.  Redhat does not want to promote a competitor to technology it
>>supports.  That seems to be a simpler explanation, and I believe is
>>better supported by observing their actions.
> 
> 
> That may well be the case; certainly RH has market self-interests to look 
> after.  And I know that folks have voiced similar suspicions about RH many 
> times regarding many details of technology choices they've made.  May be, may 
> not be, we'll see.

The first sentence parses and reads fine, though the next two, I am not 
sure what you are saying.  RH's self interest is in pushing the 
technology they actively invest in.  There is nothing wrong with this. 
No one is saying that this is wrong.  It is also in RH's self interest 
to listen to their customers.  Pushing your own preferred technology on 
your customers whom are asking for a different technology (that would 
neither cost you time/money/significant effort) is a sure fire way to 
make your customers into your competitors customers.

> 
> 
>>Note:  FC-x has xfs enabled, one only needs to use "linux xfs" on the
>>boot line during installation.
> 
> 
> Yup.  They don't have to support it, which is likely why they don't bother 
> disabling it. :)

Oddly enough, it works quite well in FC.  Imagine that.  They can have 
their cake and eat it too.  Without spending time/money/effort.  Other 
distros are doing it, and they are too in FC-x.

Only they choose not to in RHEL4.  Obviously support has nothing to do 
with it (FC-x is a beta program for RHEL, and you need to fix your 
customers bug reports to make it a useful program).


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615