[Beowulf] OS for 64 bit AMD

[Joe Landman]

David Kewley wrote:

>>David Kewley wrote:
>>>I am at this moment building a NAS server, and am working to enable and
>>>test xfs on RHEL4, for a ~8TB data volume.  I *want* to try xfs on RHEL4,
>>>and I hope to use it in production if I can satisfy myself that it's the
>>>right choice.
>>Use FC-3++ and install with the xfs option.  This will allow you to
>>format your large area as either ext3 or xfs.
> 
> FC gives about 1 year of updates from RH, then another year and a half of 
> slowly-released, community-generated updates from the Fedora Legacy project.  
> RHEL gives me 5 years of RH updates.  For servers, I'll use RHEL so I can 
> plan never to upgrade the OS, just retire the hardware eventually.

My bad...  I meant for your testing.  You get the RHEL4 kernel with what 
would have been the xfs support for testing.   For real installations, 
if support matters, go with what you are comfortable with.  I have had 
good luck with SuSE 9.x on small file servers.  I use FC's as test 
clients, but not as servers (due to the nature of the changes you indicate).

> After a few attempts at enabling xfs in RHEL4's kernel config, I am still 
> figuring out how to stop 'make oldconfig' from taking my CONFIG_XFS_FS=y and 
> making it into =n. :/  I'll solve it shortly I'm sure.  Others have reported 
> that xfs works fine in RHEL4.

Please let me know if you get there.  I might need to go down this route 
for some customer work later on this year.

[...]

>>If you read the forums as you pointed out later, you will note that
>>there has been a large chorus for quite some time about this.  Redhat's
>>response to this chorus has been "wait, ext* will get better".  Yes, it
>>has gotten better, but no, this is not what the customers asked for.
> 
> In your experience, do customers want certainly performance characteristics 
> (which could conceivably be satisfied by ext3), or do they want e.g. xfs 
> regardless of actual performance?

Most care about supportability, quality of code (e.g. how well tested is 
it), performance, ease of integration.   In most other distros, xfs is 
fairly easy to setup/use.

[...]

>>I am astounded... when the customers in the threads you mention, and in
>>the preceding threads made it clear over the last few years that it did
>>not indeed meet their needs, and can they please have support for
>>something that does ...
> 
> Sometimes the customers say they want **xfs**.  RH is ignoring that request.

Yes they are.

> Sometimes the customers say they want some of the features that xfs provides, 
> e.g. large filesystems, high performance, growing filesystems, etc.  RH says, 
> "Fine, we'll give you that in ext3."  I do not know whether RH has succeeded 
> in meeting customers' feature expectations with ext3 in 2.6.10.  Do you?

Very hard to answer question.  Put another way, yes, eventually ext3 
will be a reasonable replacement for xfs after sufficient time and 
effort to bake in goodness, and bake out badness.  When will this be? 
Who knows.  Is it able to do what xfs can do today?  No 
(size/performance issues).

So rather than offer customers what they want (which the other distros 
have done), they offer what they would prefer to offer, and make what 
they prefer to offer closer to what people want.  The question is how 
long will it take to get there.

>>I don't think that is a reasonable argument 
>>that what they provide is good enough and meets needs, when clearly
>>specific large customers are asking for something different.
> 
> If ext3 in RHEL4 fails to meet performance needs of RH customers, then I will 
> agree with you.  To my knowledge, the jury isn't in yet, given changes that 
> were just merged in 2.6.10.

So this is a major functionality upgrade into 2.6.10.  Have all the bugs 
been isolated, or the major ones?  One of Arjen's comments (or maybe 
David's) was that you don't want to mess too much with file systems. 
They are right.  Of course to get to where they need to be, they need to 
mess with file systems.  Xfs has been baking for the better part of 10 
years.  It is pretty stable/good.

>>>and they don't want to invest in talent to
>>>support other enterprise filesystems for major support-purchasing
>>>customers.
>>Again, if small one-man distros can support the system, the argument
>>rings hollow.
> 
> Either they're stonewalling, or fully supporting (not just supplying, but 
> after-sales *supporting*) a new filesystem is a major undertaking that they 
> deemed unnecessary.  I don't know which is the case; Arjan says the latter.

Most distros I am aware of kick bugs back to the relevant subsystem 
owners, unless they have experts (and folks with commit privs) in house.

>>>If you believe he's misguided, argue with what he wrote. :)
>>Don't need to.  This has been debated in various threads for the last 4+
>>years.  Ext3 will continue to improve, and it is probably good enough
>>for some section of users.  For others, who need what xfs provides, ext3
>>is not sufficient.
> 
> What does xfs provide that ext3 in 2.6.10 doesn't?  I ask sincerely.  I 
> suppose at this point you could only answer guided by feature lists (which I 
> don't know as well as you do).
> 
> To answer performance questions, we'd have to test, no?  Or does history 
> condemn ext3 to never challenge xfs in those domains where xfs has 
> historically beaten ext3? :)

No, always always test.  Lets start with say a nice 64 TB file system. 
Create a few 16 TB files on it.  md5sum them.

Sure, that might not be reasonable in the time you have, or in the space 
you have, but when you have customers building 100++ TB file systems, is 
it reasonable to tell them that tough, you are stuck at 16 TB?  Or 
worse, for the folks with huge output file requirements, or huge data 
set requirements, who have been asking for this for years, is it 
reasonable to tell them "tough, work on 2TB files or less"? .  Then on 
the performance side, when doing large block I/O (not the "standard" 
delete, mkdir, and other tests that are not representative of how most 
folks do their work on big fast disks), is it reasonable to say turn off 
journaling for better performance?  No, I don't believe so.

So, yes, please test.  With real applications, as they are the only 
benchmarks that matter in the end.

> 
>>>How can you say, "Obviously support has nothing to do with it"?  I don't
>>>get that.  They have major customers buying expensive support for RHEL. 
>>>They offer no paid support for FC.  If they choose, they can ignore bug
>>>reports on FC xfs.
>>https://bugzilla.redhat.com/beta/show_bug.cgi?id=150427
>>http://marc.free.net.ph/message/20050310.085034.9d849d03.en.html
>>
>>Redhat pushes them off to the fs maintainers, as they should.  Bugs
>>ignored make for bad practice.
> 
> This behavior is consistent with declining to develop in-house xfs expertise, 
> no?

Absolutely. They do not need to develop this expertise in house.

>>If it were a support issue for RH, it should be a support issue for
>>SuSE, for Mandrake, for Debian, for ....
>>
>>But it isn't.
> 
> Leave Debian out of it -- the Debian maintainers don't offer support 
> contracts.

Others do for them.  Ubuntu (or the folks behind them) and a number of 
others do.

> 
> So let's see how SuSE and Mandrake respond to bug reports in major components 
> like filesystems.  Do they never say, "Sorry, we can't help you with that, 
> please go upstream" with an implicit "because we don't have in-house 
> expertise"?

They don't push the customer upstream, they push the bugs upstream (at 
least on the SuSE side, this is my understanding).

> 
> Unfortunately, I don't see a quick way to find out an answer to this question.  
> Quick look-arounds on the SuSE and Mandrake sites haven't revealed to me 
> anythink like bugzilla.redhat.com.  Are there near-equivalents, where we can 
> see community bug reports, and read corporate engineers' responses?
> 
> 
> 
> Again, my main point is that testing is needed to see whether ext3's 
> developers have made major strides in 2.6.10, or have once again failed to 
> live up to promises.  Anything else is speculation (which we're doing a 
> pretty good job of in this subthread :).

Not really... For a long time xfs and jfs have been very good at doing 
what they are supposed to do.  For a long time, ext3 has not been great 
in the same workload mix.  I am sure it has gotten better.  But xfs and 
jfs have not stagnated during this time either.  From a historical point 
of view, xfs and jfs have been around longer than ext3, and have been 
delivering on the high performance promise with large file systems. 
ext3 is just getting features now that had been designed in to the xfs 
bits from the beginning (in 1995).  There is little reason to expect 
ext3 to be better/faster on the things that xfs has dominated on (e.g. I 
am quite skeptical).  Moreover, Redhat has had every opportunity to do 
what the rest of distro-dom have done, though they have declined to go 
there, despite being asked to years ago by high profile customers.

I remain skeptical that ext3 is performance equivalent to xfs until I 
test it on my workloads and see repeatable evidence that for 
non-trivial, and realistic workloads, it performs the same/better.  I 
don't have the time to test it now.  You have the time to test it on 
your workloads.  Go ahead.  Let us know how it goes.

> 
> David
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615