Processor contention(?) and network bandwidth on AMD

[Mark Hahn]

> This is probably in the category of "Yup, that's the way it is, deal with 
> it", but, just in case anyone has any ideas, I'm throwing it out there.

well, there are several contributing factors, which are probably
mitigated by running a decently modern (ie 2.4.18) kernel.

for instance, at gigabit speeds, you're almost certainly generating
nontrivial MM load.  there's been a huge amount of improvement in 2.4's
in how they handle ram.

2.4 (vs 2.2) has some fairly profound changes to the structure of the
network stack, including efforts to make zero-copy possible (which 
effects alignment of packets, I think, even if you're not sendfile'ing.)
I've also heard mutterings from Alan Cox-ish people that the scheme for
waking up user-space is a bit too timid (resulting in the stack punting,
and relying on ksoftirqd to eventually do the deed.)

and of course, there's a big update to the scheduler impending or 
already merged ("Ingo's O(1) scheduler").  it seems to be a lot smarter
about issues like migrating procs, affinity, waking up the right tasks, etc.

> unloaded:                                         11486.6 KB/real sec
> 2 matlab simulations:                             10637.8 KB/real sec
> 2 matlab simulations and 2 SETI at homes (nice -19):  6645.4 KB/real sec

SETI at home is obviously in the "so don't do that" category.  I expect your
matlab was decelerated by a similar amount.

though I think that another of Ingo's goals with the new scheduler
was to give more intuitive nice -19 behavior.  that is, most people 
think of nice -19 as a way to spend otherwise idle CPU on something.
Linux (and at least some other Unixes) have *NEVER* done this - 
figure around 5% CPU, and that's ignoring cache/etc effects.
anyway, I think Ingo tries to keep -19 pretty close to idle-only.
(there are fundamental issues with really implementing an idle-only
form of scheduling, since you wind up with "priority inversion",
where the idle-only task holds a lock when high-pri jobs want to do
something...)

> Ouch.  This is on RedHat 7.2 with kernel 2.4.9-31.

ugh.

NOTE TO ALL BEOWULF USERS: seriously consider running 2.4.18
or better, and *definitely* try out gcc 3.1 ASAP.  

these updates have changed my life </evangelical>


> 2.4.18 kernel (the one from SGI's 1.1 XFS release).  All showed the same 
> results (well, 2.4.18 didn't show much of a drop with just the two 
> matlabs, but still crashed with matlab+SETI).  The one Intel system I 

hmm, come to think of it, I think Ingo's scheduler isn't merged 
in 2.4.19-pre yet.

> tested (dual PIII 933 on an i860) showed very little bandwidth drop with 

i860 is a P4 chipset afaik...

> load, and no extra drop for an overload.

then again, it would not be astonishing if PIII and P4 showed
dramatically different effects of cache pollution.  remember that 
the P4 depends rather strongly on seeing a decent hit rate in 
its many caches (trace cache, normal I+D caches, TLB, prediction tables, etc)