[vortex] Problem with built-in dual 3c982, Dual AMD S2462 MB.
Heflin, Roger A.
Roger.A.Heflin@conoco.com
Wed Jan 16 15:11:50 2002
Here is an update:
I ran 16 machines configured with Linux Kernel 2.2.19 as
followed:
4 configured with 99Uc driver, noapic eth1 interface.
Speed of machine was 10.1MB per second, this benchmark
is an MPI cpu-cpu type program.
Number of network failures per machine:
Machine 1: IIII
Machine 2:
Machine 3: II
Machine 4: II
4 configured with 99Uc driver, apic, eth0 interface
Speed of machine is 10.1MB per second.
Number of network failures per machine:
Machine 1: II
Machine 2:
Machine 3: III
Machine 4: I
4 configured with 3com, apic, eth0
Speed of machine is 7.1MB per second
No crashes.
4 configured with 3com, noapic, eth0
Speed of machine is 7.1MB per second
No crashes.
To get the machine back the following is done:
ifdown eth0
rmmod 3c59x
insmod /lib/modules/2.2.19/net/3c59x.o
ifup eth0
From limited testing if I could detect the funny thing happening
we could do this
procedure and put the machine back when the event happens.
The messages that we get with the event are:
eth0: transmit timed out, tx_status 00, status e00
After this message is some excerpts from the debug log.
And here is something interesting, the Sender of the data seems
to be where the
lockup occurs. I ran a long test with 4 sendors and 4
receievers, and the 4 sendors
of the data all fell off of the network given a long enough
time. None of the 4 receivers
had any problems receiving the data that the senders were
sending and were still
network accessable. I suspect this is why of the 8 that
could not lock up
a couple of them, since my test is not symetrical on using
everyone evenly for
send/receiver, and this is especially true if the test locks up
early.
Roger
Here is where I believe everything is fine:
Jan 16 13:25:17 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:17 poeplx1008 kernel: eth0: interrupt, status e201,
latency 16 ticks.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:17 poeplx1008 kernel: eth0: interrupt, status e201,
latency 4 ticks.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:17 poeplx1008 kernel: eth0: interrupt, status e401,
latency 3 ticks.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:17 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:17 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:17 poeplx1008 kernel: eth0: interrupt, status e401,
latency 4 ticks.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:17 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 001c.
Jan 16 13:25:17 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:17 poeplx1008 kernel: eth0: interrupt, status e401,
latency 4 ticks.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:17 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 001c.
Jan 16 13:25:17 poeplx1008 kernel: Receiving packet size 98
status 60008062.
Jan 16 13:25:17 poeplx1008 kernel: Receiving packet size 110
status 6000806e.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:17 poeplx1008 kernel: eth0: interrupt, status e401,
latency 4 ticks.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:17 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 001c.
Jan 16 13:25:17 poeplx1008 kernel: Receiving packet size 94
status 805e.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Here it starts queuing up Tx packets - transmitting data?(no
time break here)
Jan 16 13:25:17 poeplx1008 kernel: eth0: Queuing Tx packet,
index 18842.
Jan 16 13:25:17 poeplx1008 kernel: eth0: interrupt, status e201,
latency 3 ticks.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:17 poeplx1008 kernel: eth0: Queuing Tx packet,
index 18843.
Jan 16 13:25:17 poeplx1008 kernel: eth0: Queuing Tx packet,
index 18844.
Jan 16 13:25:17 poeplx1008 kernel: eth0: Queuing Tx packet,
index 18845.
Jan 16 13:25:17 poeplx1008 kernel: eth0: Queuing Tx packet,
index 18846.
Jan 16 13:25:17 poeplx1008 kernel: eth0: interrupt, status e201,
latency 3 ticks.
Jan 16 13:25:17 poeplx1008 kernel: eth0: Queuing Tx packet,
index 18847.
Jan 16 13:25:17 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:17 poeplx1008 kernel: eth0: Queuing Tx packet,
index 18848.
Jan 16 13:25:17 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
(time break until something else different happens)
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e401,
latency 4 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:50 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:50 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e401,
latency 6 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:50 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:50 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194507.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194508.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194509.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194510.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194511.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194512.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194513.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194514.
Jan 16 13:25:50 poeplx1008 kernel: eth0: Queuing Tx packet,
index 194515.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 6 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 4 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 6 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 4 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 6 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e401,
latency 3 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:50 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:50 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 6 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 4 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e401,
latency 7 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:50 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:50 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 3 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e201,
latency 6 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e401,
latency 4 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:50 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:50 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e401,
latency 6 ticks.
Jan 16 13:25:50 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:50 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:50 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:50 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:50 poeplx1008 kernel: eth0: interrupt, status e401,
latency 4 ticks.
Continues on as above pretty much until:
Jan 16 13:25:57 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:57 poeplx1008 kernel: Receiving packet size 60
status 803c.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e401,
latency 3 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:57 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:57 poeplx1008 kernel: Receiving packet size 60
status 803c.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e401,
latency 3 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:57 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:57 poeplx1008 kernel: Receiving packet size 110
status 806e.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e401,
latency 3 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:57 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:57 poeplx1008 kernel: Receiving packet size 60
status 803c.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: eth0: transmit timed out,
tx_status 00 status e000.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e401,
latency 4 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:57 poeplx1008 kernel: eth0: Interrupt posted but
not delivered -- IRQ blocked by another device?
Jan 16 13:25:57 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:57 poeplx1008 kernel: Receiving packet size 92
status a000805c.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e001,
latency 255 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e001.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: Flags; bus-master 1, full
0; dirty 194515 current 194516.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: Transmit list 7b29ca30 vs.
fb29ca30.
Jan 16 13:25:57 poeplx1008 kernel: 0: @fb29ca00 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 1: @fb29ca10 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 2: @fb29ca20 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 3: @fb29ca30 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 4: @fb29ca40 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 5: @fb29ca50 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 6: @fb29ca60 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 7: @fb29ca70 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 8: @fb29ca80 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 9: @fb29ca90 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 10: @fb29caa0 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 11: @fb29cab0 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 12: @fb29cac0 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 13: @fb29cad0 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 14: @fb29cae0 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: 15: @fb29caf0 length
800005ea status 800105ea
Jan 16 13:25:57 poeplx1008 kernel: eth0: Resetting the Tx ring
pointer.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e201,
latency 5 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e201.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e401,
latency 6 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:57 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:57 poeplx1008 kernel: Receiving packet size 66
status 60008042.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e401,
latency 4 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:57 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:57 poeplx1008 kernel: Receiving packet size 243
status a00080f3.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
Jan 16 13:25:57 poeplx1008 kernel: eth0: interrupt, status e401,
latency 2 ticks.
Jan 16 13:25:57 poeplx1008 kernel: eth0: In interrupt loop,
status e401.
Jan 16 13:25:57 poeplx1008 kernel: In boomerang_rx(), status
e001, rx_status 007a.
Jan 16 13:25:57 poeplx1008 kernel: Receiving packet size 60
status 803c.
Jan 16 13:25:57 poeplx1008 kernel: eth0: exiting interrupt,
status e000.
And then ouptuts the Transmit list every 5 seconds from here.