[Beowulf] lspci, PCI-X 133, Mellanox

[Craig Tierney]

On Fri, 2004-08-13 at 13:09, Mikhail Kuzminsky wrote:
>    I just installed Mellanox InfiniHost MHXL-CF128-T (former 
> MTLP23108) NICs into Tyan S2880-based nodes (the location is 1st PCI-X 
> 133 slot).
> lspci -v says about 66 Mhz:
> 
> 01:03.0 PCI bridge: Mellanox Technology: Unknown device 5a46 (rev a1) 
> (prog-if 00 [Normal decode])
> 	Flags: bus master, 66Mhz, medium devsel, latency 64
> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
> 	Memory behind bridge: fc500000-fc7fffff
> 	Prefetchable memory behind bridge: 00000000eb000000-00000000fc000000
> 	Capabilities: [70] PCI-X bridge device.
> 
> 02:00.0 InfiniBand: Mellanox Technology: Unknown device 5a44 (rev a1)
> 	Subsystem: Mellanox Technology: Unknown device 5a44
> 	Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 28
> 	Memory at fc700000 (64-bit, non-prefetchable) [size=1M]
> 	Memory at fb800000 (64-bit, prefetchable) [size=8M]
> 	Memory at f0000000 (64-bit, prefetchable) [size=128M]
> 	Capabilities: [40] #11 [001f]
> 	Capabilities: [50] Vital Product Data
> 	Capabilities: [60] Message Signalled Interrupts: 64bit+ Queue=0/5 
> Enable-
> 	Capabilities: [70] PCI-X non-bridge device.
> 
> I found that /usr/include/linux/pci.h knows about 133 Mhz PCI-X.
> 
> lspci -vv says:
> 
> 01:03.0 PCI bridge: Mellanox Technology: Unknown device 5a46 (rev a1) 
> (prog-if 00 [Normal decode])
> 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B-
> 	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR-
> 	Latency: 64, cache line size 10
> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
> 	Memory behind bridge: fc500000-fc7fffff
> 	Prefetchable memory behind bridge: 00000000eb000000-00000000fc000000
> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
> 	Capabilities: [70] PCI-X bridge device.
> 		Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
> 		Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
> 		: Upstream: Capacity=0, Commitment Limit=0
> 		: Downstream: Capacity=0, Commitment Limit=0
> 
> and 02:00.0 says also about 66+ Mhz. Sorry, does it means that lspci 
> -v output is wrong (has 2 digits only, for example), but lspci -vv is 
> correct ?
> 
> BTW, is there some reasons to decrease PCI latency in BIOS setup from 
> 64 (see lspci above) to some more low level ?

Have you tested it to see how it performs?  
My Infiniband cards say the same thing.  They
perform slow (400MB/s).    It is due to bad
riser cards.  I am getting them replaced.

I would check your riser cards and see if
they are the problem.

Craig