[Beowulf] lspci, PCI-X 133, Mellanox

[Mikhail Kuzminsky]

In message from Craig Tierney <ctierney at HPTI.com> (Fri, 13 Aug 2004 
14:06:17 -0600):
>On Fri, 2004-08-13 at 13:09, Mikhail Kuzminsky wrote:
>>    I just installed Mellanox InfiniHost MHXL-CF128-T (former 
>> MTLP23108) NICs into Tyan S2880-based nodes (the location is 1st 
>>PCI-X 
>> 133 slot).
>> lspci -v says about 66 Mhz:
>> 
>> 01:03.0 PCI bridge: Mellanox Technology: Unknown device 5a46 (rev 
>>a1) 
>> (prog-if 00 [Normal decode])
>> 	Flags: bus master, 66Mhz, medium devsel, latency 64
>> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
>> 	Memory behind bridge: fc500000-fc7fffff
>> 	Prefetchable memory behind bridge: 
>>00000000eb000000-00000000fc000000
>> 	Capabilities: [70] PCI-X bridge device.
>> 
>> 02:00.0 InfiniBand: Mellanox Technology: Unknown device 5a44 (rev 
>>a1)
>> 	Subsystem: Mellanox Technology: Unknown device 5a44
>> 	Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 28
>> 	Memory at fc700000 (64-bit, non-prefetchable) [size=1M]
>> 	Memory at fb800000 (64-bit, prefetchable) [size=8M]
>> 	Memory at f0000000 (64-bit, prefetchable) [size=128M]
>> 	Capabilities: [40] #11 [001f]
>> 	Capabilities: [50] Vital Product Data
>> 	Capabilities: [60] Message Signalled Interrupts: 64bit+ Queue=0/5 
>> Enable-
>> 	Capabilities: [70] PCI-X non-bridge device.
>> 
>> I found that /usr/include/linux/pci.h knows about 133 Mhz PCI-X.
>> 
>> lspci -vv says:
>> 
>> 01:03.0 PCI bridge: Mellanox Technology: Unknown device 5a46 (rev 
>>a1) 
>> (prog-if 00 [Normal decode])
>> 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- 
>> Stepping- SERR+ FastB2B-
>> 	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
>> <TAbort- <MAbort- >SERR- <PERR-
>> 	Latency: 64, cache line size 10
>> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
>> 	Memory behind bridge: fc500000-fc7fffff
>> 	Prefetchable memory behind bridge: 
>>00000000eb000000-00000000fc000000
>> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>> 	Capabilities: [70] PCI-X bridge device.
>> 		Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
>> 		Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
>> 		: Upstream: Capacity=0, Commitment Limit=0
>> 		: Downstream: Capacity=0, Commitment Limit=0
>> 
>> and 02:00.0 says also about 66+ Mhz. Sorry, does it means that lspci 
>> -v output is wrong (has 2 digits only, for example), but lspci -vv 
>>is 
>> correct ?
>> 
>> BTW, is there some reasons to decrease PCI latency in BIOS setup 
>>from 
>> 64 (see lspci above) to some more low level ?
>
>Have you tested it to see how it performs? 

No, I just installed them physically. I'm waiting Mellanox answer 
about receiving of username/passwd for THCA driver and documentation.
  
>My Infiniband cards say the same thing.  They
>perform slow (400MB/s).    It is due to bad
>riser cards.  I am getting them replaced.

But I don't have riser cards installed ! BTW, do I understand you 
correctly that you didn't see 133 Mhz in any lspci output ?

    Yours
Mikhail