[Beowulf] lspci, PCI-X 133, Mellanox

[René Grabner]

Hello,

the flag you see in lspci output is some kind of obsolete nowadays. It
says the card is capable running at 66MHz or higher, which is likely the
case when you plugged in the HCA without any risers.
Here, we have several quite well performing InfiniBand Clusters (e.g.
also on Tyan S28xx boards), where the HCA is running at 133 but lspci
still reports 66MHz.

Btw., you don't have to wait until the reception of username/passwd from
Mellanox. Just download the HPC-GOLD package from their webpage - it
contains all the stuff you need to get the IB fabric up and running
(driver,osm, mst, ipoib, 2 mpi impl.)

Try and have fun!

regards,
 Rene


On Sat, 2004-08-14 at 17:06, Mikhail Kuzminsky wrote:
> In message from Craig Tierney <ctierney at HPTI.com> (Fri, 13 Aug 2004 
> 14:06:17 -0600):
> >On Fri, 2004-08-13 at 13:09, Mikhail Kuzminsky wrote:
> >>    I just installed Mellanox InfiniHost MHXL-CF128-T (former 
> >> MTLP23108) NICs into Tyan S2880-based nodes (the location is 1st 
> >>PCI-X 
> >> 133 slot).
> >> lspci -v says about 66 Mhz:
> >> 
> >> 01:03.0 PCI bridge: Mellanox Technology: Unknown device 5a46 (rev 
> >>a1) 
> >> (prog-if 00 [Normal decode])
> >> 	Flags: bus master, 66Mhz, medium devsel, latency 64
> >> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
> >> 	Memory behind bridge: fc500000-fc7fffff
> >> 	Prefetchable memory behind bridge: 
> >>00000000eb000000-00000000fc000000
> >> 	Capabilities: [70] PCI-X bridge device.
> >> 
> >> 02:00.0 InfiniBand: Mellanox Technology: Unknown device 5a44 (rev 
> >>a1)
> >> 	Subsystem: Mellanox Technology: Unknown device 5a44
> >> 	Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 28
> >> 	Memory at fc700000 (64-bit, non-prefetchable) [size=1M]
> >> 	Memory at fb800000 (64-bit, prefetchable) [size=8M]
> >> 	Memory at f0000000 (64-bit, prefetchable) [size=128M]
> >> 	Capabilities: [40] #11 [001f]
> >> 	Capabilities: [50] Vital Product Data
> >> 	Capabilities: [60] Message Signalled Interrupts: 64bit+ Queue=0/5 
> >> Enable-
> >> 	Capabilities: [70] PCI-X non-bridge device.
> >> 
> >> I found that /usr/include/linux/pci.h knows about 133 Mhz PCI-X.
> >> 
> >> lspci -vv says:
> >> 
> >> 01:03.0 PCI bridge: Mellanox Technology: Unknown device 5a46 (rev 
> >>a1) 
> >> (prog-if 00 [Normal decode])
> >> 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- 
> >> Stepping- SERR+ FastB2B-
> >> 	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> >> <TAbort- <MAbort- >SERR- <PERR-
> >> 	Latency: 64, cache line size 10
> >> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
> >> 	Memory behind bridge: fc500000-fc7fffff
> >> 	Prefetchable memory behind bridge: 
> >>00000000eb000000-00000000fc000000
> >> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
> >> 	Capabilities: [70] PCI-X bridge device.
> >> 		Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
> >> 		Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
> >> 		: Upstream: Capacity=0, Commitment Limit=0
> >> 		: Downstream: Capacity=0, Commitment Limit=0
> >> 
> >> and 02:00.0 says also about 66+ Mhz. Sorry, does it means that lspci 
> >> -v output is wrong (has 2 digits only, for example), but lspci -vv 
> >>is 
> >> correct ?
> >> 
> >> BTW, is there some reasons to decrease PCI latency in BIOS setup 
> >>from 
> >> 64 (see lspci above) to some more low level ?
> >
> >Have you tested it to see how it performs? 
> 
> No, I just installed them physically. I'm waiting Mellanox answer 
> about receiving of username/passwd for THCA driver and documentation.
>   
> >My Infiniband cards say the same thing.  They
> >perform slow (400MB/s).    It is due to bad
> >riser cards.  I am getting them replaced.
> 
> But I don't have riser cards installed ! BTW, do I understand you 
> correctly that you didn't see 133 Mhz in any lspci output ?
> 
>     Yours
> Mikhail
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf