[Beowulf] Anyone have experience with Cisco 6509 switches for a cluster?

[Joe Landman]

Hi folks

   A partner is rebuilding a beowulf rendering/post-production cluster, 
and they swapped out some smaller switches for a larger Cisco 6509 
switch frame.  They are encountering some issues, asked us for help. 
Unfortunately, I know very little about these switches and IOS (not the 
Apple bit), so I am hoping to get a pointer to what we should be looking 
for.  If anyone is an IOS/Cisco expert that can help today, please 
contact me offlist.

   Here's the problem:

   iperf between 2 nodes, wire speed.  117 MB/s.

   iperf between 4 nodes, 1/2 wire speed, or 55 MB/s.

   As they increase the number of pairs doing iperf, performance keeps 
dropping.

   This suggests that all traffic is being serialized somehow, possibly 
transiting a single interface.

   Anyone out there ever see something like this before?  Any clues as 
to how to handle it?  Or how to diagnose/fix this?

   Thanks in advance.

Joe
-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615