[Beowulf] anyone using 10gbaseT?

[Joe Landman]

FWIW:

Bruce Allen wrote:
> Hi Thomas,
> 
> If I understand correctly, your NFS server is connected via a 10Gb/s 
> link to the switch, and you've spent some effort to tune it.  What sort 
> of aggregate NFS performance are you seeing?  Is it at the level of 300 
> or 400 MG/sec?  Could you please provide a few details about the 
> hardware in your NFS box?

Our 15 and 16 disk JackRabbit units have no trouble with this, using 
channel bonded gigabit (no 10GbE).  We measured with 4 simultaneous 
IOzones being run over NFS and an inexpensive gigabit switch.  No jumbo 
frames, the fingers never left the hands, yadda yadda yadda.  Atop 
reported the network IO rate, dstat didn't curiously enough ... the 
individual adapters were correct, but it got the channel bond throughput 
calculation wrong (multiplied the sum of the individual adapters by 4). 
  vmstat also returned the disk IO rate.

We were seeing a sustained 420 MB/s in this configuration.  About 500 
MB/s to disk (journaling overhead).  Used RAID6.

FWIW we also uncovered some nasty channel bond crashes under intense 
loads.  We were able to get it stable and repeatable, at the cost of 
some of the more interesting channel bond modes.  These crashes didn't 
manifest themselves until we started pushing the channel bond hard.  We 
needed 4 clients banging on the bonded server.  3 would not crash it. 
These crashes manifested themselves throughout all kernels we tried 
(2.6.9 through 2.6.18).  Running 4 separate (non-channel bonded) ports 
and doing the same tests did not show this crash.  The driver was fine 
(e1000, don't even think of trying this with a Broadcom chip/driver) and 
well behaved.

> 
> Cheers,
>     Bruce
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615