[Beowulf] Question: has anyone seen strange network timing bits on Nehalem/Westmere based motherboards?

[Tom Elken]

>   I seem to recall, though I can't remember where, seeing about 6-8
> months ago, reports of some strange network jitter for these platforms.
>   This is Nehalem/Westmere specific, didn't impact the earlier CPUs.
> For some reason, my memory is of a strange driver-NUMA connection, but
> I am not sure at this time.

Hi Joe,

Not sure if this is the issue, but for best latency benchmark performance, we recommend turning all C-states off in the BIOS.  If it cannot be done there, you can add the kernel boot option: processor.max_cstate=0
and reboot.

With C-states turned on, there is high and very variable MPI/IB latency measured if the benchmark is running on CPU0, the default, since extra OS interrupts are being serviced on CPU0.

Once you've turned them off,
You can look at these files:
    /proc/acpi/processor/*/power
to make sure that only state C1 is being used in the list of
states at the end.

I think we've also seen this affecting throughput benchmarks.  Again, I'm not sure if this affects GigE.

-Tom

>
>   Trying to trace something down, and gathering the background material
> first.  Gigabit ethernet, Intel NICs on the motherboard.  Single IOH.
>
>   Thanks!
>
> Joe
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman at scalableinformatics.com
> web  : http://scalableinformatics.com
>         http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf


This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.