[Beowulf] Mixing different MTU settings on the same LAN?

Patrick Geoffray patrick at myri.com
Mon May 18 10:47:54 PDT 2009

Hi Carsten,

Carsten Aulbert wrote:
> I've run some early tests and these *seem* to suggest that the IP-layer
> takes correctly care of this (e.g. tcpdump shows that the maximum header
> length it 1514 bytes when the link is in use).

If you use TCP, the kernel will negotiate the Max Segment Size (MSS) at 
connect time. So, a device using a larger MTU will safely communicate 
with a device with a smaller MTU, as long as there is no switch/router 
in the middle that uses an MTU smaller than at both ends.

If you use UDP, or something directly on Ethernet at Layer-2, then the 
only options are to use VLANs with different MTUs, or to use a router in 
the middle.

> and others (netbsd-help page IIRC) suggests to separate these nodes from
> the main network via a router I'm a bit confused who is right.

Fragmentation is done at Layer 3, so you can use a router (Layer-3) 
instead of a switch (Layer-2) to isolate several LANs with different 
MTU. On the big->small MTU path, the router will fragment packets that 
are too large.

> Anyone here knowing much more on networking and can give me thumbs up or
> down?

Jumbo Frame does violate the Ethernet spec, but the vast majority of 
NICs support it and it's surprising to find some that do not in 2009.
If you are only using TCP, you can safely mix MTU, as long as all the 
switches support the largest MTU used. If you use other protocols than 
TCP, the best solution is probably to buy NICs than support Jumbo Frames 
for those machines. Decent GigE NICs run for $25, so it may be a no brainer.


More information about the Beowulf mailing list