[Beowulf] raw ethernet

[Robert G. Brown]

On Thu, 22 Jul 2004, Daniel Ridge wrote:

> Greetings,
> 
> On Jul 21, 2004, at 9:14 AM, Robert G. Brown wrote:
> 
> >  leeching off of arp
> > tables somehow to achieve a semi-portable mapping between hostname and
> > ethernet number on the flat network.
> 
> It's easy to take hardware too seriously. If you are building a cluster 
> with
> a private network, you don't need to worry about IANA and what they 
> think
> your Ethernet MAC addresses should be. Just staple your node number into
> your hardware address* and be done with it. Essentially all network 
> adapters
> provide some way to load a user-supplied MAC address into the card and
> nearly all linux network device drivers expose this facility.
> 
> * and make sure that the resulting ethernet address is a legit unicast 
> address
> 
> Regards,
> 	Dan Ridge

But

  a) Why invest all that time making a tool that will then be a priori
limited to a private network and which will break things horribly and
evilly if ever used (by accident or out of ignorance, say) on a
non-private network?  Things that are "safe" in the hands of experts are
very dangerous in the hands of novices who don't always even read all
the directions before trying something out...;-)

Robustness is good, and IANA compliance is clearly more robust.  To wit:

  b) One of the biggest network crashes in the triangle area occurred
back in the 80's (just after the Morris worm hit).  At that time the
internet was still flat across Duke and UNC and State -- there were
basically no IP routers on the multicampus backbone (which was largely
up via 56K twisted pair linking various equally flat ethernets), only
ethernet bridges.

Some bozo put a Shiva Fastpath (an ancient appletalk-to-IP router) on
the network and, using its nifty configuration utility, wrote a
broadcast mask into the MAC address of the fastpath.  Soon afterwards,
all the bridges learned that they could send packets for ANYBODY to the
Fastpath and it would cheerfully accept them.  I imagine that there is
still a weathered hide nailed to some barn door somewhere after they
skinned and salted the admin involved.

The moral of the story being that rewriting a MAC address is a perilous
venture and best not done, especially by a novice or anyone with less
that a perfectly clear idea of how everything works, lest you open a
metaphorical black hole in your network and make everybody really mad at
you.  Nowadays, sure, routers are cheap and common (he says with one
sitting behind his head in his home) but NOT violating IANA rules will
keep one out of all sorts of trouble.

  c) It isn't that hard to do correctly.  Even in the private network,
the nodes will still need IP.  They will still need, therefore, dhcp or
a similar tool to manage the MAC<->IP mapping at boot time.  They will
therefore have the requisite table in hand, and many, many ways of
extracting or distributing the requisite data. 

Perhaps the simplest and most portable is to use shell scripts
containing things like (from userspace):

rgb at uriel|B:1010>ping -c 1 archangel
PING archangel.rgb.private.net (192.168.1.133) 56(84) bytes of data.
64 bytes from archangel.rgb.private.net (192.168.1.133): icmp_seq=1
ttl=64 time=0.216 ms

--- archangel.rgb.private.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.216/0.216/0.216/0.000 ms
rgb at uriel|B:1011>arp archangel
Address HWtype HWaddress Flags Mask Iface
archangel.rgb.private.n ether 00:04:75:BC:87:C1 C eth0

(extracting the archangel<->00:04:75:BC:87:C1 map is left as a regexp
parsing exercise in perl, python, even bash.)

Or as root, do this in a single hit with arping:

[root at uriel rgb]# arping -c 1 -I eth0 192.168.1.134
ARPING 192.168.1.134 from 192.168.1.2 eth0
Unicast reply from 192.168.1.134 [00:04:75:EA:69:D4]  0.804ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

(which unfortunately doesn't seem to grok hostnames).

There must be a half-dozen more ways of doing it including low level
network calls.  It would probably take far more time to figure out how
to override the NIC MAC addresses and build semi-robust utilities for
doing so throughout a cluster.

   rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu