[Beowulf] Facebook's servers

[David Mathog]

Orion Poplawski <orion at cora.nwra.com> wrote
> 
> I guess not commodity (yet), but interesting.
> 

Thanks for posting that, it was interesting.  

The AMD board spec says it uses "address parity" memory, which I guess
implies ECC, since elsewhere in the spec it discusses handling ECC. They
implement reboot on lan (ROL, repurposing the WOL packet).

The Intel board spec memory section doesn't say anything about
parity/ECC, only that the memory must be registered, but again,
elsewhere it discusses logging ECC errors, so I guess ECC is assumed. 
This board also implements ROL.

The ROL sections seem to imply that both NICs will respond to such a
packet, which could have some "interesting" security implications, at
least for those of us where one interface is public.  I can imagine a
nightmare scenario where one machine is corrupted, it turns itself into
a DHCP server and starts spraying WOL packets out onto the network,
quickly converting more machines, which can carry on the same trick via
their private interfaces.  Normal WOL isn't nearly so hazardous on a
public interface, it only becomes a security risk if the attacker has
both access to another host on the subnet and some method that can
remotely force the attacked system to do an orderly shutdown. 

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech