[Beowulf] IPv6 and clusters?

[Beat Rubischon]

Hello!

On 09.06.11 15:32, Stuart Barkley wrote:
> Does IPv6 have any major place in clusters?

A good question which I asked myself many times. I'm operating my home
network dual stack since 2005 and preparing regularly clusters for
customers of my employer.

Basically most of the tools used in clusters are still IPv4. Staring
with PXE (DHCP, TFTP) and Management (IPMI). Going up to the job
schedulers, parallel filesystems, MPIs. OpenMPI is an exception as it
supports IPv6. Older interconnects are not IPv6 aware, Infiniband has
some troubles when using large MTUs on IPoIB to improve TCP throughput -
multicast drops are possible and multicast is essential for IPv6.

So the first answer is no. You have to run your cluster with an IPv4
network.

On the other hand there are many "cluster consolidations". People build
Grids, sharing resources over multiple machines. This invents usually
ugly networking hacks as the cluster networks are typically not routed.
In this specific area IPv6 could be a solution. Running a cluster dual
stack (private IPv4, public IPv6) would allow to interconnect clusters
and distribute work and data easily. The needed services (ssh,
Webservices) are IPv6 capable since the first days.

So the second answer is yes.

Still, I have no customer yet which was aware of the "new internet".
Even large universities are not using IPv6, their policies, hardware and
know how not ready. One case I know is a campus running IPv4 with a
complex firewall setup: They think about deploying IPv6, but the
stateless autoconfiguration and even more the privacy extensions are
creating so much addresses that their firewalls are not able to handle
the tables. And memory, at least with a C, F, or J brand, is pretty
expensive.

My conclusion about IPv6 in general: We have broken the internet in a
way that the deployment of IPv6 needs a major effort. 10 years back at
least academic networks were open and an additional protocol with the
design of IPv6 wouldn't be a problem. But today our setups are so
complex and firewalls such an essential part of the daily operation that
an "open protocol" like IPv6 creates a huge amount of headache.

Same is true for clusters. There are many lazy admins out there which
are operation their clusters on a "never touch a running system" policy.
In best case the headnode gets patches, compute nodes are often driven
by a complete outdated Linux installation. Opening a large amount of
identical systems to the world invents many questions in the security
area. Of course, it would be possible to firewall a cluster so no access
would be possible from outside. But in such a case the benefit of IPv6
and its open point to point architecture is lost. Nobody from the
management will be able to pay you for a migration.

So I'm still open and ready for an IPv6 deployment in a cluster. But I'm
also pretty disaffected and don't believe it will happen soon.

Beat

-- 
     \|/                           Beat Rubischon <beat at 0x1b.ch>
   ( 0-0 )                             http://www.0x1b.ch/~beat/
oOO--(_)--OOo---------------------------------------------------
Meine Erlebnisse, Gedanken und Traeume: http://www.0x1b.ch/blog/