[Beowulf] building a 96-core Ubuntu ARM solar-powered cluster

[Eugen Leitl]

(ob caveat phoronix)

http://www.phoronix.com/scan.php?page=article&item=mit_cluster_build&num=1


Building A 96-Core Ubuntu ARM Solar-Powered Cluster

Published on June 19, 2012

Written by Michael Larabel

Last week I shared results from the Phoronix 12-core ARM Linux mini cluster
that was constructed out of six PandaBoard ES development boards. Over the
weekend, a 96-core ARM cluster succeeded this build. While packing nearly 100
cores and running Ubuntu Linux, the power consumption was just a bit more
than 200 Watts. This array of nearly 100 processor cores was even powered up
by a solar panel.

This past weekend I was out at the Massachusetts Institute of Technology
(MIT) where this build took place. A massive ARM build out has been in the
plans for a few months and to even get it running off a solar panel. The
build was a success and by Sunday, the goals were realized.

Due to my past ARM Linux benchmarking on Phoronix that they have followed,
their use of the Phoronix Test Suite, and my experience with Linux
benchmarking and performance testing in general, I was invited over to MIT to
help with this 96-core ARM build after having collaborated with them for a
few months. This cluster / super-computer was built around 48 PandaBoards.

The bulk of the PandaBoards were not the ES model (I brought my collection of
PandaBoard ES models as back-ups for the PandaBoard nodes that failed), but
just the vanilla model. The non-ES model packs a Texas Instruments OMAP4430
with a dual-core 1.0GHz dual-core Cortex-A9 processor. The GPU and CPU of the
PandaBoard ES with its OMAP4460 are at higher clock speeds, but aside from
that it is very similar to the OMAP4430 model.

For maximum density and to make it easier to transport, the PandaBoards ended
up being stacked vertically. The enclosure for the 48 PandaBoards was an
industrial trashcan. Rather than using AC adapters, the PandaBoards were
running off a USB power source. The power consumption on the original
PandaBoard is similar to that of the PandaBoard ES or perhaps slightly lower
when using the more efficient USB power source. My PandaBoard ES testing
usually indicates about a 3 Watt idle per board, 5 Watt under load, or 6
Watts under extreme load. This MIT 96-core cluster would idle at just under
170 Watts and for the loads we hit it with over the weekend usually would
just go a bit above 200 Watts.

Overall, it was a fairly interesting weekend project! On the software side
was a stock Ubuntu 12.04 ARM OMAP4 installation across all 48 PandaBoards on
the SD cards.

As far as any benchmark results, MIT sent in some numbers for the Green500
and some other performance tests are still being worked out. From the
benchmarks I ran on the hardware, they dissented a bit from my expectations
based upon what I was achieving with my 12-core PandaBoard ES cluster, so for
the moment until all kinks in the new build are worked out I will refrain
from sharing any numbers. Such many-core ARM clusters though are showing
great potential in performance-per-Watt scenarios. For now, see my 12-core
ARM cluster results. I will also have more numbers on the way shortly from
the Phoronix build.

Over the weekend, there also was not much time for performance tuning. Ubuntu
12.10 presents some very impressive performance gains as Phoronix results
from earlier this month have indicated.

MIT will be putting out a video, a couple papers, and some other information
on this 96-core / 48 PandaBoard cluster so stay tuned for much more
information.