[Beowulf] i7-4770R 128MB L4 cache CPU in compact 0.79 litre box - DIY cluster?

[Robin Whittle]

In the Beowulf tradition, here are some ideas about how one might go
about making a cluster with off-the-shelf consumer products, with a few
potential advantages compared to using traditional PC cases and
motherboards: compactness, lower power consumption and ease of running
from a 24 volt lead-acid battery system for un-interruptable and/or
solar power.

This would probably not be as good for some applications as using a
smaller number of large motherboards with two or four AMD Opteron G34
socket devices with potentially very large amounts of RAM, where each
device has 8, 12 or 16 cores.  Where the bottleneck is inter-process
communications, this enables 32 to 64 cores to work in a shared memory
environment or, even if they are not sharing memory, for their processes
to communicate faster than by Ethernet or I guess by Infiniband networks.

Also, the following approach is still arguably amateur in that it
doesn't use ECC memory.  I am thinking if someone (not yet me) had a $5k
to $10k and wanted to build their own cluster.  The total cost looks
like not much more than $250 per 3.2GHz 22um Haswell core, each
supporting two threads.

Intel's i7-4770R is a BGA (soldered to the PCB) Haswell 22nm CPU with 4
cores, hyper threading (so it can support 8 processes), 256kB L2 cache
per core (as with all recent Intel CPUs, I think) and 6MB of L3 cache.
Most i7 CPUs have 8MB or more L3 cache.

http://ark.intel.com/products/76642/Intel-Core-i7-4770R-Processor-6M-Cache-up-to-3_90-GHz

The base clock frequency is 3.2GHz, with turbo mode to 3.9GHz.

The i7-4770R is intended for compact consumer desktop devices and has a
fancy graphics GPU on chip, which I guess is of little interest for HPC.
 It is compact (no socket) and relatively inexpensive (I guess, since it
is for cost-sensitive consumer products):


http://ark.intel.com/products/76642/Intel-Core-i7-4770R-Processor-6M-Cache-up-to-3_90-GHz

The price quoted is $392.  Maximum memory is 32MB, via two channels of
DDR3L-1333/1600.

The most unusual thing about it, which I assume is of interest for HPC,
is that in the same package, there is a 128MB L4 cache chip.  There are
6 other mobile and desktop CPUs with this "Crystal Well" L5 cache chip:

  http://ark.intel.com/products/codename/51802/Crystal-Well

Of these, the i7-4770R has the highest base clock frequency.  The
i5-4570R (2.7GHz base clock frequency) is used in some iMacs.

According to:

  http://en.wikipedia.org/wiki/Haswell_%28microarchitecture%29

the 128MB L4 cache functions as a "victim cache" to the CPU's L3 cache:

  http://en.wikipedia.org/wiki/CPU_cache#Victim_cache

It is a DRAM cache, rather than SRAM.  I haven't researched it any
further, but I guess this extra cache would help with at least some HPC
applications.  These may be photos of this or similar dual chip devices:


http://vr-zone.com/articles/meet-haswells-graphics-trump-card-crystalwell-gt3-edram/19604.html

http://www.pcgameshardware.de/CPU-Hardware-154106/News/IDF-2013-Intel-Core-i7-4770R-und-Iris-Pro-Grafik-im-0-72l-PC-Gigabyte-Brix-ausprobiert-1088170/galerie/2093650/

http://www.bit-tech.net/hardware/2013/06/01/intel-core-i7-4770k-cpu-review/1


According to:

  http://en.wikipedia.org/wiki/Intel_HD_and_Iris_Graphics
  http://blog.ffwll.ch/2013/09/neat-drmi915-stuff-for-312.html

the 128MB cache is supported by Linux kernel 3.12 (Nov 2013).

Another new feature is on-die voltage regulation (FIVR = Fully
Integrated Voltage Regulation) which reduces motherboard complexity:


http://hothardware.com/News/Haswell-Takes-A-Major-Step-Forward-Integrates-Voltage-Regulator/


The next question is how to get a bunch of PCs with this chip, in as
compact a form as possible.  There will no doubt be many sources, but at
present, I found this from Gigabyte - the Brix Pro Ultra Compact PC kit:
GB-BXi7-4770R.  "Kit" means it needs RAM, hard drive, operating system,
keyboard, trackball and monitor.

   http://www.gigabyte.com/products/product-page.aspx?pid=4888#ov
   http://www.anandtech.com/show/7648/gigabyte-brix-pro
   http://techreport.com/news/25377/gigabyte-brix-crams-gaming-pc-into-well

http://www.pcper.com/news/General-Tech/CES-2014-Gigabytes-New-SFF-BRIX-Pro-Comes-Iris-Pro-5200-Graphics


The box is 6.2 x 11.2 x 11.5 cm = 0.79 litres.  It has gigabit Ethernet
and two SO-DIMM DDR3L slots with a maximum of 16GB.  The memory speed is
quoted as 1333/1600 MHz, but the memory compatibility list indicates
that the 8GB G-Skill F3-1866C10D-16GRSL runs at 1866MHz.

The power supply is an external unit which provides 19 volts at 7.1
amps.  This means that it would be straightforward to power a bunch of
these devices via switching buck regulators (there are plenty of
adjustable regulators on eBay), one per Brix, from two 12 volt lead acid
batteries in series.  I think this or a conventional UPS is essential
for any HPC work, at least with the frequency of brief power outages
where I live (a north-eastern suburb of Melbourne, Australia).

The power supply and some of the innards are shown in the pcper.com page
mentioned above.  There is a fan on the CPU, but I haven't yet seen a
photo of the CPU heatsink.  There are some longevity questions about
fans.  Their bearings can wear out unless they are ball bearings - which
are generally more expensive and noisier.  Heatsinks and the fins of
centrifugal blowers can clog with dust, but presumably a cluster could
be housed somewhere with minimal dust.  Here are some photos of the
centrifugal blowers of other Brix models - not the 4770R model, which is
in a taller box and I guess has a larger heatsink and blower:

  http://semiaccurate.com/2013/09/10/gigabytes-brix/

http://www.kitguru.net/desktop-pc/base-unit/luke-hill/gigabyte-brix-i7-4500-barebones-mini-pc-review-w-g-skill-ripjaws-1600mhz-16gb/3/

http://www.computeraudiophile.com/attachments/f10-music-servers/9524d1387360616-intel-next-unit-computing-music-server-38_gigabyte_gb-xm12-3227.jpg

The last one shows a Delta Electronics BSB05505HP centrifugal blower,
which can be bought elsewhere.  I am not sure if it is a ball bearing
blower, but some or many of the Delta Electronics blowers are.

Looking at:


http://cdn.pcper.com/files/news/2014-01-06/Gigabyte%20BRIX%20Pro%20SFF%20PC.png

it seems the heatsink of the 4770R model is thicker and that the PC uses
two widely spaced PCBs with the fan in between them, rather than in the
above photo which shows the fan on the underside, I guess, of two
closely-spaced PCBs.  So the fan or blower of the 4770R model may be
totally different to those in the shorter models, which have CPUs which
dissipate less power.


The total cost in the USA, in USD$, not counting local taxes and
shipping would be:

   ~$650  Brix Pro GB-BXi7-4770R
   ~$260  2 x 8GB RAM @~$130
   ~$100  120GB SSD drive

~= $1010

All that remains is to add some Ethernet cables and an Ethernet switch
and maybe some fans for the power supply farm.  If the machines booted
via the network it may not be necessary to have a hard drive.

Ordinary mass market PC cases, motherboards and i7 CPUs might work out a
little cheaper, but they would not be as compact, would not have the
128MB L4 cache (but would have 8MB L3, rather than this 6MB) and would
probably chew more power.

 - Robin