[Beowulf] Ethernet connected drives

[Lockwood, Glenn]

On May 8, 2014, at 10:30 AM, Ellis H. Wilson III <ellis at cse.psu.edu> wrote:

> On 05/08/2014 10:29 AM, John Hearns wrote:
>> Forget building compute clusters - soon we will be building Beowulfs
>> with disk drives!
> 
> Color me dubious.  I highly doubt there will be any entire clusters of just HDDs anytime soon.  The cpu/ram you can fit on them will be far lower than a full machine, even if you consider 16 of them or so.

This is the end goal of Hadoop clusters.  Not everything needs fast CPUs and a ton of RAM.  

> What these will be good at (and what the Active Disk research espouses) is applying very simple filters (e.g., simple greps) to avoid pushing data across buses needlessly.


Yes, and there are a number of data-oriented devices in the works (not just drives) that capitalize upon this idea and treat the CPU as just a means to do ET in ETL before data leaves the storage device.  Being able to run an SQL query directly on a disk has a lot of benefits.

Is this still a beowulf cluster?  Probably not, but these sorts of devices have a lot of utility in HPC.  We make heavy use of iSCSI on our largest machine, and we'd free up a fair amount of resources if we didn't have to wrap our iSCSI targets in iSCSI servers.

Glenn