[Beowulf] Pretty Big Data

[Perry E. Metzger]

On Mon, 25 Jan 2016 15:06:44 +0000 "Lux, Jim (337C)"
<james.p.lux at jpl.nasa.gov> wrote:
> Figure that RAM has a read cycle time of 1 ns (DDR3 or DDR4)

But not a *latency* of 1ns.

> High speed disk drives (in the spinning rust, sense) in ³commodity²
> kinds of speeds seem to be in the 7200 RPM range, which is a 4
> millisecond average latency. 

But everyone uses SSD at this point for any real apps, and generally
directly attached to the PCIe.

> So, if we¹re comparing searching a TB of data in RAM vs searching a
> TB of data on a disk, I think the RAM is always going to win if
> it¹s a sequential search.

Sure, but our original question was a sequential search of RAM vs. an
indexed search of mass storage.

The point I was making was that no amount of RAM will protect you
from sufficiently stupid data structure choices if the data is
big enough and you hit the data structures hard and often enough.

Perry
-- 
Perry E. Metzger		perry at piermont.com