[Beowulf] Pretty Big Data

[Perry E. Metzger]

On Sat, 23 Jan 2016 15:14:35 +0000 "Lux, Jim (337C)"
<james.p.lux at jpl.nasa.gov> wrote:
> Dumb sequential search in ram is probably faster than a fancy
> indexing scheme on disk.

Probably not. Eventually, log(n) vs n still bites you in the end if
you only get to a large enough data set. RAM will let you survive with
very stupid data structures only to a point.

Reading sequentially through a couple of terabytes of RAM can take a
very long time (relatively speaking) compared to a read or two from
SSD. Especially if you're doing this operation many many millions
of times you'll eventually feel quite serious pain.

The answer, of course, is to use a smart data structure in memory and
then you'll blow even an SSD every time. :)

Perry
-- 
Perry E. Metzger		perry at piermont.com