[Beowulf] BIG 'ram' using SSDs - was single machine with 500 GB of RAM

[Lux, Jim (337C)]

We do a fair amount of "giant capture" or "giant playback" kinds of applications with very large buffers, which is sort of the same kind of thing.
It's a pain, and it usually requires special purpose hardware of one sort or another. Sure, it's "off the shelf", but it's small volume and expensive, and often, not delivered from stock, but built to order.
Commercially, you see this sort of thing for radar signal simulation or recording (e.g. digitizing/playback several GHz bandwidth with 10-16 bits/sample, so you need 100 Gbps kinds of rates, sustained for seconds)
Schemes we've used are "gangs of disk drives in parallel streaming", "large arrays of RAM on purpose designed cards"

You also see solid state recorders used for radar or hyperspectral imaging instruments on spacecraft.  The general philosophy is to record all the raw data and post process it on the ground with typical rates in the hundreds of Mbps to tens of Gbps.  The Shuttle Radar Topographic Mission in 2000 used boxes and boxes full of streaming tapes, for instance, to record continuous data sets lasting hours. They had two recorders that overlapped, and astronauts swapped tapes about every 15 minutes. About 8.6Tbyte of data collected in 99 hours for the C-band radar on just over 200 tapes.  The hardware for that was basically 10 years or more old, so the 200+ Mbps throughput was probably pretty much state of the art for the mid 90s. DesDynI is looking at streaming Gbps sorts of data rates back to Earth.

Jim Lux


-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Ellis H. Wilson III
Sent: Wednesday, January 09, 2013 7:33 AM
To: beowulf at beowulf.org
Subject: Re: [Beowulf] BIG 'ram' using SSDs - was single machine with 500 GB of RAM

On 01/09/2013 08:27 AM, Vincent Diepeveen wrote:
> What would be a rather interesting thought for building a single box 
> dirt cheap with huge 'RAM'
> is the idea of having 1 fast RAID array of SSD's function as the 'RAM'.

This may be a more inexpensive route, but let's all note that the raw latency differences between DDR2/3 RAM and /any/ SSD is multiple orders of magnitude.  So for a single threaded application that has been asked to run on all RAM, I have a strong suspicion that RAM latencies are what it really does need -- not just reasonable latency and high throughput. 
  But we should await Jorg's response on the application nature to better flesh that out.

> You can get a bandwidth of 2 GB/s then to the SSD's "RAM" pretty 
> easily, yet for some calculations that bandwidth might be enough given 
> the fact you can then parallelize a few cores.

I am at a loss as to how you can achieve that high of bandwidth "pretty easily."  In the /absolute/ best case a single SATA SSD can serve reads at close to 400-500MB/s, and software RAIDing them will definitely not get you the 4x you speak of.  Moreover, most hardware RAID cards aren't built for that level of IOP/s or bandwidth since they usually have HDDs strapped to them, so you are going to have to get a very pricey HW RAID controller to try and achieve this.  And whether or not you are bottlenecked at the end of the day after the RAID controller is beyond my expertise.  Does anyone have ideas/suggestions on good RAID controllers for SSD specific workloads?

To get anywhere near the aforementioned number, you are likely going to have to drop a pretty penny on a Fusion-IO or Virident PCI-Express Flash Device -- not buy just any SSD and expect to take their quoted numbers and 4x them (much less get the quoted numbers).  I suspect going the RAM route may be comparably costly at this point.

> The random latency to such SSD raid might be 70 microseconds on 
> average, maybe even faster with specific SSD's.

Random read maybe -- certainly not random write latency.  Again, it's probably best to wait on Jorg to comment on the nature of the application to decide whether we should care about read or write latency more.

> One might need to optimize which file system gets used in such case 
> and the way how to access files - as one probably wants to avoid 
> global locks that avoid several requests pending simultaneously to the 
> SSD's.

You could instead just mmap the flash device directly if you really just want to use it truly like 'RAM.'  Optimizing filesystems is entirely nontrivial.

Best,

ellis
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf