[Beowulf] BIG 'ram' using SSDs - was single machine with 500 GB of RAM

[Ellis H. Wilson III]

On 01/09/2013 08:27 AM, Vincent Diepeveen wrote:
> What would be a rather interesting thought for building a single box
> dirt cheap with huge 'RAM'
> is the idea of having 1 fast RAID array of SSD's function as the 'RAM'.

This may be a more inexpensive route, but let's all note that the raw 
latency differences between DDR2/3 RAM and /any/ SSD is multiple orders 
of magnitude.  So for a single threaded application that has been asked 
to run on all RAM, I have a strong suspicion that RAM latencies are what 
it really does need -- not just reasonable latency and high throughput. 
  But we should await Jorg's response on the application nature to 
better flesh that out.

> You can get a bandwidth of 2 GB/s then to the SSD's "RAM" pretty
> easily, yet for some calculations that bandwidth might be enough
> given the fact you can then parallelize a few cores.

I am at a loss as to how you can achieve that high of bandwidth "pretty 
easily."  In the /absolute/ best case a single SATA SSD can serve reads 
at close to 400-500MB/s, and software RAIDing them will definitely not 
get you the 4x you speak of.  Moreover, most hardware RAID cards aren't 
built for that level of IOP/s or bandwidth since they usually have HDDs 
strapped to them, so you are going to have to get a very pricey HW RAID 
controller to try and achieve this.  And whether or not you are 
bottlenecked at the end of the day after the RAID controller is beyond 
my expertise.  Does anyone have ideas/suggestions on good RAID 
controllers for SSD specific workloads?

To get anywhere near the aforementioned number, you are likely going to 
have to drop a pretty penny on a Fusion-IO or Virident PCI-Express Flash 
Device -- not buy just any SSD and expect to take their quoted numbers 
and 4x them (much less get the quoted numbers).  I suspect going the RAM 
route may be comparably costly at this point.

> The random latency to such SSD raid might be 70 microseconds on
> average, maybe even faster with specific SSD's.

Random read maybe -- certainly not random write latency.  Again, it's 
probably best to wait on Jorg to comment on the nature of the 
application to decide whether we should care about read or write latency 
more.

> One might need to optimize which file system gets used in such case
> and the way how to access files - as one probably wants to
> avoid global locks that avoid several requests pending simultaneously
> to the SSD's.

You could instead just mmap the flash device directly if you really just 
want to use it truly like 'RAM.'  Optimizing filesystems is entirely 
nontrivial.

Best,

ellis