[Beowulf] BIG 'ram' using SSDs - was single machine with 500 GB of RAM
Ellis H. Wilson III
ellis at cse.psu.edu
Wed Jan 9 07:33:22 PST 2013
On 01/09/2013 08:27 AM, Vincent Diepeveen wrote:
> What would be a rather interesting thought for building a single box
> dirt cheap with huge 'RAM'
> is the idea of having 1 fast RAID array of SSD's function as the 'RAM'.
This may be a more inexpensive route, but let's all note that the raw
latency differences between DDR2/3 RAM and /any/ SSD is multiple orders
of magnitude. So for a single threaded application that has been asked
to run on all RAM, I have a strong suspicion that RAM latencies are what
it really does need -- not just reasonable latency and high throughput.
But we should await Jorg's response on the application nature to
better flesh that out.
> You can get a bandwidth of 2 GB/s then to the SSD's "RAM" pretty
> easily, yet for some calculations that bandwidth might be enough
> given the fact you can then parallelize a few cores.
I am at a loss as to how you can achieve that high of bandwidth "pretty
easily." In the /absolute/ best case a single SATA SSD can serve reads
at close to 400-500MB/s, and software RAIDing them will definitely not
get you the 4x you speak of. Moreover, most hardware RAID cards aren't
built for that level of IOP/s or bandwidth since they usually have HDDs
strapped to them, so you are going to have to get a very pricey HW RAID
controller to try and achieve this. And whether or not you are
bottlenecked at the end of the day after the RAID controller is beyond
my expertise. Does anyone have ideas/suggestions on good RAID
controllers for SSD specific workloads?
To get anywhere near the aforementioned number, you are likely going to
have to drop a pretty penny on a Fusion-IO or Virident PCI-Express Flash
Device -- not buy just any SSD and expect to take their quoted numbers
and 4x them (much less get the quoted numbers). I suspect going the RAM
route may be comparably costly at this point.
> The random latency to such SSD raid might be 70 microseconds on
> average, maybe even faster with specific SSD's.
Random read maybe -- certainly not random write latency. Again, it's
probably best to wait on Jorg to comment on the nature of the
application to decide whether we should care about read or write latency
> One might need to optimize which file system gets used in such case
> and the way how to access files - as one probably wants to
> avoid global locks that avoid several requests pending simultaneously
> to the SSD's.
You could instead just mmap the flash device directly if you really just
want to use it truly like 'RAM.' Optimizing filesystems is entirely
More information about the Beowulf