[Beowulf] zram usage

[Ellis H. Wilson III]

On 05/01/2014 05:35 PM, Prentice Bisbal wrote:
> On 05/01/2014 05:07 PM, Ellis H. Wilson III wrote:
>> On 05/01/2014 03:57 PM, Prentice Bisbal wrote:
>>> On 05/01/2014 03:43 PM, Joe Landman wrote:
>>>> On 05/01/2014 03:33 PM, atchley tds.net wrote:
>>>>> Yes, but I don't want to buy a Dell just to get a NVMe unit. ;-)
>>>>
>>>> Not that I am trying to get the dude a Dell ... but think of it as
>>>> very expensive shrink wrap packaging ... :^
>>> On the bright side, a Dell box will be easier to open than a plastic
>>> clam-shell package.
>>
>> Dell is the only one shipping the Samsung XS1715 to the best of my
>> knowledge.  Please correct me if there is some other place which is.
>> Samsung's the only one on the market with anything providing NVMe
>> right now.  Should change within the year though.
>>
>> But, as a word of caution, let's all beat this into our heads now:
>> Flash is not RAM.  Flash is not RAM.  Flash is not RAM.  Not even slow
>> RAM. (Ok, NAND Flash is not slow RAM -- maybe you could get away with
>> saying NOR flash is slow RAM).  If it ain't byte-addressable, it ain't
>> RAM.
>
> Despite being the one who stated us down the NVMe path, I get this

I wasn't suggesting NVMe was inherently a bad direction to go.  I just 
always feel compelled to provide these warnings, as I did with you when 
we talked last.  There is a lot of pressure in academia to just find a 
way to make NAND act like RAM, and its a giant square hole round peg 
situation.  People are incredibly resistant when I suggest to them that 
maybe we should stop thinking about computing as black and white "main 
memory" and "persistent storage."  I.e., maybe there is something 
in-between, and we need to give it first-class citizenship, rethink how 
systems should leverage that, and actually build it in rather than just 
write papers about it.

> warning. How does UltraDIMM work, then? Have they found a way to
> byte-address flash RAM, or are they using some sort of translation layer

Buffers on buffers, as I expected.  Hide the real NAND access patterns 
behind bunches of RAM.  Per:

http://www.extremetech.com/computing/162944-diablos-memory-channel-storage-tech-will-deliver-terabytes-of-ram-using-nand-flash

"Even the fastest NAND flash is far behind conventional DRAM as far as 
read/write speeds are concerned — how does Memory Channel Storage get 
around that?

Through the intelligent use of buffers. Here, Diablo Technologies was 
cagey and clearly didn’t want to give too much away, but it told us that 
each TeraDIMM contains a DRAM buffer of unspecified size. The company’s 
patent filings confirm this, and imply that the amount of RAM per DIMM 
can vary depending on the wishes of the manufacturer. Stacking 
additional RAM on the TeraDIMM allows the controller to mask access 
latencies and perform write cycles in the background."

If you've got uniformly random tiny writes going across the entire 
space, which RAM wouldn't care about and where RAM shines compared to 
everything else, UltraDIMM et. al. are going to look fast for a bit and 
then jump off a performance cliff -- because it catches up with them.

> as a kludge? As I understand it, you could put it in a regular DIMM
> slot, but I could be wrong about that.

It does go in a regular DIMM slot.  But yes, kludge, hammers, the works. 
  Still damn fast if you use it intelligently, but don't expect it to 
just jump into RAM's place and work great.

Best,

ellis

-- 
Ph.D. Candidate
Department of Computer Science and Engineering
The Pennsylvania State University
www.ellisv3.com