[Beowulf] [External] Re: Clustering vs Hadoop/spark [EXT]

[Jonathan Aquilina]

I think a perfect example of what you are describing below is Minecraft. I haven’t take a dive into the mechanics of how it works code wise but it is an interesting game for sure but at times can be clunky unfortunately.

Regards,
Jonathan

From: Tim Cutts <tjrc at sanger.ac.uk>
Sent: 25 November 2020 13:04
To: Jonathan Aquilina <jaquilina at eagleeyet.net>
Cc: Prentice Bisbal <pbisbal at pppl.gov>; Beowulf <beowulf at beowulf.org>
Subject: Re: [Beowulf] [External] Re: Clustering vs Hadoop/spark [EXT]

It depends on how long the object lives.  If it’s still being used after a certain number of garbage collection cycles, it gets moved to tenured space, and will never be garbage collected until the tenured space is full, i.e. you’ve used everything you gave Java with -Xmx.  OK, it’s not a real memory leak, but it causes the java application to use more memory than it needs to.

There is a genuine memory leak in the code space that I’ve seen before; some code does something I don’t understand to do with re-allocated classes, and that does cause a genuine memory leak that can never be garbage collected at all.  I don’t remember the details though, and never understood it when I did remember them.

Tim




On 25 Nov 2020, at 11:51, Jonathan Aquilina <jaquilina at eagleeyet.net<mailto:jaquilina at eagleeyet.net>> wrote:

Tim correct me if I am wrong here.

Memory is allocated for as long as the object is being called or do you mean the memory is always allocated regardless of the object being in scope and that part of the programme being used or not?

Regards,
Jonathan

From: Beowulf <beowulf-bounces at beowulf.org<mailto:beowulf-bounces at beowulf.org>> On Behalf Of Tim Cutts
Sent: 25 November 2020 12:40
To: Prentice Bisbal <pbisbal at pppl.gov<mailto:pbisbal at pppl.gov>>
Cc: Beowulf <beowulf at beowulf.org<mailto:beowulf at beowulf.org>>
Subject: Re: [Beowulf] [External] Re: Clustering vs Hadoop/spark [EXT]

Except of course, you do really.  Java applications can end up with huge memory leaks because the programmers really need to understand the mechanism when objects get moved from Eden and Survivor space into Tenured space.

Tenured space never decreases, so every object which ends up there is allocated memory for the life of the process.  If that was actually an ephemeral object, tough, you’ve lost the memory.

So ideally, the programmer needs to understand the size of their pile of ephemeral objects and make sure the Eden/Survivor spaces are large enough, otherwise they will have a memory leak.  If they make the ephemeral spaces too large, though, garbage collection takes more time and performance decreases, so it’s a balance.

I’m not sure you can ever actually really get rid of the need to understand memory allocation… you just push the problem around!

Tim

On 24 Nov 2020, at 18:32, Prentice Bisbal via Beowulf <beowulf at beowulf.org<mailto:beowulf at beowulf.org>> wrote:

Also, with Java, you don't have to worry about low-level issues like freeing and allocating memory and doing pointer arithmetic. Not having to worry about those low-level issues allows a student to focus more on the programming concepts. I knew I screw up dereferencing pointers A LOT when learning C and C++

-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20201125/4e26a6e1/attachment-0001.htm>