[Beowulf] Update on dealing with Spectre and Meltdown

[Stephen Fralich]

I did do some testing with the original round of updates and I found the supplementary GPFS monitoring tools (mmsysmon) written in Python were having a profoundly negative affect on performance with the Spectre/Meltdown patches with even first pass things like Linpack. You can disable the supplementary monitoring though and that cleared up the performance issues I was seeing. I didn't get very deep into it before they retracted all the firmware updates and we decided to wait and see. IBM had talked about providing options to make the querying less intrusive and less frequent on the GPFS UG mailing list in newer versions of GPFS as well.

Stephen

________________________________________
From: Beowulf <beowulf-bounces at beowulf.org> on behalf of Ryan Novosielski <novosirj at rutgers.edu>
Sent: Thursday, March 8, 2018 12:55:34 PM
To: Beowulf List
Subject: Re: [Beowulf] Update on dealing with Spectre and Meltdown

I have been waiting, specifically, for Prentice Bisbal to try it. :-P

I actually have not checked to see whether any other upgrades I’ve applied have handled these vulnerabilities (I’d assume RedHat would have patched their distribution). FWIW, however, I did receive a report that one of our newer images seemed slower.  Haven’t gotten a chance to fully investigate those reports — could be any number of other things.

--
____
|| \\UTGERS,       |---------------------------*O*---------------------------
||_// the State     |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ     | Office of Advanced Research Computing - MSB C630, Newark
    `'

On Mar 8, 2018, at 13:13, Alex Chekholko via Beowulf <beowulf at beowulf.org<mailto:beowulf at beowulf.org>> wrote:

As an experienced and cynical HPC admin working with trusted users... I have not installed any kind of firmware or kernel updates.  I'll wait for you to go first :)

On Thu, Mar 8, 2018 at 10:10 AM, Prentice Bisbal <pbisbal at pppl.gov<mailto:pbisbal at pppl.gov>> wrote:
Beowulfers,

Have any of you updated the kernels on your clusters to fix the Spectre and Meltdown vulnerabilities? I was following this issue closely for the first couple of weeks. There seemed to be a lack of consensus on how much these fixed would impact HPC jobs, and if I recall correctly, some of the patches really hurt performance, or caused other problems. We took a wait-and-see approach here. So now that I've waited a while, what did you see?

--
Prentice

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.beowulf.org%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutgers.edu%7Cc4cd547515a346014d0d08d5852058a0%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636561296368291958&sdata=4fTJudM1x6k1L3KPRsLITJEy%2FKI%2FqaXOGJ6qwDJmgG4%3D&reserved=0>

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.beowulf.org%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutgers.edu%7Cc4cd547515a346014d0d08d5852058a0%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636561296368291958&sdata=4fTJudM1x6k1L3KPRsLITJEy%2FKI%2FqaXOGJ6qwDJmgG4%3D&reserved=0