[Beowulf] Update on dealing with Spectre and Meltdown

Fri Mar 16 12:16:30 PDT 2018

Thanks again, Gavin. We did the same thing at our site this week (mitigated on login nodes, disabled mitigation on compute nodes), but I intend to go back and test to see if there’s any performance penalty without the kernel arguments.

Am I correct that this is Haswell and newer that is affected?

--
____
|| \\UTGERS,       |---------------------------*O*---------------------------
||_// the State     |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ     | Office of Advanced Research Computing - MSB C630, Newark
    `'

On Mar 9, 2018, at 10:06, Gavin W. Burris <bug at wharton.upenn.edu<mailto:bug at wharton.upenn.edu>> wrote:

Hey, Ryan.  Doing well.  Hope you weathered the storm OK!

The benchmarks were inconclusive, as they were ran on a loaded system.  I should have ran a variety during the maintenance window, when queues were clear.  With so many different applications, many of which being network IO-bound, we decided to only enforce patches on login nodes, applying grub options on compute nodes.  Numbers there are good.

Cheers.

On Thu 03/08/18 02:47PM EST, Ryan Novosielski wrote:
Cheers, Gavin (hope you're well, BTW)!

Any real-world ballpark figures on how "killed" performance is (and
does your site disable those patches)?

On 03/08/2018 02:36 PM, Gavin W. Burris wrote:
Just had our biannual maintenance window.  All the things are
patched.  One could disable the performance-killing patches with
the "noibrs noibpb nopti" grub options.  Cheers.

On Thu 03/08/18 01:55PM EST, Ryan Novosielski wrote:
I have been waiting, specifically, for Prentice Bisbal to try it.
:-P

I actually have not checked to see whether any other upgrades
I’ve applied have handled these vulnerabilities (I’d assume
RedHat would have patched their distribution). FWIW, however, I
did receive a report that one of our newer images seemed slower.
Haven’t gotten a chance to fully investigate those reports —
could be any number of other things.

-- ____ || \\UTGERS,
|---------------------------*O*--------------------------- ||_//
the State     |         Ryan Novosielski -
novosirj at rutgers.edu<mailto:novosirj at rutgers.edu><mailto:novosirj at rutgers.edu> || \\
University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS
Campus ||  \\    of NJ     | Office of Advanced Research
Computing - MSB C630, Newark `'

On Mar 8, 2018, at 13:13, Alex Chekholko via Beowulf
<beowulf at beowulf.org<mailto:beowulf at beowulf.org><mailto:beowulf at beowulf.org>> wrote:

As an experienced and cynical HPC admin working with trusted
users... I have not installed any kind of firmware or kernel
updates.  I'll wait for you to go first :)

On Thu, Mar 8, 2018 at 10:10 AM, Prentice Bisbal
<pbisbal at pppl.gov<mailto:pbisbal at pppl.gov><mailto:pbisbal at pppl.gov>> wrote: Beowulfers,

Have any of you updated the kernels on your clusters to fix the
Spectre and Meltdown vulnerabilities? I was following this issue
closely for the first couple of weeks. There seemed to be a lack
of consensus on how much these fixed would impact HPC jobs, and
if I recall correctly, some of the patches really hurt
performance, or caused other problems. We took a wait-and-see
approach here. So now that I've waited a while, what did you
see?

-- Prentice

_______________________________________________ Beowulf mailing
list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org><mailto:Beowulf at beowulf.org> sponsored
by Penguin Computing To change your subscription (digest mode or
unsubscribe) visit
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.b
eowulf.org<http://eowulf.org>%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutg
ers.edu<http://ers.edu>%7Ca91c0d09885047db8ded08d5852beaa4%7Cb92d2b234d35447093ff69aca66
32ffe%7C1%7C1%7C636561346021256640&sdata=Eq4smdlMLwxS4nNkgAZlfDcc6Sl2gl3
NvuRZrt3AIxI%3D&reserved=0<https://na01.safelinks.protection.outlook.com
/?url=http%3A%2F%2Fwww.beowulf.org<http://2Fwww.beowulf.org>%2Fmailman%2Flistinfo%2Fbeowulf&data=0
2%7C01%7Cnovosirj%40rutgers.edu<http://40rutgers.edu>%7Cc4cd547515a346014d0d08d5852058a0%7Cb92
d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636561296368291958&sdata=4fTJudM
1x6k1L3KPRsLITJEy%2FKI%2FqaXOGJ6qwDJmgG4%3D&reserved=0>

_______________________________________________
Beowulf mailing list,
Beowulf at beowulf.org<mailto:Beowulf at beowulf.org><mailto:Beowulf at beowulf.org> sponsored by
Penguin Computing To change your subscription (digest mode or
unsubscribe) visit
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.b
eowulf.org<http://eowulf.org>%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutg
ers.edu<http://ers.edu>%7Cc4cd547515a346014d0d08d5852058a0%7Cb92d2b234d35447093ff69aca66
32ffe%7C1%7C0%7C636561296368291958&sdata=4fTJudM1x6k1L3KPRsLITJEy%2FKI%2
FqaXOGJ6qwDJmgG4%3D&reserved=0

_______________________________________________ Beowulf mailing
list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing To
change your subscription (digest mode or unsubscribe) visit
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.b
eowulf.org<http://eowulf.org>%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutg
ers.edu<http://ers.edu>%7Ca91c0d09885047db8ded08d5852beaa4%7Cb92d2b234d35447093ff69aca66
32ffe%7C1%7C1%7C636561346021256640&sdata=Eq4smdlMLwxS4nNkgAZlfDcc6Sl2gl3
NvuRZrt3AIxI%3D&reserved=0

--
____
|| \\UTGERS,     |----------------------*O*------------------------
||_// the State  |    Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Res. Comp. - MSB C630, Newark
     `'

--
Gavin W. Burris
Senior Project Leader for Research Computing
The Wharton School
University of Pennsylvania
Search our documentation: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fresearch-it.wharton.upenn.edu%2Fabout%2F&data=02%7C01%7Cnovosirj%40rutgers.edu%7C81518df46beb4ae2339808d585cf437d%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C1%7C636562047597701062&sdata=SyA6EIUeAjIH9oZsQXkZOVAA03ImAy607mr7Zt6nBRQ%3D&reserved=0
Subscribe to the Newsletter: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwhr.tn%2FResearchNewsletterSubscribe&data=02%7C01%7Cnovosirj%40rutgers.edu%7C81518df46beb4ae2339808d585cf437d%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C1%7C636562047597701062&sdata=UUC35NEU2oNL4O%2FPkcBaeiaZqpUy0g9OYtIQuB%2FlGD4%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20180316/970854ad/attachment-0001.html>