[Beowulf] Update on dealing with Spectre and Meltdown

[Gavin W. Burris]

Hey, Ryan.  Doing well.  Hope you weathered the storm OK!

The benchmarks were inconclusive, as they were ran on a loaded system.  I should have ran a variety during the maintenance window, when queues were clear.  With so many different applications, many of which being network IO-bound, we decided to only enforce patches on login nodes, applying grub options on compute nodes.  Numbers there are good.

Cheers.

On Thu 03/08/18 02:47PM EST, Ryan Novosielski wrote:
> Cheers, Gavin (hope you're well, BTW)!
> 
> Any real-world ballpark figures on how "killed" performance is (and
> does your site disable those patches)?
> 
> On 03/08/2018 02:36 PM, Gavin W. Burris wrote:
> > Just had our biannual maintenance window.  All the things are 
> > patched.  One could disable the performance-killing patches with 
> > the "noibrs noibpb nopti" grub options.  Cheers.
> > 
> > On Thu 03/08/18 01:55PM EST, Ryan Novosielski wrote:
> >> I have been waiting, specifically, for Prentice Bisbal to try it.
> >> :-P
> >> 
> >> I actually have not checked to see whether any other upgrades 
> >> I’ve applied have handled these vulnerabilities (I’d assume 
> >> RedHat would have patched their distribution). FWIW, however, I 
> >> did receive a report that one of our newer images seemed slower.
> >>  Haven’t gotten a chance to fully investigate those reports — 
> >> could be any number of other things.
> >> 
> >> -- ____ || \\UTGERS, 
> >> |---------------------------*O*--------------------------- ||_//
> >>  the State     |         Ryan Novosielski - 
> >> novosirj at rutgers.edu<mailto:novosirj at rutgers.edu> || \\ 
> >> University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS 
> >> Campus ||  \\    of NJ     | Office of Advanced Research 
> >> Computing - MSB C630, Newark `'
> >> 
> >> On Mar 8, 2018, at 13:13, Alex Chekholko via Beowulf 
> >> <beowulf at beowulf.org<mailto:beowulf at beowulf.org>> wrote:
> >> 
> >> As an experienced and cynical HPC admin working with trusted 
> >> users... I have not installed any kind of firmware or kernel 
> >> updates.  I'll wait for you to go first :)
> >> 
> >> On Thu, Mar 8, 2018 at 10:10 AM, Prentice Bisbal 
> >> <pbisbal at pppl.gov<mailto:pbisbal at pppl.gov>> wrote: Beowulfers,
> >> 
> >> Have any of you updated the kernels on your clusters to fix the 
> >> Spectre and Meltdown vulnerabilities? I was following this issue
> >>  closely for the first couple of weeks. There seemed to be a lack
> >>  of consensus on how much these fixed would impact HPC jobs, and
> >>  if I recall correctly, some of the patches really hurt 
> >> performance, or caused other problems. We took a wait-and-see 
> >> approach here. So now that I've waited a while, what did you 
> >> see?
> >> 
> >> -- Prentice
> >> 
> >> _______________________________________________ Beowulf mailing 
> >> list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored 
> >> by Penguin Computing To change your subscription (digest mode or
> >>  unsubscribe) visit 
> >> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.b
> eowulf.org%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutg
> ers.edu%7Ca91c0d09885047db8ded08d5852beaa4%7Cb92d2b234d35447093ff69aca66
> 32ffe%7C1%7C1%7C636561346021256640&sdata=Eq4smdlMLwxS4nNkgAZlfDcc6Sl2gl3
> NvuRZrt3AIxI%3D&reserved=0<https://na01.safelinks.protection.outlook.com
> /?url=http%3A%2F%2Fwww.beowulf.org%2Fmailman%2Flistinfo%2Fbeowulf&data=0
> 2%7C01%7Cnovosirj%40rutgers.edu%7Cc4cd547515a346014d0d08d5852058a0%7Cb92
> d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636561296368291958&sdata=4fTJudM
> 1x6k1L3KPRsLITJEy%2FKI%2FqaXOGJ6qwDJmgG4%3D&reserved=0>
> >>
> >>
> >>
> >>
> >> 
> _______________________________________________
> >> Beowulf mailing list, 
> >> Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by 
> >> Penguin Computing To change your subscription (digest mode or 
> >> unsubscribe) visit 
> >> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.b
> eowulf.org%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutg
> ers.edu%7Cc4cd547515a346014d0d08d5852058a0%7Cb92d2b234d35447093ff69aca66
> 32ffe%7C1%7C0%7C636561296368291958&sdata=4fTJudM1x6k1L3KPRsLITJEy%2FKI%2
> FqaXOGJ6qwDJmgG4%3D&reserved=0
> >
> >>
> >>
> >> 
> _______________________________________________ Beowulf mailing
> >> list, Beowulf at beowulf.org sponsored by Penguin Computing To 
> >> change your subscription (digest mode or unsubscribe) visit 
> >> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.b
> eowulf.org%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutg
> ers.edu%7Ca91c0d09885047db8ded08d5852beaa4%7Cb92d2b234d35447093ff69aca66
> 32ffe%7C1%7C1%7C636561346021256640&sdata=Eq4smdlMLwxS4nNkgAZlfDcc6Sl2gl3
> NvuRZrt3AIxI%3D&reserved=0
> >
> >>
> >>
> >> 
> > 
> 
> -- 
>  ____
>  || \\UTGERS,     |----------------------*O*------------------------
>  ||_// the State  |    Ryan Novosielski - novosirj at rutgers.edu
>  || \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
>  ||  \\    of NJ  | Office of Advanced Res. Comp. - MSB C630, Newark
>       `'

-- 
Gavin W. Burris
Senior Project Leader for Research Computing
The Wharton School
University of Pennsylvania
Search our documentation: http://research-it.wharton.upenn.edu/about/
Subscribe to the Newsletter: http://whr.tn/ResearchNewsletterSubscribe