From rouge2507 at gmail.com Sun Mar 1 11:39:14 2015 From: rouge2507 at gmail.com (Ivan Rossi) Date: Sun, 1 Mar 2015 20:39:14 +0100 Subject: [Beowulf] Beowulf Digest, Vol 132, Issue 25 In-Reply-To: References: Message-ID: I think you should take a look at ansible (http://www.ansible.com/) or salt stack (http://saltstack.com/community/) Both are open-source project, widely used in the cloud community for provisioning and configuration management. Both can be used as simple parallel shells, but they also implement higher-level idempotent commands that allow you to do complicated things in an easier and safer way. Large and vibrant communities too. Ivan 2015-02-28 21:00 GMT+01:00 : > Send Beowulf mailing list submissions to > beowulf at beowulf.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://www.beowulf.org/mailman/listinfo/beowulf > or, via email, send a message with subject or body 'help' to > beowulf-request at beowulf.org > > You can reach the person managing the list at > beowulf-owner at beowulf.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Beowulf digest..." > > > Today's Topics: > > 1. Replacement for C3 suite from ORNL (Fabricio Cannini) > 2. Re: Replacement for C3 suite from ORNL (Skylar Thompson) > 3. Re: Replacement for C3 suite from ORNL (Ashley Pittman) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 27 Feb 2015 19:34:14 -0300 > From: Fabricio Cannini > To: beowulf at beowulf.org > Subject: [Beowulf] Replacement for C3 suite from ORNL > Message-ID: <54F0F0E6.40808 at gmail.com> > Content-Type: text/plain; charset=windows-1252; format=flowed > > Hello all > > Does anybody know of a replacement for the C3 suite of commands ( cpower > ...) that is very common in SGI machines and used to be available here: > > http://www.csm.ornl.gov/torc/C3/ > > TIA, > Fabricio > > > ------------------------------ > > Message: 2 > Date: Sat, 28 Feb 2015 09:00:56 -0800 > From: Skylar Thompson > To: beowulf at beowulf.org, fcannini at gmail.com > Subject: Re: [Beowulf] Replacement for C3 suite from ORNL > Message-ID: <54F1F448.9080904 at gmail.com> > Content-Type: text/plain; charset=windows-1252 > > On 02/27/2015 02:34 PM, Fabricio Cannini wrote: >> Hello all >> >> Does anybody know of a replacement for the C3 suite of commands ( cpower >> ...) that is very common in SGI machines and used to be available here: >> >> http://www.csm.ornl.gov/torc/C3/ > > We've been using pdsh: > > https://code.google.com/p/pdsh/ > > Skylar > > > > ------------------------------ > > Message: 3 > Date: Sat, 28 Feb 2015 19:31:42 +0000 > From: Ashley Pittman > To: Skylar Thompson > Cc: beowulf at beowulf.org > Subject: Re: [Beowulf] Replacement for C3 suite from ORNL > Message-ID: <45DC4292-8F86-4349-857F-3509A30AB549 at pittman.co.uk> > Content-Type: text/plain; charset=utf-8 > > >> On 28 Feb 2015, at 17:00, Skylar Thompson wrote: >> >> On 02/27/2015 02:34 PM, Fabricio Cannini wrote: >>> Hello all >>> >>> Does anybody know of a replacement for the C3 suite of commands ( cpower >>> ...) that is very common in SGI machines and used to be available here: >>> >>> http://www.csm.ornl.gov/torc/C3/ >> >> We've been using pdsh: >> >> https://code.google.com/p/pdsh/ > > I?m a long-time user of pdsh although more recently I?ve been looking at clush which has a lot more options and a python interface should you need it. > > https://github.com/cea-hpc/clustershell/wiki/clush > > Ashley, > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Beowulf mailing list > Beowulf at beowulf.org > http://www.beowulf.org/mailman/listinfo/beowulf > > > ------------------------------ > > End of Beowulf Digest, Vol 132, Issue 25 > **************************************** From samuel at unimelb.edu.au Sun Mar 1 14:43:19 2015 From: samuel at unimelb.edu.au (Christopher Samuel) Date: Mon, 02 Mar 2015 09:43:19 +1100 Subject: [Beowulf] Replacement for C3 suite from ORNL In-Reply-To: <54F0F0E6.40808@gmail.com> References: <54F0F0E6.40808@gmail.com> Message-ID: <54F39607.2020109@unimelb.edu.au> On 28/02/15 09:34, Fabricio Cannini wrote: > Does anybody know of a replacement for the C3 suite of commands ( cpower > ...) that is very common in SGI machines and used to be available here: That link seems to work for me, were you having problems getting to it? We use xCAT for our HPC systems, and we run that on SGI (their rebadged SuperMicro gear) as well as IBM systems. Provides rpower, reventlog, rinv, rsetboot, etc, as well as xdsh, xdcp and the like. http://xcat.sourceforge.net/ Just note that the latest release has dropped support for RHEL/CentOS 5. http://sourceforge.net/p/xcat/wiki/XCAT_2.9_Release_Notes/ All the best! Chris -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci From hearnsj at googlemail.com Mon Mar 2 01:30:57 2015 From: hearnsj at googlemail.com (John Hearns) Date: Mon, 2 Mar 2015 09:30:57 +0000 Subject: [Beowulf] Replacement for C3 suite from ORNL In-Reply-To: <54F39607.2020109@unimelb.edu.au> References: <54F0F0E6.40808@gmail.com> <54F39607.2020109@unimelb.edu.au> Message-ID: I have found the C3 tools to be very useful on SGI kit in the past. I was looking around this morning for tools to look at mcelog entries. and came across OpenLMI http://www.openlmi.org Has anyone done much with this? On 1 March 2015 at 22:43, Christopher Samuel wrote: > On 28/02/15 09:34, Fabricio Cannini wrote: > > > Does anybody know of a replacement for the C3 suite of commands ( cpower > > ...) that is very common in SGI machines and used to be available here: > > That link seems to work for me, were you having problems getting to it? > > We use xCAT for our HPC systems, and we run that on SGI (their rebadged > SuperMicro gear) as well as IBM systems. > > Provides rpower, reventlog, rinv, rsetboot, etc, as well as xdsh, xdcp > and the like. > > http://xcat.sourceforge.net/ > > Just note that the latest release has dropped support for RHEL/CentOS 5. > > http://sourceforge.net/p/xcat/wiki/XCAT_2.9_Release_Notes/ > > All the best! > Chris > -- > Christopher Samuel Senior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.org.au/ http://twitter.com/vlsci > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fcannini at gmail.com Tue Mar 3 07:16:38 2015 From: fcannini at gmail.com (Fabricio Cannini) Date: Tue, 03 Mar 2015 12:16:38 -0300 Subject: [Beowulf] Replacement for C3 suite from ORNL In-Reply-To: <54F39607.2020109@unimelb.edu.au> References: <54F0F0E6.40808@gmail.com> <54F39607.2020109@unimelb.edu.au> Message-ID: <54F5D056.6010307@gmail.com> On 01-03-2015 19:43, Christopher Samuel wrote: > On 28/02/15 09:34, Fabricio Cannini wrote: > >> Does anybody know of a replacement for the C3 suite of commands ( cpower >> ...) that is very common in SGI machines and used to be available here: > > That link seems to work for me, were you having problems getting to it? Yes, here from .br all I have is a blank page when accessing the url. Maybe a route problem ? > We use xCAT for our HPC systems, and we run that on SGI (their rebadged > SuperMicro gear) as well as IBM systems. > > Provides rpower, reventlog, rinv, rsetboot, etc, as well as xdsh, xdcp > and the like. > > http://xcat.sourceforge.net/ xcat seems overkill because of its deployment capabilities, but it may fit the bill better than the others. > Just note that the latest release has dropped support for RHEL/CentOS 5. > > http://sourceforge.net/p/xcat/wiki/XCAT_2.9_Release_Notes/ I'm dying to leave the 6.x series, so this shouldn't be a problem. ;) Thanks! From fcannini at gmail.com Tue Mar 3 07:26:38 2015 From: fcannini at gmail.com (Fabricio Cannini) Date: Tue, 03 Mar 2015 12:26:38 -0300 Subject: [Beowulf] Replacement for C3 suite from ORNL In-Reply-To: References: <54F0F0E6.40808@gmail.com> <54F39607.2020109@unimelb.edu.au> Message-ID: <54F5D2AE.9050906@gmail.com> On 02-03-2015 06:30, John Hearns wrote: > I have found the C3 tools to be very useful on SGI kit in the past. > > I was looking around this morning for tools to look at mcelog entries. > and came across OpenLMI http://www.openlmi.org > > Has anyone done much with this? Seems interesting, but couldn't it conflict with the cluster's configuration system ( rocks, xcat, sgi mc, oscar, puppet/chef/ansible ... ) ? [ ]'s From fcannini at gmail.com Tue Mar 3 07:49:12 2015 From: fcannini at gmail.com (Fabricio Cannini) Date: Tue, 03 Mar 2015 12:49:12 -0300 Subject: [Beowulf] Replacement for C3 suite from ORNL In-Reply-To: <45DC4292-8F86-4349-857F-3509A30AB549@pittman.co.uk> References: <54F0F0E6.40808@gmail.com> <54F1F448.9080904@gmail.com> <45DC4292-8F86-4349-857F-3509A30AB549@pittman.co.uk> Message-ID: <54F5D7F8.8000207@gmail.com> On 28-02-2015 16:31, Ashley Pittman wrote: > >> On 28 Feb 2015, at 17:00, Skylar Thompson wrote: >> >> On 02/27/2015 02:34 PM, Fabricio Cannini wrote: >>> Hello all >>> >>> Does anybody know of a replacement for the C3 suite of commands ( cpower >>> ...) that is very common in SGI machines and used to be available here: >>> >>> http://www.csm.ornl.gov/torc/C3/ >> >> We've been using pdsh: >> >> https://code.google.com/p/pdsh/ > > I?m a long-time user of pdsh although more recently I?ve been looking at clush which has a lot more options and a python interface should you need it. > > https://github.com/cea-hpc/clustershell/wiki/clush Python API is always welcome. :) Clush seems to fall short of what C3 does, but I'll check it out anyway. Thnaks! [ ]'s From prentice.bisbal at rutgers.edu Tue Mar 3 08:04:52 2015 From: prentice.bisbal at rutgers.edu (Prentice Bisbal) Date: Tue, 03 Mar 2015 11:04:52 -0500 Subject: [Beowulf] Replacement for C3 suite from ORNL In-Reply-To: <54F5D056.6010307@gmail.com> References: <54F0F0E6.40808@gmail.com> <54F39607.2020109@unimelb.edu.au> <54F5D056.6010307@gmail.com> Message-ID: <54F5DBA4.7080200@rutgers.edu> On 03/03/2015 10:16 AM, Fabricio Cannini wrote: > On 01-03-2015 19:43, Christopher Samuel wrote: >> On 28/02/15 09:34, Fabricio Cannini wrote: >> >>> Does anybody know of a replacement for the C3 suite of commands ( >>> cpower >>> ...) that is very common in SGI machines and used to be available here: >> >> That link seems to work for me, were you having problems getting to it? > > > Yes, here from .br all I have is a blank page when accessing the url. > Maybe a route problem ? I was able to see the site download the source for the commands if from here in the US the other day. From cbergstrom at pathscale.com Wed Mar 4 11:07:28 2015 From: cbergstrom at pathscale.com (=?UTF-8?B?QyBCZXJnc3Ryw7Zt?=) Date: Thu, 5 Mar 2015 02:07:28 +0700 Subject: [Beowulf] What happened to AMD GPU? Message-ID: This is part shameless self promotion, but also meant to stir some discussion.. Take a look at some of the recent top results for SPEC ACCEL http://spec.org/accel/results/accel_acc.html /* It may not seem like much, but I can say that is the visible result to a heck of a lot of work */ I have to word this carefully, but after that was submitted we fixed a cooling issue and the score went higher.. expect better public results in 2 weeks. (Anyone want to party at GTC?) Frankly, it's visible we still aren't the best performance on some codes.. Those issues basically fall into two camps and we're working on it. ----------- Having said this.. Would anyone build an AMD GPU cluster if 1) The GPU was 30-40% faster than a K40 2) It cost less 3) OpenACC/OpenMP4/CUDA programming models were high performance and supported 4) More oncard ram ----------- The s9150 is a top end card and priced relatively in the same ballpark as the competition.. Forgive me for sounding like a salesperson. If you don't need all the ram on the card - there's the w8100 (with fan on the card) - $1050 on Newegg! (Why do I feel like an infomercial) I have 2x w8100 in a server for benchmarking and they are nice GPU. Not as fast as a K40, but beats other cards. From mdidomenico4 at gmail.com Wed Mar 4 11:26:39 2015 From: mdidomenico4 at gmail.com (Michael Di Domenico) Date: Wed, 4 Mar 2015 14:26:39 -0500 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: On Wed, Mar 4, 2015 at 2:07 PM, C Bergstr?m wrote: > Frankly, it's visible we still aren't the best performance on some > codes.. Those issues basically fall into two camps and we're working > on it. > ----------- > Having said this.. > Would anyone build an AMD GPU cluster if > 1) The GPU was 30-40% faster than a K40 > 2) It cost less > 3) OpenACC/OpenMP4/CUDA programming models were high performance and supported > 4) More oncard ram > ----------- if the AMD's showed a 30-40% increase in performance over a k40, i'd buy them. but our lock in with CUDA would make it difficult... From mfatica at gmail.com Wed Mar 4 11:33:39 2015 From: mfatica at gmail.com (Massimiliano Fatica) Date: Wed, 4 Mar 2015 11:33:39 -0800 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: Looking at the list: FirePro s9150 has a score of 2.89/2.99 while the K40c has a score of 2.98/3.15. Where is the 30-40% advantage? M On Wed, Mar 4, 2015 at 11:26 AM, Michael Di Domenico wrote: > On Wed, Mar 4, 2015 at 2:07 PM, C Bergstr?m > wrote: > > Frankly, it's visible we still aren't the best performance on some > > codes.. Those issues basically fall into two camps and we're working > > on it. > > ----------- > > Having said this.. > > Would anyone build an AMD GPU cluster if > > 1) The GPU was 30-40% faster than a K40 > > 2) It cost less > > 3) OpenACC/OpenMP4/CUDA programming models were high performance and > supported > > 4) More oncard ram > > ----------- > > if the AMD's showed a 30-40% increase in performance over a k40, i'd > buy them. but our lock in with CUDA would make it difficult... > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cbergstrom at pathscale.com Wed Mar 4 11:38:27 2015 From: cbergstrom at pathscale.com (=?UTF-8?B?QyBCZXJnc3Ryw7Zt?=) Date: Thu, 5 Mar 2015 02:38:27 +0700 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: On Thu, Mar 5, 2015 at 2:33 AM, Massimiliano Fatica wrote: > Looking at the list: > FirePro s9150 has a score of 2.89/2.99 while the K40c has a score of > 2.98/3.15. > Where is the 30-40% advantage? How to say this without putting much mud on my face... I said we have 2 issues in the compiler where we're working to fix. The benchmarks where AMD is slower isn't their fault. The geomean on those 6 benchmarks drags the numbers down. Look at the performance for the other benchmarks to get an idea of where it should be.. From prentice.bisbal at rutgers.edu Wed Mar 4 11:53:32 2015 From: prentice.bisbal at rutgers.edu (Prentice Bisbal) Date: Wed, 04 Mar 2015 14:53:32 -0500 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: <54F762BC.9000805@rutgers.edu> The top system on the Green500 right now is using AMD Fire S9150 GPUs: http://www.green500.org/lists/green201411 While you might be promoting yourself, I am glad someone brought this up, because I've been wondering what AMD is going to do about this, and I just had a conversation about this last week. I think the best thing you can do to improve market share would be to license the CUDA syntax from NVidia, or work to make it a standardized language. Sure there are open standard languages like OpenACC, but CUDA got there first and is so well entrenched, it's safe to say it's the de facto standard, and will probably retroactively become an official standard. I don't see OpenACC really taking off and supplanting CUDA. The fact that your processors are different under the hood won't matter if a researcher can take their existing CUDA source code, and just recompile to to run on AMD GPUs. (Almost) Everyone in HPC already compiles their code all the time anyway, Prentice On 03/04/2015 02:07 PM, C Bergstr?m wrote: > This is part shameless self promotion, but also meant to stir some discussion.. > > Take a look at some of the recent top results for SPEC ACCEL > http://spec.org/accel/results/accel_acc.html > /* It may not seem like much, but I can say that is the visible result > to a heck of a lot of work */ > > I have to word this carefully, but after that was submitted we fixed a > cooling issue and the score went higher.. expect better public results > in 2 weeks. (Anyone want to party at GTC?) > > Frankly, it's visible we still aren't the best performance on some > codes.. Those issues basically fall into two camps and we're working > on it. > ----------- > Having said this.. > Would anyone build an AMD GPU cluster if > 1) The GPU was 30-40% faster than a K40 > 2) It cost less > 3) OpenACC/OpenMP4/CUDA programming models were high performance and supported > 4) More oncard ram > ----------- > The s9150 is a top end card and priced relatively in the same ballpark > as the competition.. > > Forgive me for sounding like a salesperson. > If you don't need all the ram on the card - there's the w8100 (with > fan on the card) - $1050 on Newegg! (Why do I feel like an > infomercial) > > I have 2x w8100 in a server for benchmarking and they are nice GPU. > Not as fast as a K40, but beats other cards. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From prentice.bisbal at rutgers.edu Wed Mar 4 11:59:26 2015 From: prentice.bisbal at rutgers.edu (Prentice Bisbal) Date: Wed, 04 Mar 2015 14:59:26 -0500 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: <54F7641E.1030601@rutgers.edu> M, You're reading his original e-mail wrong. He didn't say AMD GPUs had a 30-40% advantage over NVidia. He's asking IF people would switch to AMD if that were true. Personally, I think it would have to be much more than 30-40%. The next NVidia card would probably provide that kind of improvement, so most NVidia owners would just wait for the next generation of NVidia product to come out instead of switching to another processor and port their code. How much better? I'd say at least 100%. You can't just have better performance, you need an overwhelming performance advantage to convince people who are already using CUDA to switch. 30-40% isn't going to do it. If you can't do that, you have to go after new users who haven't started using CUDA yet, provide them with a value alternative to NVidia. Since a lot of public codes already support CUDA, you'd have to go after users writing their own code, preferably at the start of a project who haven't already invested in CUDA. Prentice On 03/04/2015 02:33 PM, Massimiliano Fatica wrote: > Looking at the list: > FirePro s9150 has a score of 2.89/2.99 while the K40c has a score of > 2.98/3.15. > Where is the 30-40% advantage? > > M > > On Wed, Mar 4, 2015 at 11:26 AM, Michael Di Domenico > > wrote: > > On Wed, Mar 4, 2015 at 2:07 PM, C Bergstr?m > > wrote: > > Frankly, it's visible we still aren't the best performance on some > > codes.. Those issues basically fall into two camps and we're working > > on it. > > ----------- > > Having said this.. > > Would anyone build an AMD GPU cluster if > > 1) The GPU was 30-40% faster than a K40 > > 2) It cost less > > 3) OpenACC/OpenMP4/CUDA programming models were high performance > and supported > > 4) More oncard ram > > ----------- > > if the AMD's showed a 30-40% increase in performance over a k40, i'd > buy them. but our lock in with CUDA would make it difficult... > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -------------- next part -------------- An HTML attachment was scrubbed... URL: From cbergstrom at pathscale.com Wed Mar 4 12:06:56 2015 From: cbergstrom at pathscale.com (=?UTF-8?B?QyBCZXJnc3Ryw7Zt?=) Date: Thu, 5 Mar 2015 03:06:56 +0700 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: <54F762BC.9000805@rutgers.edu> References: <54F762BC.9000805@rutgers.edu> Message-ID: On Thu, Mar 5, 2015 at 2:53 AM, Prentice Bisbal wrote: > The top system on the Green500 right now is using AMD Fire S9150 GPUs: > > http://www.green500.org/lists/green201411 > > While you might be promoting yourself, I am glad someone brought this up, > because I've been wondering what AMD is going to do about this, and I just > had a conversation about this last week. > > I think the best thing you can do to improve market share would be to > license the CUDA syntax from NVidia, or work to make it a standardized > language. I am unaware of any patents in CUDA or that CUDA depends on. We did everything 100% clean and our GPGPU stack is written entirely by us. CUDA is likely a trademark of NVIDIA, but that's the most complicated thing to marketing a "CUDA compatible compiler"... I don't think it's only a 30% additional margin of performance.. I think it's a combined set of things which broaden the scope. 1) For N budget how many cards can you buy.. In this case I think you're either saving money or getting more cards.. (which may make up that magic 100% target) 2) Do you need more ram on the card? If yes.. they provide it.. if not.. you can get sweet deals on cards like the w8100 Clusters typically have a longer lifecycle, but at some point it does make sense to upgrade them. I think if the AMD solution catches that sweetspot some existing users could be converted (assuming their code will run relatively pain free) - That's my goal... --------- OpenACC and other directive based approaches - we're working to fix the performance issues with that in our compiler and write a best practices guide.. CUDA provides some really explicit mapping mechanisms for the GPU, but I hope we can make the general case a lot more compelling.. From craig.tierney at noaa.gov Wed Mar 4 12:10:54 2015 From: craig.tierney at noaa.gov (Craig Tierney - NOAA Affiliate) Date: Wed, 4 Mar 2015 13:10:54 -0700 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: It appears to me that the numbers posted on that page for the card you are testing are with ECC off? I know you are asking the question "what if", but the current test isn't even apples-to-apples. We want best price-performance for our codes. We have not gone down the road of CUDA because that would lock us out of other technologies (still hoping that OpenACC and OpenMP merge). We have never ruled out AMD, but we have never seen an argument to try it. Craig On Wed, Mar 4, 2015 at 12:07 PM, C Bergstr?m wrote: > This is part shameless self promotion, but also meant to stir some > discussion.. > > Take a look at some of the recent top results for SPEC ACCEL > http://spec.org/accel/results/accel_acc.html > /* It may not seem like much, but I can say that is the visible result > to a heck of a lot of work */ > > I have to word this carefully, but after that was submitted we fixed a > cooling issue and the score went higher.. expect better public results > in 2 weeks. (Anyone want to party at GTC?) > > Frankly, it's visible we still aren't the best performance on some > codes.. Those issues basically fall into two camps and we're working > on it. > ----------- > Having said this.. > Would anyone build an AMD GPU cluster if > 1) The GPU was 30-40% faster than a K40 > 2) It cost less > 3) OpenACC/OpenMP4/CUDA programming models were high performance and > supported > 4) More oncard ram > ----------- > The s9150 is a top end card and priced relatively in the same ballpark > as the competition.. > > Forgive me for sounding like a salesperson. > If you don't need all the ram on the card - there's the w8100 (with > fan on the card) - $1050 on Newegg! (Why do I feel like an > infomercial) > > I have 2x w8100 in a server for benchmarking and they are nice GPU. > Not as fast as a K40, but beats other cards. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From deadline at eadline.org Wed Mar 4 12:15:45 2015 From: deadline at eadline.org (Douglas Eadline) Date: Wed, 4 Mar 2015 15:15:45 -0500 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: <79f593e16b97f6c813b86daf778ac950.squirrel@mail.eadline.org> The one thing NVidia did right was port/support popular applications to CUDA (Amber comes to mind). I don't see how AMD can compete in this space if they don't do the same thing, but do it for their APU Opteron-X processors using HSA (should they come out with some heftier models) -- Doug > This is part shameless self promotion, but also meant to stir some > discussion.. > > Take a look at some of the recent top results for SPEC ACCEL > http://spec.org/accel/results/accel_acc.html > /* It may not seem like much, but I can say that is the visible result > to a heck of a lot of work */ > > I have to word this carefully, but after that was submitted we fixed a > cooling issue and the score went higher.. expect better public results > in 2 weeks. (Anyone want to party at GTC?) > > Frankly, it's visible we still aren't the best performance on some > codes.. Those issues basically fall into two camps and we're working > on it. > ----------- > Having said this.. > Would anyone build an AMD GPU cluster if > 1) The GPU was 30-40% faster than a K40 > 2) It cost less > 3) OpenACC/OpenMP4/CUDA programming models were high performance and > supported > 4) More oncard ram > ----------- > The s9150 is a top end card and priced relatively in the same ballpark > as the competition.. > > Forgive me for sounding like a salesperson. > If you don't need all the ram on the card - there's the w8100 (with > fan on the card) - $1050 on Newegg! (Why do I feel like an > infomercial) > > I have 2x w8100 in a server for benchmarking and they are nice GPU. > Not as fast as a K40, but beats other cards. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -- > Mailscanner: Clean > -- Doug -- Mailscanner: Clean From cbergstrom at pathscale.com Wed Mar 4 12:22:06 2015 From: cbergstrom at pathscale.com (=?UTF-8?B?QyBCZXJnc3Ryw7Zt?=) Date: Thu, 5 Mar 2015 03:22:06 +0700 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: <79f593e16b97f6c813b86daf778ac950.squirrel@mail.eadline.org> References: <79f593e16b97f6c813b86daf778ac950.squirrel@mail.eadline.org> Message-ID: On Thu, Mar 5, 2015 at 3:15 AM, Douglas Eadline wrote: > > > The one thing NVidia did right was port/support popular > applications to CUDA (Amber comes to mind). I don't see > how AMD can compete in this space if they don't do the > same thing, but do it for their APU Opteron-X processors > using HSA (should they come out with some heftier models) Small clarification - I do believe there will eventually be HPC capable APU, but "HSA" is an overloaded term. It can refer to multiple things. In some cases it's hardware and other cases it's software. Not all of HSA software was designed (or suitable) for HPC. From cbergstrom at pathscale.com Wed Mar 4 12:26:37 2015 From: cbergstrom at pathscale.com (=?UTF-8?B?QyBCZXJnc3Ryw7Zt?=) Date: Thu, 5 Mar 2015 03:26:37 +0700 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: On Thu, Mar 5, 2015 at 3:10 AM, Craig Tierney - NOAA Affiliate wrote: > > It appears to me that the numbers posted on that page for the card you are > testing are with ECC off? I know you are asking the question "what if", but > the current test isn't even apples-to-apples. SPEC does allow you 1:1 comparisons. In this case we're not yet showing the gains I know we can achieve. I'm mostly trying to stir the pot to see the level of interest. Here's NVIDIA's best published result http://spec.org/accel/results/res2014q1/accel-20140303-00018.html compared to ours http://spec.org/accel/results/res2015q1/accel-20150218-00045.html The specific Intel CPU is less a factor if you're concerned about that. I could put this card in the exact same system NVIDIA used and show some decent performance. (That 3.8Ghz boost in fact may help more than anything) From prentice.bisbal at rutgers.edu Wed Mar 4 12:32:40 2015 From: prentice.bisbal at rutgers.edu (Prentice Bisbal) Date: Wed, 04 Mar 2015 15:32:40 -0500 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: <79f593e16b97f6c813b86daf778ac950.squirrel@mail.eadline.org> References: <79f593e16b97f6c813b86daf778ac950.squirrel@mail.eadline.org> Message-ID: <54F76BE8.3010708@rutgers.edu> Doug, Thanks for bringing this up. I forgot to mention this in one of my earlier posts. If AMD created their own version of NVidia's CUDA Centers of Excellence (CCoE) or Intel's Parallel Computing Centers (IPCC), that would go far in reducing the main barrier to adoption of AMD GPUs. I believe one of the goals of these centers is to port popular applications to the respective architectures. https://developer.nvidia.com/cuda_centers_ccoe https://software.intel.com/en-us/ipcc -- Prentice On 03/04/2015 03:15 PM, Douglas Eadline wrote: > > The one thing NVidia did right was port/support popular > applications to CUDA (Amber comes to mind). I don't see > how AMD can compete in this space if they don't do the > same thing, but do it for their APU Opteron-X processors > using HSA (should they come out with some heftier models) > > -- > Doug > > >> This is part shameless self promotion, but also meant to stir some >> discussion.. >> >> Take a look at some of the recent top results for SPEC ACCEL >> http://spec.org/accel/results/accel_acc.html >> /* It may not seem like much, but I can say that is the visible result >> to a heck of a lot of work */ >> >> I have to word this carefully, but after that was submitted we fixed a >> cooling issue and the score went higher.. expect better public results >> in 2 weeks. (Anyone want to party at GTC?) >> >> Frankly, it's visible we still aren't the best performance on some >> codes.. Those issues basically fall into two camps and we're working >> on it. >> ----------- >> Having said this.. >> Would anyone build an AMD GPU cluster if >> 1) The GPU was 30-40% faster than a K40 >> 2) It cost less >> 3) OpenACC/OpenMP4/CUDA programming models were high performance and >> supported >> 4) More oncard ram >> ----------- >> The s9150 is a top end card and priced relatively in the same ballpark >> as the competition.. >> >> Forgive me for sounding like a salesperson. >> If you don't need all the ram on the card - there's the w8100 (with >> fan on the card) - $1050 on Newegg! (Why do I feel like an >> infomercial) >> >> I have 2x w8100 in a server for benchmarking and they are nice GPU. >> Not as fast as a K40, but beats other cards. >> _______________________________________________ >> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> -- >> Mailscanner: Clean >> > > -- > Doug > From deadline at eadline.org Wed Mar 4 13:03:40 2015 From: deadline at eadline.org (Douglas Eadline) Date: Wed, 4 Mar 2015 16:03:40 -0500 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: <79f593e16b97f6c813b86daf778ac950.squirrel@mail.eadline.org> Message-ID: <96e06e48c1c179bb9e6851f97177cdfa.squirrel@mail.eadline.org> > On Thu, Mar 5, 2015 at 3:15 AM, Douglas Eadline > wrote: >> >> >> The one thing NVidia did right was port/support popular >> applications to CUDA (Amber comes to mind). I don't see >> how AMD can compete in this space if they don't do the >> same thing, but do it for their APU Opteron-X processors >> using HSA (should they come out with some heftier models) > > Small clarification - > I do believe there will eventually be HPC capable APU, but "HSA" is an > overloaded term. It can refer to multiple things. In some cases it's > hardware and other cases it's software. > > Not all of HSA software was designed (or suitable) for HPC. thanks for that clarification -- Doug > > -- > Mailscanner: Clean > -- Doug -- Mailscanner: Clean From mfatica at gmail.com Wed Mar 4 14:18:18 2015 From: mfatica at gmail.com (Massimiliano Fatica) Date: Wed, 4 Mar 2015 14:18:18 -0800 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: I would not draw too many conclusions, the SpecAcc is just telling you the quality of the OpenACC compiler and the quality of the porting. For example, if you look at the results for CloverLeaf ( I am familiar with this application and have other reference points), you have: AMD/Pathscale: 3.13 specaccel_peak NVIDIA/PGI: 3.45 specaccel_peak Keeping the HW constant and changing the software ( adding CUDA C and CUDA Fortran to the mix) will give you for the 3840x3840 grid the following average times per cell (measured in 10^-8s): OpenACC loops: 1.92 OpenACC kernels: 1.78 CUDA Fortran; 1.33 CUDA C: 1.25 Timing is on a K20c, but we are interested in the relative performance. Cuda C/Fortran in 30% faster. There is also an OpenCL implementation of CloverLeaf but I don't have the results. It is probably in the same ballpark. This is a "simple" CFD code with regular access pattern, a directive base porting gives you decent results. You could try to run the OpenCL code on the AMD card and see how far the Pathscale compiler is from it, but I am expecting something similar. OpenACC is an interesting option for people looking for high level programming, but you usually pay a penalty. How big is the penalty will depend on a lot of factors and it is very difficult to generalize. M On Wed, Mar 4, 2015 at 12:26 PM, C Bergstr?m wrote: > On Thu, Mar 5, 2015 at 3:10 AM, Craig Tierney - NOAA Affiliate > wrote: > > > > It appears to me that the numbers posted on that page for the card you > are > > testing are with ECC off? I know you are asking the question "what if", > but > > the current test isn't even apples-to-apples. > > SPEC does allow you 1:1 comparisons. In this case we're not yet > showing the gains I know we can achieve. I'm mostly trying to stir the > pot to see the level of interest. > > Here's NVIDIA's best published result > http://spec.org/accel/results/res2014q1/accel-20140303-00018.html > compared to ours > http://spec.org/accel/results/res2015q1/accel-20150218-00045.html > > The specific Intel CPU is less a factor if you're concerned about > that. I could put this card in the exact same system NVIDIA used and > show some decent performance. (That 3.8Ghz boost in fact may help more > than anything) > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cbergstrom at pathscale.com Wed Mar 4 14:29:40 2015 From: cbergstrom at pathscale.com (=?UTF-8?B?QyBCZXJnc3Ryw7Zt?=) Date: Thu, 5 Mar 2015 05:29:40 +0700 Subject: [Beowulf] What happened to AMD GPU? In-Reply-To: References: Message-ID: On Thu, Mar 5, 2015 at 5:18 AM, Massimiliano Fatica wrote: > I would not draw too many conclusions, the SpecAcc is just telling you the > quality of the OpenACC compiler and the quality of the porting. > For example, if you look at the results for CloverLeaf ( I am familiar with > this application and have other reference points), you have: > AMD/Pathscale: 3.13 specaccel_peak > NVIDIA/PGI: 3.45 specaccel_peak To state it again - our compiler is not perfect. There's a couple things blocking us from hitting numbers 4+ in certain benchmarks. > > > Keeping the HW constant and changing the software ( adding CUDA C and CUDA > Fortran to the mix) will give you > for the 3840x3840 grid the following average times per cell (measured in > 10^-8s): > OpenACC loops: 1.92 > OpenACC kernels: 1.78 > CUDA Fortran; 1.33 > CUDA C: 1.25 I would not compare PGI OpenACC to CUDA and draw a conclusion that OpenACC is bound to lose. If we beat PGI OpenACC by 30% that difference starts to narrow quickly. > > Timing is on a K20c, but we are interested in the relative performance. Cuda > C/Fortran in 30% faster. > There is also an OpenCL implementation of CloverLeaf but I don't have the > results. It is probably in the same ballpark. > This is a "simple" CFD code with regular access pattern, a directive base > porting gives you decent results. > You could try to run the OpenCL code on the AMD card and see how far the > Pathscale compiler is from it, but I am > expecting something similar. > > OpenACC is an interesting option for people looking for high level > programming, but you usually pay a penalty. > How big is the penalty will depend on a lot of factors and it is very > difficult to generalize. I think with poorly written CUDA or poorly written OpenACC you'll pay a penalty in both cases. I think with good OpenACC and a good compiler (after we fix some bugs) - that general perception will start to narrow. (Yes highly tuned CUDA will probably always win, but by how much) The thing to keep in mind is that in our compiler, unlike every other implementation - we are not doing any source-to-source or dumping byte-code. 1) Our code generator targets bare metal instructions 2) It's optimized for HPC - not just a recycled shader compiler Our GPU transformations *know* the hardware and how to map the right grid sizes to the resources underneath. When that mapping is done correctly and in combination with good old code generation == win. From cbergstrom at pathscale.com Tue Mar 10 09:13:18 2015 From: cbergstrom at pathscale.com (=?UTF-8?B?QyBCZXJnc3Ryw7Zt?=) Date: Tue, 10 Mar 2015 23:13:18 +0700 Subject: [Beowulf] [OT] Power8+GPU access Message-ID: Hi.. It's a longshot, but anyone have a Power8 system (preferably with a GPU attached) that I could get access to before the end of the week? We support Power7/BGQ, but that's BE and I'm looking to build a Power8 release on a LE system. (I'm a US citizen if that makes it any easier) Thanks ./C From fcannini at gmail.com Tue Mar 10 09:37:30 2015 From: fcannini at gmail.com (Fabricio Cannini) Date: Tue, 10 Mar 2015 13:37:30 -0300 Subject: [Beowulf] [OT] Power8+GPU access In-Reply-To: References: Message-ID: <54FF1DCA.1070904@gmail.com> On 10-03-2015 13:13, C Bergstr?m wrote: > Hi.. > > It's a longshot, but anyone have a Power8 system (preferably with a > GPU attached) that I could get access to before the end of the week? > We support Power7/BGQ, but that's BE and I'm looking to build a Power8 > release on a LE system. > > (I'm a US citizen if that makes it any easier) > > Thanks Hi there You can request access here (not sure there are gpus available, but worth a try anyway ) : http://openpower.ic.unicamp.br/minicloud/ From cbergstrom at pathscale.com Tue Mar 10 09:44:17 2015 From: cbergstrom at pathscale.com (=?UTF-8?B?QyBCZXJnc3Ryw7Zt?=) Date: Tue, 10 Mar 2015 23:44:17 +0700 Subject: [Beowulf] [OT] Power8+GPU access In-Reply-To: <54FF1DCA.1070904@gmail.com> References: <54FF1DCA.1070904@gmail.com> Message-ID: On Tue, Mar 10, 2015 at 11:37 PM, Fabricio Cannini wrote: > On 10-03-2015 13:13, C Bergstr?m wrote: >> >> Hi.. >> >> It's a longshot, but anyone have a Power8 system (preferably with a >> GPU attached) that I could get access to before the end of the week? >> We support Power7/BGQ, but that's BE and I'm looking to build a Power8 >> release on a LE system. >> >> (I'm a US citizen if that makes it any easier) >> >> Thanks > > > Hi there > > You can request access here (not sure there are gpus available, but worth a > try anyway ) : > > http://openpower.ic.unicamp.br/minicloud/ Thanks for the heads up. Request submitted From hahn at mcmaster.ca Tue Mar 10 12:42:45 2015 From: hahn at mcmaster.ca (Mark Hahn) Date: Tue, 10 Mar 2015 15:42:45 -0400 (EDT) Subject: [Beowulf] Xeon D systems? (and 10G in general) Message-ID: Intel recently introduced an interesting product: Xeon D is a Broadwell (Haswell shrink) SoC. It only has 8 cores, not high-clocked and only 2 dimm channels, so it's definitely not at the same level of fat-node goodness as an e5-26xx v3. But for 45W, you also get 2x onboard 10Gb! Anyone working on an HPC system based on these quite compact building blocks? the SoC also has stuff like PCIe and SATA, which is why a lot of the coverage is calling it a chip for desktop NAS, etc. But for HPC purposes, the CPU is quite decent, memory balance is reasonable, and it's hard to argue with two free 10G... On that topic, I've read some work recently on performance tuning of Intel 10G, but not in an HPC context. Is 10G still sucking for MPI latency? (SFP+ DA noticably better than 10GbT?) If you're thinking of saying "why bother with an x86_64 SoC when you can get a 64b Atom SoC", well, can you? (for cheap, at commodity volume, etc...) Do any of the surviving Atom SoCs still have onboard multiport switching fabrics? thanks, Mark Hahn. From hearnsj at googlemail.com Wed Mar 11 02:52:35 2015 From: hearnsj at googlemail.com (John Hearns) Date: Wed, 11 Mar 2015 09:52:35 +0000 Subject: [Beowulf] Xeon D systems? (and 10G in general) In-Reply-To: References: Message-ID: Very good article on The Platform: http://www.theplatform.net/2015/03/09/intel-crafts-broadwell-xeon-d-for-hyperscale/ On 10 March 2015 at 19:42, Mark Hahn wrote: > Intel recently introduced an interesting product: > Xeon D is a Broadwell (Haswell shrink) SoC. > > It only has 8 cores, not high-clocked and only 2 dimm channels, so it's > definitely not at the same level of fat-node goodness as an e5-26xx v3. > But for 45W, you also get 2x onboard 10Gb! > > Anyone working on an HPC system based on these quite compact building > blocks? the SoC also has stuff like PCIe and SATA, > which is why a lot of the coverage is calling it a chip for desktop NAS, > etc. But for HPC purposes, the CPU is quite decent, memory balance is > reasonable, and it's hard > to argue with two free 10G... > > On that topic, I've read some work recently on performance tuning of Intel > 10G, but not in an HPC context. Is 10G still > sucking for MPI latency? (SFP+ DA noticably better than 10GbT?) > > If you're thinking of saying "why bother with an x86_64 SoC > when you can get a 64b Atom SoC", well, can you? (for cheap, > at commodity volume, etc...) Do any of the surviving Atom SoCs > still have onboard multiport switching fabrics? > > thanks, Mark Hahn. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdidomenico4 at gmail.com Wed Mar 11 05:37:06 2015 From: mdidomenico4 at gmail.com (Michael Di Domenico) Date: Wed, 11 Mar 2015 08:37:06 -0400 Subject: [Beowulf] Xeon D systems? (and 10G in general) In-Reply-To: References: Message-ID: Seems like an interesting chip. 256GF in 45 watts seems nice, but i'm a little leery on the 10G onboard. I'm not sure 10G is cost effective on the large scale yet. Is it even port-for-port to infiniband yet? I've not looked in a while. On Wed, Mar 11, 2015 at 5:52 AM, John Hearns wrote: > Very good article on The Platform: > > http://www.theplatform.net/2015/03/09/intel-crafts-broadwell-xeon-d-for-hyperscale/ > > On 10 March 2015 at 19:42, Mark Hahn wrote: >> >> Intel recently introduced an interesting product: >> Xeon D is a Broadwell (Haswell shrink) SoC. >> >> It only has 8 cores, not high-clocked and only 2 dimm channels, so it's >> definitely not at the same level of fat-node goodness as an e5-26xx v3. But >> for 45W, you also get 2x onboard 10Gb! >> >> Anyone working on an HPC system based on these quite compact building >> blocks? the SoC also has stuff like PCIe and SATA, >> which is why a lot of the coverage is calling it a chip for desktop NAS, >> etc. But for HPC purposes, the CPU is quite decent, memory balance is >> reasonable, and it's hard >> to argue with two free 10G... >> >> On that topic, I've read some work recently on performance tuning of Intel >> 10G, but not in an HPC context. Is 10G still >> sucking for MPI latency? (SFP+ DA noticably better than 10GbT?) >> >> If you're thinking of saying "why bother with an x86_64 SoC >> when you can get a 64b Atom SoC", well, can you? (for cheap, >> at commodity volume, etc...) Do any of the surviving Atom SoCs >> still have onboard multiport switching fabrics? >> >> thanks, Mark Hahn. >> _______________________________________________ >> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From hearnsj at googlemail.com Wed Mar 11 05:51:03 2015 From: hearnsj at googlemail.com (John Hearns) Date: Wed, 11 Mar 2015 12:51:03 +0000 Subject: [Beowulf] Mellanox Multi-host Message-ID: Talking about 10Gbps networking... and above: http://www.theregister.co.uk/2015/03/11/mellanox_adds_networking_specs_to_ocp/ "In the configuration Mellanox demonstrated, a 648-node cluster would only need 162 each of NICs, ports and cables." So looks like one switch port can fan out to four hosts, and they talk about mixing FPGA and GPU Might make for a very interesting cluster. -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.scott.atchley at gmail.com Wed Mar 11 07:44:16 2015 From: e.scott.atchley at gmail.com (Scott Atchley) Date: Wed, 11 Mar 2015 10:44:16 -0400 Subject: [Beowulf] Mellanox Multi-host In-Reply-To: References: Message-ID: Looking at this and the above link: http://www.mellanox.com/page/press_release_item?id=1501 It seems that the OCP Yosemite is a motherboard that allows four compute cards to be plugged into it. The compute cards can even have different CPUs (x86, ARM, Power). The Yosemite board has the NIC and connection to the switch. It is not clear if the "multi-host connection" is tunneled over the PCIe connection between the compute card and the Yosemite board or if network communication is handled over the compute card's NIC to the aggregator on the Yosemite board. Expect it is tunneled over PCIe, but more details would be nice. It seems the whole OCP Yosemite project is geared towards avoiding NUMA and using cheaper, simpler CPUs. On Wed, Mar 11, 2015 at 8:51 AM, John Hearns wrote: > Talking about 10Gbps networking... and above: > > > http://www.theregister.co.uk/2015/03/11/mellanox_adds_networking_specs_to_ocp/ > > "In the configuration Mellanox demonstrated, a 648-node cluster would only > need 162 each of NICs, ports and cables." > > So looks like one switch port can fan out to four hosts, > and they talk about mixing FPGA and GPU > Might make for a very interesting cluster. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fcannini at gmail.com Thu Mar 12 06:49:48 2015 From: fcannini at gmail.com (Fabricio Cannini) Date: Thu, 12 Mar 2015 10:49:48 -0300 Subject: [Beowulf] Xeon D systems? (and 10G in general) In-Reply-To: References: Message-ID: <5501997C.1000808@gmail.com> On 11-03-2015 06:52, John Hearns wrote: > Very good article on The Platform: > > http://www.theplatform.net/2015/03/09/intel-crafts-broadwell-xeon-d-for-hyperscale/ > > On 10 March 2015 at 19:42, Mark Hahn > wrote: > > Intel recently introduced an interesting product: > Xeon D is a Broadwell (Haswell shrink) SoC. > > It only has 8 cores, not high-clocked and only 2 dimm channels, so > it's definitely not at the same level of fat-node goodness as an > e5-26xx v3. But for 45W, you also get 2x onboard 10Gb! > > Anyone working on an HPC system based on these quite compact > building blocks? the SoC also has stuff like PCIe and SATA, > which is why a lot of the coverage is calling it a chip for desktop > NAS, etc. But for HPC purposes, the CPU is quite decent, memory > balance is reasonable, and it's hard > to argue with two free 10G... > > On that topic, I've read some work recently on performance tuning of > Intel 10G, but not in an HPC context. Is 10G still > sucking for MPI latency? (SFP+ DA noticably better than 10GbT?) > > If you're thinking of saying "why bother with an x86_64 SoC > when you can get a 64b Atom SoC", well, can you? (for cheap, > at commodity volume, etc...) Do any of the surviving Atom SoCs > still have onboard multiport switching fabrics? I'm not sure of what to make of this new Xeon, especially because it cuts right through the E3-1200 series, as you can see in the link that John provided, but may I speculate a little with two possibilities: - Intel is phasing out the E3-1200 series ( Unlikely, IMHO ) - Intel is beefing up its [def|off]ensive options against ARMv8, especially after Cray and Lenovo announced tests with Cavium's Thunder-X chip http://www.eweek.com/servers/cray-to-evaluate-arm-chips-in-its-supercomputers.html http://www.theplatform.net/2015/02/27/prototype-arm-clusters-muscle-hpc/ Any other bet ? From ghenriks at gmail.com Thu Mar 12 09:56:56 2015 From: ghenriks at gmail.com (Gerald Henriksen) Date: Thu, 12 Mar 2015 12:56:56 -0400 Subject: [Beowulf] Xeon D systems? (and 10G in general) In-Reply-To: <5501997C.1000808@gmail.com> References: <5501997C.1000808@gmail.com> Message-ID: On Thu, 12 Mar 2015 10:49:48 -0300, you wrote: >I'm not sure of what to make of this new Xeon, especially because it >cuts right through the E3-1200 series, as you can see in the link that >John provided, but may I speculate a little with two possibilities: > >- Intel is phasing out the E3-1200 series >( Unlikely, IMHO ) > >- Intel is beefing up its [def|off]ensive options against ARMv8, >especially after Cray and Lenovo announced tests with Cavium's Thunder-X >chip > http://www.eweek.com/servers/cray-to-evaluate-arm-chips-in-its-supercomputers.html > http://www.theplatform.net/2015/02/27/prototype-arm-clusters-muscle-hpc/ > > > >Any other bet ? Don't know enough about Hadoop, but it almost seems designed for it. The 10G network would help with both getting data onto the drives, and then with the distributed file system, and the systems will be mostly I/O bound so you don't need the greatest CPU. From fcannini at gmail.com Thu Mar 12 14:56:18 2015 From: fcannini at gmail.com (Fabricio Cannini) Date: Thu, 12 Mar 2015 18:56:18 -0300 Subject: [Beowulf] Xeon D systems? (and 10G in general) In-Reply-To: References: <5501997C.1000808@gmail.com> Message-ID: <55020B82.9070700@gmail.com> On 12-03-2015 13:56, Gerald Henriksen wrote: > On Thu, 12 Mar 2015 10:49:48 -0300, you wrote: > >> I'm not sure of what to make of this new Xeon, especially because it >> cuts right through the E3-1200 series, as you can see in the link that >> John provided, but may I speculate a little with two possibilities: >> >> - Intel is phasing out the E3-1200 series >> ( Unlikely, IMHO ) >> >> - Intel is beefing up its [def|off]ensive options against ARMv8, >> especially after Cray and Lenovo announced tests with Cavium's Thunder-X >> chip >> http://www.eweek.com/servers/cray-to-evaluate-arm-chips-in-its-supercomputers.html >> http://www.theplatform.net/2015/02/27/prototype-arm-clusters-muscle-hpc/ >> >> >> >> Any other bet ? > > Don't know enough about Hadoop, but it almost seems designed for it. > The 10G network would help with both getting data onto the drives, and > then with the distributed file system, and the systems will be mostly > I/O bound so you don't need the greatest CPU. I think you're into something, Gerald: http://www.theplatform.net/2015/03/12/lessons-and-time-drive-better-microserver-designs/ [ ]'s From samuel at unimelb.edu.au Fri Mar 13 17:27:51 2015 From: samuel at unimelb.edu.au (Chris Samuel) Date: Sat, 14 Mar 2015 11:27:51 +1100 Subject: [Beowulf] Heads up - CVE-2014-8159 - Infiniband security bug Message-ID: <2310426.ZFXuUnPtiT@quad> Happiness and joy - this bug appears to be in all distros and OFEDs. https://access.redhat.com/security/cve/CVE-2014-8159 # It was found that the Linux kernel's Infiniband subsystem # did not properly sanitize input parameters while registering # memory regions from user space via the (u)verbs API. A local # user with access to a /dev/infiniband/uverbsX device could use # this flaw to crash the system or, potentially, escalate their # privileges on the system. # # Find out more about CVE-2014-8159 from the MITRE CVE dictionary # and NIST NVD. # # Statement # # This issue does affect the Linux kernel packages as shipped # with Red Hat Enterprise Linux 5, 6, and 7, and Red Hat # Enterprise MRG 2. Future Linux kernel updates for the # respective releases will address this issue. Of course if you use a 3rd party OFED stack you'll need to look to them for any fixes (if available). -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci From mathog at caltech.edu Fri Mar 13 17:52:26 2015 From: mathog at caltech.edu (mathog) Date: Fri, 13 Mar 2015 17:52:26 -0700 Subject: [Beowulf] RAID question Message-ID: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> A bit off topic, but some of you may have run into something similar. Today I was called in to try and fix a server which had stopped working. Not my machine, the usual sysop is out sick. The model is a Dell PowerEdge T320 with a Raid PERC H710P controller. The symptoms reported were "it stopped working, could not find 'ls', and wouldn't reboot past grub". (Evidently it could find 'reboot'.) Got into the BIOS and ran RAID consistency check, which took 3 hours. It didn't say if it had passed or failed, or put up any sort of status message whatsoever, but there were no failure lights lit on the disks. On a reboot it gives: grub error 8: kernel must be loaded before booting. It is a Centos 6.5 system, so booted it with an installation disk of that flavor, and dropped down into a shell. This is where it gets strange. /boot is in /dev/sdb1. When mounted that directory is empty but when unmounted fsck shows 10 files in it taking up about 12Mb. Pretty clear why it wouldn't boot with nothing in /boot. Not sure what the 10 files fsck sees are, perhaps part of the filesystem. (ext2 I think). I had never tried running fsck on an empty file system in a partition before. /bin is missing entirely, so that's why "ls" stopped working. /usr/bin is still there, which is why reboot was OK. /var/log/messages shows that the machine was logging what look like corrected disk errors (sense errors) for /dev/sdb1 for days before it failed. Tried copying the contents of another machine's /boot (which is supposed to be an exact copy of this one) into /boot, and rebooting, but grub didn't get any farther than it had before. Probably grub needs to be reinstalled, but with /bin missing, and who knows what else gone besides, it seems like a full OS reinstall would be in order. Off the top of my head, if it weren't for the sense errors on /dev/sdb1, I would think that this might have been the result of an accidental (or hacker's) rm -rf / Anybody run into a hardware/software glitch with symptoms like this on a similar system??? Is there some way on these sorts of Dell's to run per disk diagnostics from BIOS or UEFI even if they are already grouped into a virtual disk by the controller? I suspect that the disk which is /dev/sdb may really be on its way out, but I couldn't get smartctl to work off the DVD or from the copy on disk. (The smartctl commands used were tested on the twin machine, and they worked there.) The BIOS showed that SMART was disabled on all of the disks. Web searches for diagnostics for this controller all referenced software that requires a running OS, nothing built into the BIOS/UEFI. (It is set to use BIOS.) Thanks, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From mathog at caltech.edu Fri Mar 13 18:13:05 2015 From: mathog at caltech.edu (mathog) Date: Fri, 13 Mar 2015 18:13:05 -0700 Subject: [Beowulf] RAID question In-Reply-To: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> References: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> Message-ID: On 13-Mar-2015 17:52, mathog wrote: > /boot is in /dev/sdb1. When mounted that directory is empty but > when unmounted fsck shows 10 files in it taking up about 12Mb. Forgot to mention that there were no errors when fsck was run on the partitions holding / and /home. So the losses of /boot and /bin does not look like it resulted from file system corruption. Regards, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From skylar.thompson at gmail.com Sat Mar 14 14:50:21 2015 From: skylar.thompson at gmail.com (Skylar Thompson) Date: Sat, 14 Mar 2015 14:50:21 -0700 Subject: [Beowulf] RAID question In-Reply-To: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> References: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> Message-ID: <5504AD1D.90002@gmail.com> On 3/13/2015 5:52 PM, mathog wrote: > A bit off topic, but some of you may have run into something similar. > > Today I was called in to try and fix a server which had stopped > working. Not my machine, the usual sysop is out sick. The > model is a Dell PowerEdge T320 with a Raid PERC H710P controller. > > The symptoms reported were "it stopped working, could not find 'ls', > and wouldn't reboot past grub". (Evidently it could find 'reboot'.) > > Got into the BIOS and ran RAID consistency check, which took 3 hours. > It didn't say if it had passed or failed, or put up any sort of status > message whatsoever, but there were no failure lights lit on the disks. > > On a reboot it gives: > > grub error 8: kernel must be loaded before booting. > > It is a Centos 6.5 system, so booted it with an installation disk of > that flavor, and dropped down into a shell. > > This is where it gets strange. > > /boot is in /dev/sdb1. When mounted that directory is empty but > when unmounted fsck shows 10 files in it taking up about 12Mb. Pretty > clear why it wouldn't boot with nothing in /boot. Not sure > what the 10 files fsck sees are, perhaps part of the filesystem. (ext2 > I think). I had never tried running fsck on an empty file system in a > partition before. > > /bin is missing entirely, so that's why "ls" stopped working. /usr/bin > is still there, which is why reboot was OK. > > /var/log/messages shows that the machine was logging what look like > corrected disk errors (sense errors) for /dev/sdb1 for days before it > failed. > > Tried copying the contents of another machine's /boot (which is > supposed to be an exact copy of this one) into /boot, and rebooting, > but grub didn't get any farther than it had before. Probably grub > needs to be reinstalled, but with /bin missing, and who knows what > else gone besides, it seems like a full OS reinstall would be in order. > > Off the top of my head, if it weren't for the sense errors on > /dev/sdb1, I would think that this might have been the result of an > accidental (or hacker's) > > rm -rf / > > Anybody run into a hardware/software glitch with symptoms like this on > a similar system??? > > Is there some way on these sorts of Dell's to run per disk diagnostics > from BIOS or UEFI even if they are already grouped into a virtual disk > by the controller? I suspect that the disk which is /dev/sdb may > really be on its way out, but I couldn't get smartctl to work off the > DVD or from the copy on disk. (The smartctl commands used were > tested on the twin machine, and they worked there.) The BIOS showed > that SMART was disabled on all of the disks. Web searches for > diagnostics for this controller all referenced software that requires > a running OS, nothing built into the BIOS/UEFI. (It is set to use BIOS.) I might start looking at non-RAID problems first. Maybe you have some bad memory or CPU? Errant rm could do it too, as you mentioned. Skylar From bug at wharton.upenn.edu Sat Mar 14 07:53:40 2015 From: bug at wharton.upenn.edu (Gavin W. Burris) Date: Sat, 14 Mar 2015 10:53:40 -0400 Subject: [Beowulf] RAID question In-Reply-To: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> References: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> Message-ID: <20150314145340.GB2384@shadow.home> Hi, David. This might not be the best forum for Linux technical support, a bit off-topic. But I can't resist... It sounds like you either had a controller glitch that corrupted the filesystem, or have an actual failed disks. I wouldn't rule out memory failure or bad cables, either. Each mount point would be its own filesystem, and depending how you have done RAID, the failure could punch holes in any one of the filesystems. That said, I recommend trying to salvage the important data immediately, with either centos rescue, or a fedora live cd / usb, keeping the partitions read-only to prevent further corruption. Dell does have a number of diagnostics. Usually a dset utility is run and the logs are sent to support for analysis. You may want to consider purchasing support for this incident, if you aren't under warranty. After you get the important data off, you may want to attempt repair. Hope you have backups. Note that once the boot partition is manipulated, grub needs to be reinstalled to the drive to map the booting of the kernel. This assumes you have consistent RAID and repaired filesystems. Something like: boot: linux rescue # chroot /mnt/sysimage/ # grub-install /dev/sda You may also need to go into grub. Something like: # grub grub> root (hd0,0) grub> setup (hd0) grub> quit # reboot Good luck! On 05:52PM Fri 03/13/15 -0700, mathog wrote: > A bit off topic, but some of you may have run into something similar. > > Today I was called in to try and fix a server which had stopped working. > Not my machine, the usual sysop is out sick. The > model is a Dell PowerEdge T320 with a Raid PERC H710P controller. > > The symptoms reported were "it stopped working, could not find 'ls', and > wouldn't reboot past grub". (Evidently it could find 'reboot'.) > > Got into the BIOS and ran RAID consistency check, which took 3 hours. It > didn't say if it had passed or failed, or put up any sort of status message > whatsoever, but there were no failure lights lit on the disks. > > On a reboot it gives: > > grub error 8: kernel must be loaded before booting. > > It is a Centos 6.5 system, so booted it with an installation disk of that > flavor, and dropped down into a shell. > > This is where it gets strange. > > /boot is in /dev/sdb1. When mounted that directory is empty but > when unmounted fsck shows 10 files in it taking up about 12Mb. Pretty clear > why it wouldn't boot with nothing in /boot. Not sure > what the 10 files fsck sees are, perhaps part of the filesystem. (ext2 I > think). I had never tried running fsck on an empty file system in a > partition before. > > /bin is missing entirely, so that's why "ls" stopped working. /usr/bin is > still there, which is why reboot was OK. > > /var/log/messages shows that the machine was logging what look like > corrected disk errors (sense errors) for /dev/sdb1 for days before it > failed. > > Tried copying the contents of another machine's /boot (which is supposed to > be an exact copy of this one) into /boot, and rebooting, > but grub didn't get any farther than it had before. Probably grub needs to > be reinstalled, but with /bin missing, and who knows what else gone besides, > it seems like a full OS reinstall would be in order. > > Off the top of my head, if it weren't for the sense errors on /dev/sdb1, I > would think that this might have been the result of an accidental (or > hacker's) > > rm -rf / > > Anybody run into a hardware/software glitch with symptoms like this on a > similar system??? > > Is there some way on these sorts of Dell's to run per disk diagnostics from > BIOS or UEFI even if they are already grouped into a virtual disk by the > controller? I suspect that the disk which is /dev/sdb may really be on its > way out, but I couldn't get smartctl to work off the DVD or from the copy on > disk. (The smartctl commands used were tested on the twin machine, and > they worked there.) The BIOS showed that SMART was disabled on all of the > disks. Web searches for diagnostics for this controller all referenced > software that requires a running OS, nothing built into the BIOS/UEFI. (It > is set to use BIOS.) > > Thanks, > > David Mathog > mathog at caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Gavin W. Burris Senior Project Leader for Research Computing The Wharton School University of Pennsylvania From bug at wharton.upenn.edu Sat Mar 14 08:01:36 2015 From: bug at wharton.upenn.edu (Gavin W. Burris) Date: Sat, 14 Mar 2015 11:01:36 -0400 Subject: [Beowulf] RAID question In-Reply-To: <20150314145340.GB2384@shadow.home> References: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> <20150314145340.GB2384@shadow.home> Message-ID: <20150314150136.GC2384@shadow.home> Just realized you are working with sdb. If you have to go into grub, sdb is hd1. I don't think it can hurt to install a boot sector to both sda and sdb, though. Cheers. From landman at scalableinformatics.com Sat Mar 14 08:48:59 2015 From: landman at scalableinformatics.com (Joe Landman) Date: Sat, 14 Mar 2015 11:48:59 -0400 Subject: [Beowulf] RAID question In-Reply-To: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> References: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> Message-ID: <5504586B.9090905@scalableinformatics.com> On 03/13/2015 08:52 PM, mathog wrote: > A bit off topic, but some of you may have run into something similar. > [...] > Anybody run into a hardware/software glitch with symptoms like this on > a similar system??? > > Is there some way on these sorts of Dell's to run per disk diagnostics > from BIOS or UEFI even if they are already grouped into a virtual disk > by the controller? I suspect that the disk which is /dev/sdb may > really be on its way out, but I couldn't get smartctl to work off the > DVD or from the copy on disk. (The smartctl commands used were > tested on the twin machine, and they worked there.) The BIOS showed > that SMART was disabled on all of the disks. Web searches for > diagnostics for this controller all referenced software that requires > a running OS, nothing built into the BIOS/UEFI. (It is set to use BIOS.) > System Rescue CD is your friend http://www.sysresccd.org/SystemRescueCd_Homepage I've seen a number of RAID cards hiccup and blow away data in a number of cases (and sadly the software only RAIDs are minimally better, only as you can see their code, but they all have bugs). These days, we tend to boot everything stateless (pure PXE/ramboot). This solves the RAID OS disks going away issue (rather completely), as we pull our config from our database (replicated/distributed, as the PXE boot images are as well). For stateful installs, I'd recommend getting a copy of system rescue as noted above. Lots of tools, full environment, including an incredible "boot off OS on drive" even if grub is blown away. Unfortunately, you can't infer much from drive labelling (sda/sdb etc.) so you'd need to see if your RAID became split-brained somehow. I've seen that a few times, actually with MD raid more than with hardware RAID. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. e: landman at scalableinformatics.com w: http://scalableinformatics.com t: @scalableinfo p: +1 734 786 8423 x121 c: +1 734 612 4615 From hearnsj at googlemail.com Mon Mar 16 02:21:14 2015 From: hearnsj at googlemail.com (John Hearns) Date: Mon, 16 Mar 2015 09:21:14 +0000 Subject: [Beowulf] Heads up - CVE-2014-8159 - Infiniband security bug In-Reply-To: <2310426.ZFXuUnPtiT@quad> References: <2310426.ZFXuUnPtiT@quad> Message-ID: Mellanox had a release of their OFED on Friday which included the relevant patch. 2.4-1.0.4 http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers On 14 March 2015 at 00:27, Chris Samuel wrote: > Happiness and joy - this bug appears to be in all distros and OFEDs. > > https://access.redhat.com/security/cve/CVE-2014-8159 > > # It was found that the Linux kernel's Infiniband subsystem > # did not properly sanitize input parameters while registering > # memory regions from user space via the (u)verbs API. A local > # user with access to a /dev/infiniband/uverbsX device could use > # this flaw to crash the system or, potentially, escalate their > # privileges on the system. > # > # Find out more about CVE-2014-8159 from the MITRE CVE dictionary > # and NIST NVD. > # > # Statement > # > # This issue does affect the Linux kernel packages as shipped > # with Red Hat Enterprise Linux 5, 6, and 7, and Red Hat > # Enterprise MRG 2. Future Linux kernel updates for the > # respective releases will address this issue. > > Of course if you use a 3rd party OFED stack you'll need to look to them for > any fixes (if available). > > -- > Christopher Samuel Senior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.org.au/ http://twitter.com/vlsci > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathog at caltech.edu Mon Mar 16 13:17:20 2015 From: mathog at caltech.edu (mathog) Date: Mon, 16 Mar 2015 13:17:20 -0700 Subject: [Beowulf] RAID question In-Reply-To: <5504AD1D.90002@gmail.com> References: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> <5504AD1D.90002@gmail.com> Message-ID: <3139dfc38d9d1cfed541189dfff68743@saf.bio.caltech.edu> Thanks for the feedback. After copying /boot and /bin from another machine and mucking about with grub for far too long (had to edit grub.conf to change virtual disk names, and in CentOS's rescue disk it saw the boot disk as hd1, but when grub actually started, it saw it as hd0) the system is back on line. The logs don't show a root command line that specifically took out those directories. They do show a bunch of scripts being run. My best guess is that one of them did something like this: AVAR=`command that failed and returned an empty string` rm -rf ${AVAR}/b* It seems unlikely that a low level controller failure would have snipped out those files/directories without resulting in a file system that was seen as corrupt by fsck. That said, there is something hardware related going on, since /var/log/messages has a lot of these (sorry about the wrap): Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb] Sense Key : Recovered Error [current] [descriptor] Mar 16 12:37:27 mandolin kernel: Descriptor sense data with sense descriptors (in hex): Mar 16 12:37:27 mandolin kernel: 72 01 04 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Mar 16 12:37:27 mandolin kernel: 00 4f 00 c2 40 50 Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb] ASC=0x4 ASCQ=0x1d That group has several other similar Dell servers, and this is the only one logging these. sdb1 holds /boot and sdb2 is where the lvm keeps its information. Regards, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From samuel at unimelb.edu.au Mon Mar 16 16:30:33 2015 From: samuel at unimelb.edu.au (Christopher Samuel) Date: Tue, 17 Mar 2015 10:30:33 +1100 Subject: [Beowulf] Heads up - CVE-2014-8159 - Infiniband security bug In-Reply-To: References: <2310426.ZFXuUnPtiT@quad> Message-ID: <55076799.4000801@unimelb.edu.au> On 16/03/15 20:21, John Hearns wrote: > Mellanox had a release of their OFED on Friday which included the > relevant patch. 2.4-1.0.4 That's correct, they had a previous attempt (2.4-1.0.0) that missed an integer overflow fix. https://community.mellanox.com/thread/2082 -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci From j.sassmannshausen at ucl.ac.uk Tue Mar 17 14:57:43 2015 From: j.sassmannshausen at ucl.ac.uk (=?iso-8859-1?q?J=F6rg_Sa=DFmannshausen?=) Date: Tue, 17 Mar 2015 21:57:43 +0000 Subject: [Beowulf] RAID question In-Reply-To: <3139dfc38d9d1cfed541189dfff68743@saf.bio.caltech.edu> References: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> <5504AD1D.90002@gmail.com> <3139dfc38d9d1cfed541189dfff68743@saf.bio.caltech.edu> Message-ID: <201503172157.43934.j.sassmannshausen@ucl.ac.uk> Hi David, for me it looks like either a controller or disc issue. I have seen these problems before on SCSI discs when the controller had a problem. Depending on the manufacturer it might be a good idea to contact them and see if they got more informations here. I have had some problems in the past with RAID controllers and the manufacturer here was ever so helpful in the diagnosis and repair of a failed RAID5 for example. So it might be a good idea to try them. All the best from a cold London J?rg On Montag 16 M?rz 2015 mathog wrote: > Thanks for the feedback. > > After copying /boot and /bin from another machine and mucking about with > grub for far too long (had to edit grub.conf to change virtual disk > names, and in CentOS's rescue disk it saw the boot disk as hd1, but when > grub actually started, it saw it as hd0) the system is back on line. > > The logs don't show a root command line that specifically took out those > directories. They do show a bunch of scripts being run. My best guess > is that one of them did something like this: > > AVAR=`command that failed and returned an empty string` > rm -rf ${AVAR}/b* > > It seems unlikely that a low level controller failure would have snipped > out those files/directories without resulting in a file system that was > seen as corrupt by fsck. > > That said, there is something hardware related going on, since > /var/log/messages has a lot of these (sorry about the wrap): > > Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb] Sense Key : > Recovered Error [current] [descriptor] > Mar 16 12:37:27 mandolin kernel: Descriptor sense data with sense > descriptors (in hex): > Mar 16 12:37:27 mandolin kernel: 72 01 04 1d 00 00 00 0e 09 0c 00 > 00 00 00 00 00 > Mar 16 12:37:27 mandolin kernel: 00 4f 00 c2 40 50 > Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb] ASC=0x4 ASCQ=0x1d > > That group has several other similar Dell servers, and this is the only > one logging these. sdb1 holds /boot and sdb2 is where the lvm keeps its > information. > > Regards, > > David Mathog > mathog at caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- ************************************************************* Dr. J?rg Sa?mannshausen, MRSC University College London Department of Chemistry Gordon Street London WC1H 0AJ email: j.sassmannshausen at ucl.ac.uk web: http://sassy.formativ.net Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 230 bytes Desc: This is a digitally signed message part. URL: From moloney.brendan at gmail.com Tue Mar 17 17:34:16 2015 From: moloney.brendan at gmail.com (Brendan Moloney) Date: Tue, 17 Mar 2015 17:34:16 -0700 Subject: [Beowulf] RAID question In-Reply-To: <201503172157.43934.j.sassmannshausen@ucl.ac.uk> References: <7249cb2cae1965fa60d52fd84c837292@saf.bio.caltech.edu> <5504AD1D.90002@gmail.com> <3139dfc38d9d1cfed541189dfff68743@saf.bio.caltech.edu> <201503172157.43934.j.sassmannshausen@ucl.ac.uk> Message-ID: Dell controllers are (almost?) always rebranded LSI. You can use the LSI tool "megacli", or if you prefer a slightly less insane UI you can use the newer "storcli". To see what happened during your consistency check you would need to check the logs that can be generated with these tools. These logs will probably also help you determine what went wrong initially. Unfortunately, these RAID controllers do not allow you to access any SMART data unless they are in JBOD mode so you pretty much have to pull the disk and check it on another machine. Last I looked, the default behavior for a consistency check is to assume the primary copy is correct for any discrepancies and overwrite the parity/mirror versions with that. Which is pretty dumb for a default setting (you can at least disable this automatic "repair"). So your consistency check might have wiped the good data... On Tue, Mar 17, 2015 at 2:57 PM, J?rg Sa?mannshausen < j.sassmannshausen at ucl.ac.uk> wrote: > Hi David, > > for me it looks like either a controller or disc issue. > > I have seen these problems before on SCSI discs when the controller had a > problem. Depending on the manufacturer it might be a good idea to contact > them > and see if they got more informations here. I have had some problems in the > past with RAID controllers and the manufacturer here was ever so helpful in > the diagnosis and repair of a failed RAID5 for example. > > So it might be a good idea to try them. > > All the best from a cold London > > J?rg > > > On Montag 16 M?rz 2015 mathog wrote: > > Thanks for the feedback. > > > > After copying /boot and /bin from another machine and mucking about with > > grub for far too long (had to edit grub.conf to change virtual disk > > names, and in CentOS's rescue disk it saw the boot disk as hd1, but when > > grub actually started, it saw it as hd0) the system is back on line. > > > > The logs don't show a root command line that specifically took out those > > directories. They do show a bunch of scripts being run. My best guess > > is that one of them did something like this: > > > > AVAR=`command that failed and returned an empty string` > > rm -rf ${AVAR}/b* > > > > It seems unlikely that a low level controller failure would have snipped > > out those files/directories without resulting in a file system that was > > seen as corrupt by fsck. > > > > That said, there is something hardware related going on, since > > /var/log/messages has a lot of these (sorry about the wrap): > > > > Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb] Sense Key : > > Recovered Error [current] [descriptor] > > Mar 16 12:37:27 mandolin kernel: Descriptor sense data with sense > > descriptors (in hex): > > Mar 16 12:37:27 mandolin kernel: 72 01 04 1d 00 00 00 0e 09 0c 00 > > 00 00 00 00 00 > > Mar 16 12:37:27 mandolin kernel: 00 4f 00 c2 40 50 > > Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb] ASC=0x4 ASCQ=0x1d > > > > That group has several other similar Dell servers, and this is the only > > one logging these. sdb1 holds /boot and sdb2 is where the lvm keeps its > > information. > > > > Regards, > > > > David Mathog > > mathog at caltech.edu > > Manager, Sequence Analysis Facility, Biology Division, Caltech > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > -- > ************************************************************* > Dr. J?rg Sa?mannshausen, MRSC > University College London > Department of Chemistry > Gordon Street > London > WC1H 0AJ > > email: j.sassmannshausen at ucl.ac.uk > web: http://sassy.formativ.net > > Please avoid sending me Word or PowerPoint attachments. > See http://www.gnu.org/philosophy/no-word-attachments.html > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.sassmannshausen at ucl.ac.uk Wed Mar 18 08:46:43 2015 From: j.sassmannshausen at ucl.ac.uk (=?iso-8859-15?q?J=F6rg_Sa=DFmannshausen?=) Date: Wed, 18 Mar 2015 15:46:43 +0000 Subject: [Beowulf] RAID question Message-ID: <201503181546.44117.j.sassmannshausen@ucl.ac.uk> Hi all, you can access the disc if it is hooked up to (some) LSI controllers: Try: $ smartctl -i /dev/sda -d megaraid,X You need to play around a bit with X as that is the port the controller is using. As you can see, I am using a megaraid controller card and that is working well here. As always: your mileage may vary. All the best from a sunny London J?rg On Wednesday 18 Mar 2015 00:34:16 you wrote: > Dell controllers are (almost?) always rebranded LSI. You can use the LSI > tool "megacli", or if you prefer a slightly less insane UI you can use the > newer "storcli". To see what happened during your consistency check you > would need to check the logs that can be generated with these tools. These > logs will probably also help you determine what went wrong initially. > Unfortunately, these RAID controllers do not allow you to access any SMART > data unless they are in JBOD mode so you pretty much have to pull the disk > and check it on another machine. > > Last I looked, the default behavior for a consistency check is to assume > the primary copy is correct for any discrepancies and overwrite the > parity/mirror versions with that. Which is pretty dumb for a default > setting (you can at least disable this automatic "repair"). So your > consistency check might have wiped the good data... > > > > On Tue, Mar 17, 2015 at 2:57 PM, J?rg Sa?mannshausen < > > j.sassmannshausen at ucl.ac.uk> wrote: > > Hi David, > > > > for me it looks like either a controller or disc issue. > > > > I have seen these problems before on SCSI discs when the controller had a > > problem. Depending on the manufacturer it might be a good idea to contact > > them > > and see if they got more informations here. I have had some problems in > > the past with RAID controllers and the manufacturer here was ever so > > helpful in the diagnosis and repair of a failed RAID5 for example. > > > > So it might be a good idea to try them. > > > > All the best from a cold London > > > > J?rg > > > > On Montag 16 M?rz 2015 mathog wrote: > > > Thanks for the feedback. > > > > > > After copying /boot and /bin from another machine and mucking about > > > with grub for far too long (had to edit grub.conf to change virtual > > > disk names, and in CentOS's rescue disk it saw the boot disk as hd1, > > > but when grub actually started, it saw it as hd0) the system is back > > > on line. > > > > > > The logs don't show a root command line that specifically took out > > > those directories. They do show a bunch of scripts being run. My > > > best guess > > > > > > is that one of them did something like this: > > > AVAR=`command that failed and returned an empty string` > > > rm -rf ${AVAR}/b* > > > > > > It seems unlikely that a low level controller failure would have > > > snipped out those files/directories without resulting in a file system > > > that was seen as corrupt by fsck. > > > > > > That said, there is something hardware related going on, since > > > /var/log/messages has a lot of these (sorry about the wrap): > > > > > > Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb] Sense Key : > > > Recovered Error [current] [descriptor] > > > Mar 16 12:37:27 mandolin kernel: Descriptor sense data with sense > > > descriptors (in hex): > > > Mar 16 12:37:27 mandolin kernel: 72 01 04 1d 00 00 00 0e 09 0c > > > 00 00 00 00 00 00 > > > Mar 16 12:37:27 mandolin kernel: 00 4f 00 c2 40 50 > > > Mar 16 12:37:27 mandolin kernel: sd 7:0:0:0: [sdb] ASC=0x4 ASCQ=0x1d > > > > > > That group has several other similar Dell servers, and this is the only > > > one logging these. sdb1 holds /boot and sdb2 is where the lvm keeps > > > its information. > > > > > > Regards, > > > > > > David Mathog > > > mathog at caltech.edu > > > Manager, Sequence Analysis Facility, Biology Division, Caltech > > > _______________________________________________ > > > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin > > > Computing To change your subscription (digest mode or unsubscribe) > > > visit http://www.beowulf.org/mailman/listinfo/beowulf > > > > -- > > ************************************************************* > > Dr. J?rg Sa?mannshausen, MRSC > > University College London > > Department of Chemistry > > Gordon Street > > London > > WC1H 0AJ > > > > email: j.sassmannshausen at ucl.ac.uk > > web: http://sassy.formativ.net > > > > Please avoid sending me Word or PowerPoint attachments. > > See http://www.gnu.org/philosophy/no-word-attachments.html > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf -- ************************************************************* Dr. J?rg Sa?mannshausen, MRSC University College London Department of Chemistry Gordon Street London WC1H 0AJ email: j.sassmannshausen at ucl.ac.uk web: http://sassy.formativ.net Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 230 bytes Desc: This is a digitally signed message part. URL: From mathog at caltech.edu Wed Mar 18 14:00:15 2015 From: mathog at caltech.edu (mathog) Date: Wed, 18 Mar 2015 14:00:15 -0700 Subject: [Beowulf] RAID question In-Reply-To: References: Message-ID: <01719fb05aed1ab069d551ab0b64da3c@saf.bio.caltech.edu> J?rg Sa?mannshausen wrote: > $ smartctl -i /dev/sda -d megaraid,X Right. The issues have been resolved. If anybody is still curious, this is what happened. The disappearing files/directories were the result of a script that was run as root which moved /boot and /bin to an obscure subdirectory belonging to that user. The disk errors were a red herring. The system had a Seagate USB disk plugged into it which I was not aware of. (It was less not obvious because of the rats nest of cables behind it.) This disk's partition table was marked bootable - even though there was nothing on that disk which would have supported a boot. This was the disk that was showing up as /dev/sdb. When CentOS booted normally it was automatically mounting this disk, which is why there was no mention of it in /etc/fstab. However, nothing was using this disk. It looks like at 30 minute intervals the OS "pinged" the device to see if it was still there, and the enclosure/disk did not fully support whatever command was being used for this operation, resulting in the sense error messages in the log files. When the rescue DVD was used it saw this device, created /dev/sda for it (yes, device names were exchanged in the two environments) and didn't mount it. Long SMART tests have now been run on each of the internal disks using smartctl commands like the one above, and all the disks are fine. megacli also comes up clean. The USB disk is no longer plugged in, which solved the issue of sense error messages going to /var/log/messages. Thanks for all of the suggestions, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From ghenriks at gmail.com Wed Mar 18 21:06:24 2015 From: ghenriks at gmail.com (Gerald Henriksen) Date: Thu, 19 Mar 2015 00:06:24 -0400 Subject: [Beowulf] RAID question In-Reply-To: <01719fb05aed1ab069d551ab0b64da3c@saf.bio.caltech.edu> References: <01719fb05aed1ab069d551ab0b64da3c@saf.bio.caltech.edu> Message-ID: <8likga523r8n781tapksgfcqodjlcqh1eo@4ax.com> On Wed, 18 Mar 2015 14:00:15 -0700, you wrote: >The disk errors were a red herring. The system had a Seagate USB disk >plugged into it which I was not aware of. (It was less not obvious >because of the rats nest of cables behind it.) This disk's partition >table was marked bootable - even though there was nothing on that disk >which would have supported a boot. This was the disk that was showing >up as /dev/sdb. When CentOS booted normally it was automatically >mounting this disk, which is why there was no mention of it in >/etc/fstab. However, nothing was using this disk. It looks like at 30 >minute intervals the OS "pinged" the device to see if it was still >there, and the enclosure/disk did not fully support whatever command was >being used for this operation, resulting in the sense error messages in >the log files. When the rescue DVD was >used it saw this device, created /dev/sda for it (yes, device names were >exchanged in the two environments) and didn't mount it. Linux does not guarantee device names to remain the same, which is why partitions are usually mounted via a unique partition ID in /etc/fstab From bug at wharton.upenn.edu Thu Mar 19 06:13:49 2015 From: bug at wharton.upenn.edu (Gavin W. Burris) Date: Thu, 19 Mar 2015 09:13:49 -0400 Subject: [Beowulf] RAID question In-Reply-To: <01719fb05aed1ab069d551ab0b64da3c@saf.bio.caltech.edu> References: <01719fb05aed1ab069d551ab0b64da3c@saf.bio.caltech.edu> Message-ID: <20150319131349.GI21895@shadow.wharton.upenn.edu> Wow, David. This should be nominated for some kind of sysadmin story award. What a rabbit hole, with plenty of misdirection. Too many cooks! Glad it worked out. -- Gavin W. Burris Senior Project Leader for Research Computing The Wharton School University of Pennsylvania