<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">All,<div><br></div><div>We are now perhaps unknowingly and implicitly discussing the definitions and differences between synthetic and applications benchmarks, and what emphasis each should be given in evaluating competing candidate system for purchase. </div><div><br></div><div>It is of course best to use both and to recognize the particular narrowness of stand-alone synthetics. </div><div><br></div><div>For instance, as suggested, adding communication synthetics to the mix is good, while recognizing their limitations … they by themselves tell you little about how a fabric will behave under the congestive load of a job mix that will typically be running in a production system. If you are interested in that (which I think you should be), then you might like to run GPCnet:</div><div><br></div><div><a href="https://github.com/netbench/GPCNET">https://github.com/netbench/GPCNET</a></div><div><br></div><div>Or, as is often done define a job mix throughout test.</div><div><br></div><div>No complex object intended to be used for a complex purpose can be evaluated by a couple of numbers. </div><div><br></div><div>Think of the problem of choosing an HPC system fit for your purposes as a job interview for a local open position for a system or performance engineer, and benchmark it accordingly.</div><div><br></div><div>rbw</div><div><br><div dir="ltr">Sent from my iPhone</div><div dir="ltr"><br><blockquote type="cite">On Mar 19, 2022, at 2:18 AM, Benson Muite <benson_muite@emailplus.org> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><span>For memory bandwidth, single node tests such as Likwid are helpful https://github.com/RRZE-HPC/likwid</span><br><span></span><br><span>MPI communication benchmarks are a good complement to this.</span><br><span></span><br><span>Full applications do more than the above, but these are easier starting points that require less domain specific application knowledge for general performance measurement.</span><br><span></span><br><span>On 3/19/22 3:58 AM, Richard Walsh wrote:</span><br><blockquote type="cite"><span>J,</span><br></blockquote><blockquote type="cite"><span>Trying to add a bit to the preceding useful answers …</span><br></blockquote><blockquote type="cite"><span>In my experience running these codes on very large systems for acceptances, to get optimal (HPCG or HPL) performance on GPUs (MI200 or A100) you need to obtain the optimized versions from the vendors which include scripts with ENV variable tunings specific the their versions and optimal affinity settings to manage the non-simple relationship between the NICs, the GPUs, and CPUs … you have iterate through the settings to find optimal settings for you system.</span><br></blockquote><blockquote type="cite"><span>If you set out to do this on your own, the chances of getting values similar to those posted on the TOP500 website are vanishingly small …</span><br></blockquote><blockquote type="cite"><span>As already noted, buyers of large HPC systems almost always require large scale runs of both HPCG (to demonstrate peak bandwidth) and HPL (to demonstrated peak processor) performance.</span><br></blockquote><blockquote type="cite"><span>Cheers!</span><br></blockquote><blockquote type="cite"><span>rbw</span><br></blockquote><blockquote type="cite"><span>Sent from my iPhone</span><br></blockquote><blockquote type="cite"><blockquote type="cite"><span>On Mar 18, 2022, at 7:35 PM, Massimiliano Fatica <mfatica@gmail.com> wrote:</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>HPCG measures memory bandwidth, the FLOPS capability of the chip is completely irrelevant.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Pretty much all the vendor implementations reach very similar efficiency if you compare them to the available memory bandwidth.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>There is some effect of the network at scale, but you need to have a really large system to see it in play.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>M</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>On Fri, Mar 18, 2022 at 5:20 PM Brian Dobbins <bdobbins@gmail.com <mailto:bdobbins@gmail.com>> wrote:</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Hi Jorg,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> We (NCAR - weather/climate applications) tend to find that HPCG</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> more closely tracks the performance we see from hardware than</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Linpack, so it definitely is of interest and watched, but our</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> procurements tend to use actual code that vendors run as part of</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> the process, so we don't 'just' use published HPCG numbers. Still, I'd say it's still very much a useful number, though.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> As one example, while I haven't seen HPCG numbers for the MI250x</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> accelerators, Prof. Matuoka of RIKEN tweeted back in November that</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> he anticipated that to score around 0.4% of peak on HPCG, vs 2% on</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> the NVIDIA A100 (while the A64FX they use hits an impressive 3%):</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> https://twitter.com/ProfMatsuoka/status/1458159517590384640</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> <https://twitter.com/ProfMatsuoka/status/1458159517590384640></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Why is that relevant? Well, /on paper/, the MI250X has ~96 TF</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> FP64 w/ Matrix operations, vs 19.5 TF on the A100. So, 5x in</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> theory, but Prof Matsuoka anticipated a ~5x differential in HPCG,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> /erasing/ that differential. Now, surely /someone/ has HPCG</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> numbers on the MI250X, but I've not yet seen any. Would love to</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> know what they are. But absent that information I tend to bet</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Matsuoka isn't far off the mark.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Ultimately, it may help knowing more about what kind of</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> applications you run - for memory bound CFD-like codes, HPCG tends</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> to be pretty representative.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Maybe it's time to update the saying that 'numbers never lie' to</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> something more accurate - 'numbers never lie, but they also rarely</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> tell the whole story'.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Cheers,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> - Brian</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> On Fri, Mar 18, 2022 at 5:08 PM Jörg Saßmannshausen</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> <sassy-work@sassy.formativ.net</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> <mailto:sassy-work@sassy.formativ.net>> wrote:</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Dear all,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> further the emails back in 2020 around the HPCG benchmark</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> test, as we are in</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> the process of getting a new cluster I was wondering if</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> somebody else in the</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> meantime has used that test to benchmark the particular</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> performance of the</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> cluster.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> From what I can see, the latest HPCG version is 3.1 from</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> August 2019. I also</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> have noticed that their website has a link to download a</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> version which</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> includes the latest A100 GPUs from nVidia.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> https://www.hpcg-benchmark.org/software/view.html?id=280</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> <https://www.hpcg-benchmark.org/software/view.html?id=280></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> What I was wondering is: has anybody else apart from Prentice</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> tried that test</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> and is it somehow useful, or does it just give you another set</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> of numbers?</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Our new cluster will not be at the same league as the</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> supercomputers, but we</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> would like to have at least some kind of handle so we can</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> compare the various</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> offers from vendors. My hunch is the benchmark will somehow</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> (strongly?) depend</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> on how it is tuned. As my former colleague used to say: I am</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> looking for some</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> war stories (not very apt to say these days!).</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Either way, I hope you are all well given the strange new</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> world we are living</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> in right now.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> All the best from a spring like dark London</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Jörg</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> _______________________________________________</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Beowulf mailing list, Beowulf@beowulf.org</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> <mailto:Beowulf@beowulf.org> sponsored by Penguin Computing</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> To change your subscription (digest mode or unsubscribe) visit</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> <https://beowulf.org/cgi-bin/mailman/listinfo/beowulf></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> _______________________________________________</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> Beowulf mailing list, Beowulf@beowulf.org</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> <mailto:Beowulf@beowulf.org> sponsored by Penguin Computing</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> To change your subscription (digest mode or unsubscribe) visit</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> <https://beowulf.org/cgi-bin/mailman/listinfo/beowulf></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>_______________________________________________</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</span><br></blockquote></blockquote><blockquote type="cite"><span>_______________________________________________</span><br></blockquote><blockquote type="cite"><span>Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing</span><br></blockquote><blockquote type="cite"><span>To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</span><br></blockquote><span></span><br><span>_______________________________________________</span><br><span>Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing</span><br><span>To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</span><br></div></blockquote></div></body></html>