[Beowulf] breaking Amdahl's law

Joshua Mora joshua_mora at usa.net
Wed Jun 19 03:12:58 PDT 2013

Why are we still avoiding to model efficiency/massive_scalability and to
create runtimes/APIs/DSLs/compiler from power consumption constraints of
moving data ?

Congrats to Extoll on the competitive low latency. I still remember telling to
Holger Froning "your latency sucks compared to IB QDR on gen2" about 3 years

Eugen, a link to the post would have been sufficient.

Joshua Mora.

------ Original Message ------
Received: 02:26 AM PDT, 06/19/2013
From: Eugen Leitl <eugen at leitl.org>
To: Beowulf at beowulf.org, info at postbiota.org
Subject: [Beowulf] breaking Amdahl's law

> http://www.isc-events.com/isc13/isc_blog/items/breaking-the-law.html
> Breaking The Law
> Posted: 05-22-2013 17:00
> As a theoretical physicist, I am fascinated by “fundamental” laws of
> describing basic phenomena of nature. Many of these laws are tested through
> numerous experiments and are “valid” within the constraints defined by
> experiments. In this sense we “believe”, for example, in Newton's law
> gravity.
> However, any further experiment might challenge our fundamental concepts.
> Sometimes, they even can be broken. For example. Michelson's interferometer
> experiment questioned some of Newton's basic assumptions on the structure
> space and time and paved the way for new ideas.  Eventually this led to
> Einstein's theories of special and general relativity, thus changing our
> understanding of the structure of space-time once and forever.
> It is the experiments that differentiate natural science and engineering
> mathematics.  Mathematicians can "choose" their axioms – often inspired
> real world phenomena – while successful physicists, chemists and
> infer relevant axioms from controlled, robust and reproducible experiments
> and might formulate far reaching theories. However, while theories often
> believed to be true laws of nature, experiments might teach us better.
> Conclusion: challenging fundamental laws by experiments is crucial for
> progress in science and engineering.
> In parallel computing, there is a fundamental law stating that the fastest
> speedup achievable through parallelization is restricted by the part of the
> program that cannot be parallelized. This law, named after Gene Amdahl,
> appears to be fundamental for strong scaling. Its generalization governs
> efficiencies in the presence of different concurrency level. From Gustafson
> we learned how weak scaling can explain why many problems scale very far,
> sometimes to hundreds of thousands of cores. He showed us a specific
> in Amdahl’s law. Still the exponentially growing numbers of processors of
> future supercomputers will entail increasing restrictions on the efficiency
> of  massively parallel computing and will make life very hard in the
> era.
> I believe, time is ripe to challenge Amdahl's generalized law by exposing
> to a new class of  experiments in parallel computing. With the DEEP project
> we are about to demonstrate that the pitfalls of Amdahl’s law can be
> in specific situations.
> DEEP keeps the code parts of a simulation that can only be parallelized up
> a concurrency of p = L on a Cluster Computer equipped with fast general
> purpose processors. The highly parallelizable parts of the simulation are
> on a massively parallel Booster-system with a concurrency of p = H,  H >>
> The booster is equipped with many-core Xeon Phi processors and connected by
> 3D-torus network of sub-microsecond latency based on EXTOLL technology.
> The DEEP system software allows to dynamically distribute the tasks to the
> most appropriate parts of the hardware in order to achieve highest
> computational efficiency. The MPI programming paradigm in combination with
> improved version of BSC's OmpSs task-based programming environment allows
> application programmers to abstract from the system software by simply
> requesting the necessary resources. The rest is done dynamically by the
> system. Hence the name DEEP, the “Dynamical Exascale Entry Platform”.
> The applications adapted to DEEP are selected in order to investigate and
> demonstrate the usefulness of the combination of hardware, system software
> and the programming model to leave ground and leap beyond the limits of
> Amdahl’s law of parallel computing. We are eager to show our first results
> the ISC’13 in Leipzig.
> The DEEP project (www.deep-project.eu), comprising 16 partners from 8
> different countries and funded by the European commission, started in
> December 2011. In the first BoF session of ISC’13 we will present results
> achieved since then and demonstrate the hardware that already is up and
> running at the Jülich Supercomputing Centre.
> So, learn more about our experiment aimed at breaking the fundamental law
> parallel computing! Join the BoF 1 “Exascale Research ­ The European
> Approach” of the three EU funded projects in exascale computing DEEP,
> and Mont-Blanc on Tuesday, June 18, 2013, 9 am – 10 am at Hall 4.
> Biography:
> Prof. Dr. Dr. Thomas Lippert received his diploma in Theoretical Physics in
> 1987 from the University of Würzburg. He completed Ph.D. theses in
> theoretical physics at Wuppertal University on simulations of lattice
> chromodynamics and at Groningen University in the field of parallel
> with systolic algorithms. He is director of the Jülich Supercomputing
> at Forschungzentrum Jülich, member of the board of directors of the John
> Neumann Institute for Computing (NIC), and he holds the chair for
> Computational Theoretical Physics at the University of Wuppertal. His
> research interests include lattice gauge theories, quantum computing,
> numerical and parallel algorithms, and cluster computing.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list