Ten Years of Beowulf (a look back)
To honor the 10th anniversary of the Beowulf Project, co-founder Tom Sterling, assorted Beowulf champions, and I celebrated at a party hosted by Scyld Software and Penguin Computing. Coincident with Linux World in San Francisco, we introduced the new www.beowulf.org site.
The evening gave many of us a chance to reminisce about the origins of the Beowulf Project and reflect on its lessons, implications, and potential.
Thomas Sterling was teaching at MIT when we met during my student days. He was an admitted shared memory bigot and I was obsessed with writing open source network drivers. I ran into him when my bosses wouldn't let me keep working on the topic. Sterling had a funded research contract with NASA while I needed support, so a collaboration of mutual advantage was born. He claims that writing the memo to bring me onto the project produced the exothermic event that became the basis of Beowulf (though I remain skeptical to this day). In any case, together we conceived a novel way to leverage commodity technology to achieve unprecedented performance for problems suited to a distributed processing model.
As Linux godfather John "Mad Dog" Hall reminded the crowd at the party, Beowulf's advantages compared to other supercomputers are measurable (it costs less, is faster, is cheaper to maintain and easier to repair, AND you can actually get working applications for the Beowulf). I agree with Mad Dog and Mike Fitzmaurice, president of the Baltimore-Washington Beowulf Users Group, BWBUG.org, that there are other intangible benefits. Neither Sterling nor I anticipated people's sense of empowerment nor the intense emotional reaction Beowulf would produce.
The Beowulf approach enables individuals to solve their own problems, (admittedly a subversive idea in some circles). It has become apparent that Beowulf pioneers seem to possess a rare combination of talents:
- Strong domain expertise,
- Multidimensional thinking,
- Skillful programming (often self-taught).
This unusual form of intelligence, combined with a passion for what these computers can do is why the Beowulf architecture has been applied to such a wide variety of applications. Subject matter specialists who previously couldn't afford to solve compute-intensive problems with a supercomputer could suddenly gain access to embarrassingly parallel processors whether they're students or the leading experts in their field.
Our challenge for the next phase is providing the middleware so a broader community of intelligent mortals can take advantage of Beowulf clusters. To achieve this, the creativity and ingenuity of the pioneers will need to be complemented by commercial development and support. (more next month on the Future of Beowulf)
August Column: Beowulf.org Transformed