[Beowulf] Ineternet cluster

[Warren Turkal]

On Saturday 30 September 2006 01:15, Maxence Dunnewind wrote:
> i would do a "packaging farm" because i know some people who packages some
> big app, and the building time is about 20 hours :/
> So, do you think there really is no solution for parrallel works over
> Internet ?
> (Maybe just with some computer with a big connection over Internet ...)

For the  most part, compiling is about having really small files be processed 
really quickly. Doing it in parallel would mean having many identically 
configured machines (same compiler, same libraries, etc.). This might be hard 
to coordinate if the machines are not maintained by the same administrator. 
Not to mention, the latency of sending a file to another machine to be 
compiled or sending prerequisite files for linking would probably overweigh 
the benefit of the parallel compile. If you had extremely low latency and 
identically configured servers, you might see some benefit.

However, I would have to question the utility of something like that when I 
look at the Debian buildd system, I see that the ETA for all packages on all 
architectures page ([1]) shows very low numbers for all major architectures. 
The only architectures that are over 10 hours out are armeb and m68k. M68k is 
not very important these days, and armeb seems to be an embedded 
architecture. I don't think either of those are release architectures for 
etch. For the other architectures, I don't think it's uncommon to have 
buildds turned off at times and not build. The Debian buildds are not based 
on a beowulf style setup and don't do parallel compiles to my knowledge. Each 
build daemon builds one whole package and moves on to the next IIRC.

All of these factors lead me to believe that parallel builds are not required 
to keep up with the flow of even massive bodies of software, like the archive 
that makes up Debian, for instance. According to the Debian home page, Debian 
contains 15490 binary packages.

[1] http://www.buildd.net/cgi/all_ETA.cgi

Ciao,
wt
-- 
Warren Turkal, Research Associate III/Systems Administrator
Colorado State University, Dept. of Atmospheric Science