[Beowulf] Ineternet cluster

[Geoff Jacobs]

Warren Turkal wrote:
> On Saturday 30 September 2006 01:15, Maxence Dunnewind wrote:
>> i would do a "packaging farm" because i know some people who packages some
>> big app, and the building time is about 20 hours :/
>> So, do you think there really is no solution for parrallel works over
>> Internet ?
>> (Maybe just with some computer with a big connection over Internet ...)
> 
> For the  most part, compiling is about having really small files be processed 
> really quickly. Doing it in parallel would mean having many identically 
> configured machines (same compiler, same libraries, etc.).
Yes.

> This might be hard 
> to coordinate if the machines are not maintained by the same administrator.
The machines automatically sync against the central server/dispatcher.

> Not to mention, the latency of sending a file to another machine to be 
> compiled or sending prerequisite files for linking would probably overweigh 
> the benefit of the parallel compile. If you had extremely low latency and 
> identically configured servers, you might see some benefit.
It's parallel at the package level.

> However, I would have to question the utility of something like that when I 
> look at the Debian buildd system, I see that the ETA for all packages on all 
> architectures page ([1]) shows very low numbers for all major architectures. 
> The only architectures that are over 10 hours out are armeb and m68k. M68k is 
> not very important these days, and armeb seems to be an embedded 
> architecture. I don't think either of those are release architectures for 
> etch. For the other architectures, I don't think it's uncommon to have 
> buildds turned off at times and not build. The Debian buildds are not based 
> on a beowulf style setup and don't do parallel compiles to my knowledge. Each 
> build daemon builds one whole package and moves on to the next IIRC.
Yes, but there are many machines for each arch on the buildd network.

> All of these factors lead me to believe that parallel builds are not required 
> to keep up with the flow of even massive bodies of software, like the archive 
> that makes up Debian, for instance. According to the Debian home page, Debian 
> contains 15490 binary packages.
> 
> [1] http://www.buildd.net/cgi/all_ETA.cgi
Not official. Try:
http://buildd.debian.org/stats/

By the trendline for m68k, I'm thinking we're gonna have to start using
emulators to compile m68k software.

> Ciao,
> wt


-- 
Geoffrey D. Jacobs

Go to the Chinese Restaurant,
Order the Special