[Beowulf] Ineternet cluster
gdjacobs at gmail.com
Mon Oct 2 00:07:12 PDT 2006
Warren Turkal wrote:
> On Saturday 30 September 2006 01:15, Maxence Dunnewind wrote:
>> i would do a "packaging farm" because i know some people who packages some
>> big app, and the building time is about 20 hours :/
>> So, do you think there really is no solution for parrallel works over
>> Internet ?
>> (Maybe just with some computer with a big connection over Internet ...)
> For the most part, compiling is about having really small files be processed
> really quickly. Doing it in parallel would mean having many identically
> configured machines (same compiler, same libraries, etc.).
> This might be hard
> to coordinate if the machines are not maintained by the same administrator.
The machines automatically sync against the central server/dispatcher.
> Not to mention, the latency of sending a file to another machine to be
> compiled or sending prerequisite files for linking would probably overweigh
> the benefit of the parallel compile. If you had extremely low latency and
> identically configured servers, you might see some benefit.
It's parallel at the package level.
> However, I would have to question the utility of something like that when I
> look at the Debian buildd system, I see that the ETA for all packages on all
> architectures page () shows very low numbers for all major architectures.
> The only architectures that are over 10 hours out are armeb and m68k. M68k is
> not very important these days, and armeb seems to be an embedded
> architecture. I don't think either of those are release architectures for
> etch. For the other architectures, I don't think it's uncommon to have
> buildds turned off at times and not build. The Debian buildds are not based
> on a beowulf style setup and don't do parallel compiles to my knowledge. Each
> build daemon builds one whole package and moves on to the next IIRC.
Yes, but there are many machines for each arch on the buildd network.
> All of these factors lead me to believe that parallel builds are not required
> to keep up with the flow of even massive bodies of software, like the archive
> that makes up Debian, for instance. According to the Debian home page, Debian
> contains 15490 binary packages.
>  http://www.buildd.net/cgi/all_ETA.cgi
Not official. Try:
By the trendline for m68k, I'm thinking we're gonna have to start using
emulators to compile m68k software.
Geoffrey D. Jacobs
Go to the Chinese Restaurant,
Order the Special
More information about the Beowulf