[Beowulf] Ineternet cluster
wt at atmos.colostate.edu
Sat Sep 30 01:25:19 PDT 2006
On Saturday 30 September 2006 01:15, Maxence Dunnewind wrote:
> i would do a "packaging farm" because i know some people who packages some
> big app, and the building time is about 20 hours :/
> So, do you think there really is no solution for parrallel works over
> Internet ?
> (Maybe just with some computer with a big connection over Internet ...)
For the most part, compiling is about having really small files be processed
really quickly. Doing it in parallel would mean having many identically
configured machines (same compiler, same libraries, etc.). This might be hard
to coordinate if the machines are not maintained by the same administrator.
Not to mention, the latency of sending a file to another machine to be
compiled or sending prerequisite files for linking would probably overweigh
the benefit of the parallel compile. If you had extremely low latency and
identically configured servers, you might see some benefit.
However, I would have to question the utility of something like that when I
look at the Debian buildd system, I see that the ETA for all packages on all
architectures page () shows very low numbers for all major architectures.
The only architectures that are over 10 hours out are armeb and m68k. M68k is
not very important these days, and armeb seems to be an embedded
architecture. I don't think either of those are release architectures for
etch. For the other architectures, I don't think it's uncommon to have
buildds turned off at times and not build. The Debian buildds are not based
on a beowulf style setup and don't do parallel compiles to my knowledge. Each
build daemon builds one whole package and moves on to the next IIRC.
All of these factors lead me to believe that parallel builds are not required
to keep up with the flow of even massive bodies of software, like the archive
that makes up Debian, for instance. According to the Debian home page, Debian
contains 15490 binary packages.
Warren Turkal, Research Associate III/Systems Administrator
Colorado State University, Dept. of Atmospheric Science
More information about the Beowulf