dealing with lots of sockets (was Re: [Beowulf] automount on high ports)

[Greg Lindahl]

On Wed, Jul 02, 2008 at 08:48:32PM -0400, Lawrence Stewart wrote:

> Back in 1994, with 90 MHz pentiums, NCSA's httpd was the leading
> webserver with a design that forked a new process for every request.

Apache eventually moved to a model where forked processes handled
several requests serially before eventually dying and being re-forked.
This reduces the fork overhead per request to something reasonable.

Recently there's a threaded version, but that's not the default.

Our web crawler at Blekko is event-driven: the work is divided up into
short subroutines which do non-blocking things, and when blocking is
needed, you return to the "system" indicating what code to execute
when the answer you're waiting for comes back.

This is just event-driven programming inside-out. Works great, too,
because the code is prettier than your typical event-driven code.

Now Legion had pretty code, but the fact that all of the contexts
shared a single stack meant that only the guy at the top of the stack
could execute. But I digress.

-- greg