[Beowulf] iptables

[Robert G. Brown]

Bogdan Costescu writes:

> 
> [ OK, I learnt the hard way that (this version of) pine doesn't do
> spell checking on the Subjectline , but what excuse do you have for
> not correcting it ? :-) ]
> 
> On Thu, 29 Sep 2005, Robert [UTF-8] G. Brown wrote:
> 
>> does anybody have any actual benchmark measurements of the
>> comparative impact of iptables on code execution rates
> 
> Well, benchmarking implies a controlled environment. But remember that

Ya, but I was just interested in any old "all things being equal"
numbers.  Gerry's report was nice, for example, although not as
microscopic as I would like because one has to know the details of his
application set in order to estimate what one really wants to know.  To
wit:

Isn't the major question what the effect of running iptables is on the
TASK ASSOCIATED network traffic latencies?  Or even, if you like, on
pure netpipe data streams in between otherwise isolated systems?  I was
assuming that we were addressing Mark's specific issue, phrased as: 

  "What is the net cost, in increased latency and decreased bandwidth on
  the network side and decreased userspace CPU available to the
  application on the application side, of running iptables for various
  kinds of streams and network traffic patterns?"

This is really a microbenchmark question.  If I had the time I'd answer
it myself, e.g. take two systems on a reasonably quiet network since the
edge case where you're whonking the boxes with nmap while running the
benchmark is -- if not irrelevant -- something that you fix by
rearranging your network until it is quiet and because we want to know
ROUGHLY what the cost per packet is to network latency.

Then just:

  a) Run netpipe or netperf or lmbench (or all three:-) for at least TCP
and UDP streams for various packet sizes, with and without iptables on.
Also Build an iptables-free kernel (if it is still possible to do so --
I THINK that there are CONFIG macros that let you shut it out still, but
I haven't built a kernel for a while or looked at iptables as anything
but an asset when I last did and am not certain).  Repeat.

  b) Build table of results.  Present curves of latencies, bws, for
various packet sizes and these three cases, reporting on base
architecture, network architecture (NIC/switch etc) and so on.

A secondary test series of interest would be the same three cases used
to get wall-clock runtime of one or more parallel apps (again run on a
reasonably quiet network).  With the cost "per packet" in hand you can
estimate latency increases from the first results.  The second gives you
a measure of how much CPU the additional kernel processing costs your
running application.  This COULD be predicted by the first result, but I
hesitate to conclude that it WILL because I'm not certain how iptables
coexists with DMA NICs for e.g. intermediate scale packet sizes.  List
comments suggest that it looks at every packet including secondary
packets in an incoming TCP stream rather than switching itself out of
the decisioning process once a stream is approved.  List traffic also
suggests that the mere presence of the code in the kernel is enough to
cause most of the negative effects even if no rules are applied.  

I'd just like to see both of those statements translated into
microbenchmark performance measures so we can all arrive at (at least)
some rules of thumb: "iptables is irrelevant to EP computations"
(probably true).  "iptables is the kiss of death and anathema to tightly
coupled computations that are still running on TCP and ethernet in the
first place" (possibly true, possibly mostly irrelevant since TCP on
ethernet isn't the best engineering choice for tightly coupled
computations anyway, unless the applications are pretty coarsely
grained, in which case yeah, sure, we're interested in the edge case
boundaries I guess); "if you excise iptables from your kernel ALL
applications will run 2% faster on average under even moderate levels
of background network traffic because of its crushing effect on context
switches and cache flushes" (who knows?).

So I don't think that it's that difficult to get some meaningful
numbers, aside from the kernel building bit.  The iptables on, iptables
off numbers I might even be able to generate today, if I ignore all my
email, neglect my students, leave my kids in after school (and pay out
$20 or so for the privilege), and leave Raof (my level 46 WoW character)
languishing with quests unfilled and gold unwon;-)

Or I could hope that some young (CS grad student?) blood on the list
with time hanging heavy on their hands and a few idle nodes might spend
a few hours this morning doing it all for us and maybe getting a nice
little publication out of it (and, if they like, learning a whole lot
about the kernel's TCP stack and how it does the iptables filtering, but
that kind of LEARNING stuff is strictly optional -- the numbers are the
key to the engineering, unless you plan to hack on the kernel more
deeply than the CONFIG_ options...:-).

   rgb
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://www.scyld.com/pipermail/beowulf/attachments/20050930/1f7eb6bb/attachment.bin