[Beowulf] iptaled (was: hpl size problems)

[Robert G. Brown]

Bogdan Costescu writes:

> 
> RGB writes:
> 
>> In other words, it contributes to per-connection latency but not 
>> much to streaming traffic once a connection is made.  So one might 
>> expect that udp (connectionless) traffic would be more expensive 
>> overall than sustained tcp connections...?
> 
> Once you turn on iptables, each and every packet has to be inspected 
> for rules matching - it's all or nothing. For each packet there is:
> - code that has to be executed, that takes precious time, and code 
> that takes (code) cache size which might kick part of your 
> application's innermost loop out
> - data that has to be inspected, that takes (data) cache size which 
> might kick part of your application's hot data out
> 
> The fact that in some cases (earlier matches) there is less code to be 
> executed and less data to be inspected is IMHO not so relevant: the 
> end result is cache misses anyway. Especially when you use optimized 
> libraries or optimizing compilers which make some assumptions about 
> the cache size(s), how much of the theoretical peak performance are 
> you willing to pay for iptables ? ;-)

I think that here Mark's question is still the relevant one.  This is
all theoretical, but the bottom line is just what IS the actual impact
on any given application (or even class of application).  That is,
where's the benchmark data that shows that these cache misses are NOT in
fact optimized away to irrelevance (not zero, of course, mere
irrelevance).

Or to echo his question again for the day, does anybody have any actual
benchmark measurements of the comparative impact of iptables on code
execution rates, for some standard piece(s) of code?  Spec with it and
without it?  Some MPI or PVM apps with it and without it?  Even linpack
with it and without it?

I'm quite prepared to believe that it costs you anywhere from a tens of
necs to tens of usecs, in the context of a network hit that already
costs you (typically) tens to hundreds of usecs.  In one limit it
matters, or might matter for some code.  In the other it probably
doesn't matter and in fact probably isn't even resolvable with anything
but a really high precision benchmarking timer and lots of statistics.

Note that I'm not disagreeing with you -- remember I was the one that
asserted that iptables were probably undesireable in the first place.
I'm just uncomfortable, now that Mark pointed it out, with the
"probably".  We should be able to state something like "using iptables
slows the X MPI application parallelized across 16 nodes down by Y%" or
we should just say "We don't really know what its effect is".  Finding
out may be difficult as it has ALSO been asserted that just "not using
it" isn't going to help you much as the problem is that it is built into
your kernel already and will slow you down regardless of whether or not
it is on.  Minimally somebody would need to test an iptables-free
kernel, and iptables-off kernel and an iptables-on kernel (with a couple
of fairly standard rulesets) for a range of application types and then
publish the results.

Periodically there are grad students looking for publishable projects on
the list (I recall one asking for something to do just last week,
right?).  Are you there?  Are you listening?

> Furthermore, I think that it's rather impractical to use iptables with 
> MPI jobs. For LAM/MPI for example, you need to allow between all nodes 
> TCP connections between high random ports (between application 
> instances) and UDP packets between high random ports (for the LAM 
> daemons). Isn't then better to just put the whole network behind some 
> firewall and forget about protection ?

Especially if you're going to use rsh instead of ssh, or otherwise open
the cluster nodes up to nearly random attacks.  Security of all sorts is
"expensive" (in principle, at least) compared to leaving it off.

    rgb

> 
> -- 
> Bogdan Costescu
> 
> IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
> Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
> Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
> E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20050929/6bb32f77/attachment.sig>