[Shorewall-users] Problem with ssh limit and scp stalling

[Jonathan Underwood]

Hi,

I have a very simple server setup, using shorewall as my firewall. I
have a line like this at the top of my rules file to allow ssh
connections, but limited to 3 connection per minute with a burst rate
of 3:

SSH/ACCEPT net $FW - - -
- 3/min:3 -

Now when I have that in place, and from a remote machine run scp
server:/some/file ., I find that the scp stalls after a few kb
reproducibly. Altering the above line to

SSH/ACCEPT net $FW

i.e. with no connection rate limiting fixes the issue and all is well.

My understanding of scp is that it opens a connection and uses that
for the whole file, and so it shouldn't be exceeding the connection
limit. So what am I doing wrong? I really would like to limit ssh
connections.

Thanks
Jonathan

[Roberto C. Sánchez]

Post by Jonathan Underwood
SSH/ACCEPT net $FW - - -
- 3/min:3 -
Now when I have that in place, and from a remote machine run scp
server:/some/file ., I find that the scp stalls after a few kb
reproducibly. Altering the above line to

Could you give an *exact* command line that produces this error?

Regards,

-Roberto

[Jonathan Underwood]

Post by Roberto C. Sánchez

Post by Jonathan Underwood
SSH/ACCEPT net $FW - - -
- 3/min:3 -
Now when I have that in place, and from a remote machine run scp
server:/some/file ., I find that the scp stalls after a few kb
reproducibly. Altering the above line to

Could you give an *exact* command line that produces this error?

scp servername:/home/username/filename .

[Roberto C. Sánchez]

Post by Jonathan Underwood

Post by Roberto C. Sánchez

Post by Jonathan Underwood
SSH/ACCEPT net $FW - - -
- 3/min:3 -
Now when I have that in place, and from a remote machine run scp
server:/some/file ., I find that the scp stalls after a few kb
reproducibly. Altering the above line to

Could you give an *exact* command line that produces this error?

scp servername:/home/username/filename .

You completely missed the point.

I meant something like this:

***@miami:~/tmp$ scp santiago:tmp/*.deb .
openswan-modules-2.4.27-3-386_2.4.6+dfsg-1~bp 100% 140KB 139.8KB/s 00:00

As in, *don't* obfuscate what you are doing.

Regards,

-Roberto

[Jonathan Underwood]

Post by Roberto C. Sánchez

Post by Jonathan Underwood

Post by Roberto C. Sánchez

Post by Jonathan Underwood
SSH/ACCEPT net $FW - - -
- 3/min:3 -
Now when I have that in place, and from a remote machine run scp
server:/some/file ., I find that the scp stalls after a few kb
reproducibly. Altering the above line to

Could you give an *exact* command line that produces this error?

$ scp withnail.phys.ucl.ac.uk:/home/jgu/220107.tar.bz2 .
Enter passphrase for key '/home/jgu/.ssh/id_dsa':
220107.tar.bz2 0% 9464KB 492.9KB/s - stalled -

When it stalls, it periodically has bursts of a few KB/s and then
returns to zero. Normally on this link without that line in the rules
file, I can get 5 MB/s.

I'm still not sure I answered your questions though.

Post by Roberto C. Sánchez

Post by Jonathan Underwood
scp servername:/home/username/filename .

You completely missed the point.
openswan-modules-2.4.27-3-386_2.4.6+dfsg-1~bp 100% 140KB 139.8KB/s 00:00
As in, *don't* obfuscate what you are doing.
Regards,
-Roberto
--
Roberto C. Sánchez
http://people.connexer.com/~roberto
http://www.connexer.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFGVwaB5SXWIKfIlGQRAsWeAJ9u+8qL9cKUssnOL3GhPXhHlmUZvwCeKNXK
/H5x6Z9Zcoti20R9Uip56IQ=
=85xI
-----END PGP SIGNATURE-----
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users

[Jonathan Underwood]

Post by Jonathan Underwood

Post by Roberto C. Sánchez

Post by Jonathan Underwood
SSH/ACCEPT net $FW - - -
- 3/min:3 -
Now when I have that in place, and from a remote machine run scp
server:/some/file ., I find that the scp stalls after a few kb
reproducibly. Altering the above line to

Could you give an *exact* command line that produces this error?

$ scp withnail.phys.ucl.ac.uk:/home/jgu/220107.tar.bz2 .
220107.tar.bz2 0% 9464KB 492.9KB/s - stalled -
When it stalls, it periodically has bursts of a few KB/s and then
returns to zero. Normally on this link without that line in the rules
file, I can get 5 MB/s.
I'm still not sure I answered your questions though.

I should also add that, if when the scp is in the stalled state as
described above, I log into the server (withnail) and comment out the
limited SSH/ACCEPT line in rules and replace it with the non limited
line and restart shorewall, the scp will then resume at full speed.

[Roberto C. Sánchez]

Post by Jonathan Underwood
I should also add that, if when the scp is in the stalled state as
described above, I log into the server (withnail) and comment out the
limited SSH/ACCEPT line in rules and replace it with the non limited
line and restart shorewall, the scp will then resume at full speed.

Wow. What does shorewall log just before and during the stall of scp?

Regards,

-Roberto

[Jonathan Underwood]

Post by Roberto C. Sánchez

Post by Jonathan Underwood
I should also add that, if when the scp is in the stalled state as
described above, I log into the server (withnail) and comment out the
limited SSH/ACCEPT line in rules and replace it with the non limited
line and restart shorewall, the scp will then resume at full speed.

Wow. What does shorewall log just before and during the stall of scp?

Well, oddly, nothing is logged. I will look into turning up what is logged.

[Simon Hobson]

SSH/ACCEPT net $FW - - - - 3/min:3

I would add logging to that statement and see what happens.
eg:

SSH/ACCEPT:info net $FW - - - - 3/min:3

[Jonathan Underwood]

Post by Simon Hobson

SSH/ACCEPT net $FW - - - - 3/min:3

I would add logging to that statement and see what happens.
SSH/ACCEPT:info net $FW - - - - 3/min:3

This results in these messages (with a couple of obfuscations in IP addresses:

Shorewall:net2fw:ACCEPT:IN=eth0 OUT=
MAC=00:xx:xx:xx:c2:49:00:d0:79:xx:98:00:08:00 SRC=130.xx.69.87
DST=128.xx.2.35 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=26203 DF PROTO=TCP
SPT=53708 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
Shorewall:net2fw:ACCEPT:IN=eth0 OUT=
MAC=00:xx:xx:xx:xx:49:00:xx:79:xx:98:00:08:00 SRC=130.xx.69.87
DST=128.xx.2.35 LEN=64 TOS=0x00 PREC=0x00 TTL=53 ID=26293 DF PROTO=TCP
SPT=53708 DPT=22 WINDOW=2485 RES=0x00 ACK URGP=0
Shorewall:net2fw:ACCEPT:IN=eth0 OUT=
MAC=00:xx:xx:xx:xx:49:00:d0:79:95:98:00:08:00 SRC=130.xx.69.87
DST=128.xx.2.35 LEN=64 TOS=0x00 PREC=0x00 TTL=53 ID=26324 DF PROTO=TCP
SPT=53708 DPT=22 WINDOW=3995 RES=0x00 ACK URGP=0
Shorewall:net2fw:ACCEPT:IN=eth0 OUT=
MAC=00:xx:xx:xx:xx:49:00:d0:79:95:98:00:08:00 SRC=130.xx.69.87
DST=128.xx.2.35 LEN=72 TOS=0x00 PREC=0x00 TTL=53 ID=27573 DF PROTO=TCP
SPT=53708 DPT=22 WINDOW=6036 RES=0x00 ACK URGP=0

Post by Simon Hobson
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users

[Jonathan Underwood]

Post by Jonathan Underwood

Post by Simon Hobson

SSH/ACCEPT net $FW - - - - 3/min:3

I would add logging to that statement and see what happens.
SSH/ACCEPT:info net $FW - - - - 3/min:3

Shorewall:net2fw:ACCEPT:IN=eth0 OUT=
MAC=00:xx:xx:xx:c2:49:00:d0:79:xx:98:00:08:00 SRC=130.xx.69.87
DST=128.xx.2.35 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=26203 DF PROTO=TCP
SPT=53708 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
Shorewall:net2fw:ACCEPT:IN=eth0 OUT=
MAC=00:xx:xx:xx:xx:49:00:xx:79:xx:98:00:08:00 SRC=130.xx.69.87
DST=128.xx.2.35 LEN=64 TOS=0x00 PREC=0x00 TTL=53 ID=26293 DF PROTO=TCP
SPT=53708 DPT=22 WINDOW=2485 RES=0x00 ACK URGP=0
Shorewall:net2fw:ACCEPT:IN=eth0 OUT=
MAC=00:xx:xx:xx:xx:49:00:d0:79:95:98:00:08:00 SRC=130.xx.69.87
DST=128.xx.2.35 LEN=64 TOS=0x00 PREC=0x00 TTL=53 ID=26324 DF PROTO=TCP
SPT=53708 DPT=22 WINDOW=3995 RES=0x00 ACK URGP=0
Shorewall:net2fw:ACCEPT:IN=eth0 OUT=
MAC=00:xx:xx:xx:xx:49:00:d0:79:95:98:00:08:00 SRC=130.xx.69.87
DST=128.xx.2.35 LEN=72 TOS=0x00 PREC=0x00 TTL=53 ID=27573 DF PROTO=TCP
SPT=53708 DPT=22 WINDOW=6036 RES=0x00 ACK URGP=0

oh. Duh. I'm dumb - they're obviously the messages corresponding to
the ssh session I have open to examine the logs on the remote server
:)

So it seems the stalled scp transfer isn't causing anything to be logged.

[Roberto C. Sánchez]

Post by Jonathan Underwood
oh. Duh. I'm dumb - they're obviously the messages corresponding to
the ssh session I have open to examine the logs on the remote server
:)
So it seems the stalled scp transfer isn't causing anything to be logged.

I'm completely baffled.

Regards,

-Roberto

[Jonathan Underwood]

Post by Roberto C. Sánchez

Post by Jonathan Underwood
oh. Duh. I'm dumb - they're obviously the messages corresponding to
the ssh session I have open to examine the logs on the remote server
:)
So it seems the stalled scp transfer isn't causing anything to be logged.

I'm completely baffled.

Me too :)

[Brian J. Murrell]

Post by Jonathan Underwood

Post by Roberto C. Sánchez

Post by Jonathan Underwood
oh. Duh. I'm dumb - they're obviously the messages corresponding to
the ssh session I have open to examine the logs on the remote server
:)
So it seems the stalled scp transfer isn't causing anything to be logged.

I'm completely baffled.

Me too :)

Maybe a silly question, and maybe covered at the start of the thread,
but does this all work without shorewall installing a ruleset? i.e. if
you do a "shorewall clear" does everything magically work again?

This smacks of an MSS/PPPoE type problem where only full TCP segments
get dropped. Or there is that sub-protocol that does probes of the
connection that gets fouled up by routers doing ICMP blackholing. I
forget what that sub-protocol was though. Anyone remember?

b.

[Roberto C. Sánchez]

Post by Brian J. Murrell
Maybe a silly question, and maybe covered at the start of the thread,
but does this all work without shorewall installing a ruleset? i.e. if
you do a "shorewall clear" does everything magically work again?

He did mention that upon removing the rate limit part of the rule and
restarting shorewall, everything works correctly.

Regards,

-Roberto

[Brian J. Murrell]

Post by Brian J. Murrell
This smacks of an MSS/PPPoE type problem where only full TCP segments
get dropped.

I.e. the type of problem that "clamp_mss" was invented to solve.

Post by Brian J. Murrell
Or there is that sub-protocol that does probes of the
connection that gets fouled up by routers doing ICMP blackholing. I
forget what that sub-protocol was though. Anyone remember?

I do. :-) PMTU discovery.

Both of the above problems should exist with or without Shorewall's
rules installed in iptables.

b.

[Jonathan Underwood]

Post by Brian J. Murrell
Maybe a silly question, and maybe covered at the start of the thread,
but does this all work without shorewall installing a ruleset? i.e. if
you do a "shorewall clear" does everything magically work again?

Yes, issuing a shorewall clear, or alternatively removing the limit
part of the SSH rule fixes the problem reproducibly.

Post by Brian J. Murrell
This smacks of an MSS/PPPoE type problem where only full TCP segments
get dropped. Or there is that sub-protocol that does probes of the
connection that gets fouled up by routers doing ICMP blackholing. I
forget what that sub-protocol was though. Anyone remember?

Hm. This is a bit over my head I'm afraid.

[Brian J. Murrell]

Post by Jonathan Underwood
Yes, issuing a shorewall clear, or alternatively removing the limit
part of the SSH rule fixes the problem reproducibly.

I'm afraid my day was too busy to pay attention to the start of this
thread, but that you can install rules with shorewall and just not
install the limit part suggests to me that it's not either of the
problems below.

Post by Jonathan Underwood

Post by Brian J. Murrell
This smacks of an MSS/PPPoE type problem where only full TCP segments
get dropped. Or there is that sub-protocol that does probes of the
connection that gets fouled up by routers doing ICMP blackholing. I
forget what that sub-protocol was though. Anyone remember?

Hm. This is a bit over my head I'm afraid.

Given the above, it's probably not relevant, so you probably needed
worry about it.

b.

[Andrew Suffield]

Post by Roberto C. Sánchez

Post by Jonathan Underwood
oh. Duh. I'm dumb - they're obviously the messages corresponding to
the ssh session I have open to examine the logs on the remote server
:)
So it seems the stalled scp transfer isn't causing anything to be logged.

I'm completely baffled.

Capturing the traffic with tcpdump -s 4096 -w (simultaneously on all
involved hosts) may be more informative.

[Jonathan Underwood]

Post by Andrew Suffield

Post by Roberto C. Sánchez

Post by Jonathan Underwood
oh. Duh. I'm dumb - they're obviously the messages corresponding to
the ssh session I have open to examine the logs on the remote server
:)
So it seems the stalled scp transfer isn't causing anything to be logged.

I'm completely baffled.

Capturing the traffic with tcpdump -s 4096 -w (simultaneously on all
involved hosts) may be more informative.

I've never managed to climb the tcpdump learning curve, but I just
fired up wireshark on the client machine and set it to filter all
packets to/from the server machine. What jumps out at me is this:

a) When I hit a stall in the scp transfer, the first suspect packet I
see contains "A segment before this frame was lost" in the TCP
analysis flags. This packet has source being the server and
destination being the local machine.

b) Then the next packet's TCP analysis flags contains "Duplicate to
the ACK in frame 299". This packet has source being the local client
machine, and destination being the server. Frame 299 is the packet
before the packet flagged as "A segment before this frame was lost"
that I mentioned in (a) above.

c) The I see a continual sequence of packets of type (b), followed by
a packet from the server to the client with protocol SSHv2 and
seemingly being an ACK to the previous packet.

d) After a while of (c), I see packets tagged as retransmission.

I'm afraid I don't know enough to interpret all this. If it's useful
to send over the wireshark capture file, let me know.

J

[Jonathan Underwood]

Another thing that may help as a sanity check is that at the point
where an scp is stalling, on the server there are no entries under
/proc/net/ipt_recent

[Andrew Suffield]

Post by Jonathan Underwood

Post by Andrew Suffield

Post by Roberto C. Sánchez

Post by Jonathan Underwood
oh. Duh. I'm dumb - they're obviously the messages corresponding to
the ssh session I have open to examine the logs on the remote server
:)
So it seems the stalled scp transfer isn't causing anything to be logged.

I'm completely baffled.

Capturing the traffic with tcpdump -s 4096 -w (simultaneously on all
involved hosts) may be more informative.

I've never managed to climb the tcpdump learning curve, but I just
fired up wireshark on the client machine and set it to filter all
packets to/from the server machine.
I'm afraid I don't know enough to interpret all this. If it's useful
to send over the wireshark capture file, let me know.

tcpdump -w just saves the traffic to a file. Saving the wireshark
capture does exactly the same thing, it's just easier to install
tcpdump; either way will work fine. Posting the captures so we can
look at it is probably the only thing left to do at this point, given
how bizarre this problem is.

Remember - it's important to get a capture of the *same* session from
all the interesting points (at least the server, client, and both
interfaces of the firewall).

We'll also need the output of 'shorewall dump' (I don't think you
posted that yet). Follow #3 on http://shorewall.net/support.htm

Post by Jonathan Underwood
a) When I hit a stall in the scp transfer, the first suspect packet I
see contains "A segment before this frame was lost" in the TCP
analysis flags. This packet has source being the server and
destination being the local machine.

This part tells you that something is going wrong at a point closer to
the server than the one where you captured. The next thing you would
want to do is to look at the packet that was dropped and figure out
what made it different from all the other ones, so you would need to
compare this to a capture from the other end - hence why you need to
capture at multiple points (one to tell you which packet is dropped,
the other to show you that packet's contents).

[Jonathan Underwood]

Post by Andrew Suffield
tcpdump -w just saves the traffic to a file. Saving the wireshark
capture does exactly the same thing, it's just easier to install
tcpdump; either way will work fine. Posting the captures so we can
look at it is probably the only thing left to do at this point, given
how bizarre this problem is.
Remember - it's important to get a capture of the *same* session from
all the interesting points (at least the server, client, and both
interfaces of the firewall).

OK, I'll need a bit of time to do this...

Post by Andrew Suffield
We'll also need the output of 'shorewall dump' (I don't think you
posted that yet). Follow #3 on http://shorewall.net/support.htm

But this bit I have just done. I restarted shorewall with rate
limiting in the ssh rule, on the server, and on my local machine tried
to scp a file from the server to local machine, which stalled. While
it was stalled (i.e. I didn't ctrl-c out) i did a dump, the result of
which is attached.

I'll work on getting useful tcpdump/wireshark output from the server.

J.

[Tom Eastep]

Post by Jonathan Underwood

Post by Andrew Suffield
tcpdump -w just saves the traffic to a file. Saving the wireshark
capture does exactly the same thing, it's just easier to install
tcpdump; either way will work fine. Posting the captures so we can
look at it is probably the only thing left to do at this point, given
how bizarre this problem is.
Remember - it's important to get a capture of the *same* session from
all the interesting points (at least the server, client, and both
interfaces of the firewall).

OK, I'll need a bit of time to do this...

Post by Andrew Suffield
We'll also need the output of 'shorewall dump' (I don't think you
posted that yet). Follow #3 on http://shorewall.net/support.htm

But this bit I have just done. I restarted shorewall with rate
limiting in the ssh rule, on the server, and on my local machine tried
to scp a file from the server to local machine, which stalled. While
it was stalled (i.e. I didn't ctrl-c out) i did a dump, the result of
which is attached.
I'll work on getting useful tcpdump/wireshark output from the server.

A couple of things.

a) You are using the RATE LIMIT column of the rules file to limit SSH.
That is *not* recommended. Rather, we prefer the 'Limit' built-in
action. The former limits the total number of connections from all
sources while the latter is per-IP. So your observation that there was
nothing in /proc/.../ipt_recent is correct; with your ruleset, there
will *never* be anything there.

b) You are getting a lot of INVALID state packets. You might try adding
this to your /etc/shorewall/init file:

echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

-Tom

[Tom Eastep]

Post by Tom Eastep

Post by Jonathan Underwood

Post by Andrew Suffield
tcpdump -w just saves the traffic to a file. Saving the wireshark
capture does exactly the same thing, it's just easier to install
tcpdump; either way will work fine. Posting the captures so we can
look at it is probably the only thing left to do at this point, given
how bizarre this problem is.
Remember - it's important to get a capture of the *same* session from
all the interesting points (at least the server, client, and both
interfaces of the firewall).

OK, I'll need a bit of time to do this...

Post by Andrew Suffield
We'll also need the output of 'shorewall dump' (I don't think you
posted that yet). Follow #3 on http://shorewall.net/support.htm

But this bit I have just done. I restarted shorewall with rate
limiting in the ssh rule, on the server, and on my local machine tried
to scp a file from the server to local machine, which stalled. While
it was stalled (i.e. I didn't ctrl-c out) i did a dump, the result of
which is attached.
I'll work on getting useful tcpdump/wireshark output from the server.

A couple of things.
a) You are using the RATE LIMIT column of the rules file to limit SSH.
That is *not* recommended. Rather, we prefer the 'Limit' built-in
action. The former limits the total number of connections from all
sources while the latter is per-IP. So your observation that there was
nothing in /proc/.../ipt_recent is correct; with your ruleset, there
will *never* be anything there.
b) You are getting a lot of INVALID state packets. You might try adding
echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

You can cause Netfilter to display invalid state packets that it drops
with this command

echo 255 >/proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
modprobe ipt_LOG

Warning: This will log to all terminal console sessions!

-Tom

[Andrew Suffield]

Post by Tom Eastep

Post by Jonathan Underwood

Post by Andrew Suffield
We'll also need the output of 'shorewall dump' (I don't think you
posted that yet). Follow #3 on http://shorewall.net/support.htm

But this bit I have just done. I restarted shorewall with rate
limiting in the ssh rule, on the server, and on my local machine tried
to scp a file from the server to local machine, which stalled. While
it was stalled (i.e. I didn't ctrl-c out) i did a dump, the result of
which is attached.
I'll work on getting useful tcpdump/wireshark output from the server.

b) You are getting a lot of INVALID state packets.

Which leads me to suspect that we're looking at a clusterfuck
here. Hypothesis: something *else* is wrong, and is breaking TCP
connections at intervals. Under normal circumstances, some kind of
error recovery manages to get the connection going again, and the
problem is not so pronounced that you've noticed it before
(particularly given scp's highly inaccurate reporting of the transfer
rate, which tends to hide jitter). However, with the rate limit in
place, it's somehow blocking that from happening. There's no evidence
to back this up, but it's the first thing I've been able to think of
which explains what could be going on.

I do notice this, which is interesting:

Chain net2fw (1 references)
pkts bytes target prot opt in out source destination
216 17116 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
4 276 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 limit: avg 3/min burst 3

Why *four* new ssh connections? That'll certainly have hit the rate
limit, but where did the other three come from?

[Tom Eastep]

Post by Andrew Suffield

Post by Tom Eastep

Post by Jonathan Underwood

Post by Andrew Suffield
We'll also need the output of 'shorewall dump' (I don't think you
posted that yet). Follow #3 on http://shorewall.net/support.htm

But this bit I have just done. I restarted shorewall with rate
limiting in the ssh rule, on the server, and on my local machine tried
to scp a file from the server to local machine, which stalled. While
it was stalled (i.e. I didn't ctrl-c out) i did a dump, the result of
which is attached.
I'll work on getting useful tcpdump/wireshark output from the server.

b) You are getting a lot of INVALID state packets.

Which leads me to suspect that we're looking at a clusterfuck
here. Hypothesis: something *else* is wrong, and is breaking TCP
connections at intervals. Under normal circumstances, some kind of
error recovery manages to get the connection going again, and the
problem is not so pronounced that you've noticed it before
(particularly given scp's highly inaccurate reporting of the transfer
rate, which tends to hide jitter). However, with the rate limit in
place, it's somehow blocking that from happening. There's no evidence
to back this up, but it's the first thing I've been able to think of
which explains what could be going on.
Chain net2fw (1 references)
pkts bytes target prot opt in out source destination
216 17116 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
4 276 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 limit: avg 3/min burst 3
Why *four* new ssh connections? That'll certainly have hit the rate
limit, but where did the other three come from?

They are INVALID (both NEW and INVALID go through the rules) and are
silently dropped in Drop.

-Tom

[Tom Eastep]

Post by Tom Eastep

Post by Andrew Suffield
Chain net2fw (1 references)
pkts bytes target prot opt in out source destination
216 17116 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
4 276 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 limit: avg 3/min burst 3
Why *four* new ssh connections? That'll certainly have hit the rate
limit, but where did the other three come from?

They are INVALID (both NEW and INVALID go through the rules) and are
silently dropped in Drop.

Note that if the ACCEPT rule has no 'limit' then the INVALID packets are
accepted and the problem magically goes away. But because these packets
occur regularly, they eventually exhaust any imposed 'limit' and the
connection then stalls.

-Tom

[Tom Eastep]

Post by Tom Eastep

Post by Tom Eastep

Post by Andrew Suffield
Chain net2fw (1 references)
pkts bytes target prot opt in out source destination
216 17116 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
4 276 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 limit: avg 3/min burst 3
Why *four* new ssh connections? That'll certainly have hit the rate
limit, but where did the other three come from?

They are INVALID (both NEW and INVALID go through the rules) and are
silently dropped in Drop.

Note that if the ACCEPT rule has no 'limit' then the INVALID packets are
accepted and the problem magically goes away. But because these packets
occur regularly, they eventually exhaust any imposed 'limit' and the
connection then stalls.

Until Jonathan finds and corrects the root cause, a workaround would be
to precede his Limit/(ACCEPT...limit) rule with:

allowInvalid net fw tcp 22

-Tom

[Jonathan Underwood]

Post by Tom Eastep
Note that if the ACCEPT rule has no 'limit' then the INVALID packets are
accepted and the problem magically goes away. But because these packets
occur regularly, they eventually exhaust any imposed 'limit' and the
connection then stalls.

Just to make sure I understand this correctly - do you mean that
INVALID packets are "counted" as NEW packets as far as the limit is
concerned?

J.

[Tom Eastep]

Post by Jonathan Underwood

Post by Tom Eastep
Note that if the ACCEPT rule has no 'limit' then the INVALID packets are
accepted and the problem magically goes away. But because these packets
occur regularly, they eventually exhaust any imposed 'limit' and the
connection then stalls.

Just to make sure I understand this correctly - do you mean that
INVALID packets are "counted" as NEW packets as far as the limit is
concerned?

Sort of. Shorewall sends both INVALID and NEW packets through the rules
generated by /etc/shorewall/rules. This gives users the choice of adding
'allowInvalid' and 'dropInvalid' rules to control INVALID separately
from NEW. If you only want to deal with 'NEW' connections in your rules
file, then just add this as the first entry in the file:

dropInvalid all all

In the absence of any acceptInvalid/dropInvalid rules, then any rule in
your /etc/shorewall/rules applies equally to NEW and INVALID packets. In
many cases, this approach allows Shorewall to mask problems like you are
experiencing. It doesn't mask those problems when rate limiting is
applied, however.

-Tom

[Jonathan Underwood]

Post by Tom Eastep
A couple of things.
a) You are using the RATE LIMIT column of the rules file to limit SSH.
That is *not* recommended. Rather, we prefer the 'Limit' built-in
action. The former limits the total number of connections from all
sources while the latter is per-IP. So your observation that there was
nothing in /proc/.../ipt_recent is correct; with your ruleset, there
will *never* be anything there.

Ok. so, acting on this piece of information alone, I changed the ssh
rule to read

Limit:info:SSHA,3,60 net $FW tcp 22

That alone did not help the situation in that when I try and scp from
the server the scp stalls. If I stop the scp (Ctrl-C) and then repeat
the command, no connection is allowed and shorewall on the server logs
that it is dropping the packets from my local machine:

May 26 21:41:20 withnail kernel: Shorewall:SSHA:DROP:IN=eth0 OUT=
MAC=00:19:b9:00:c2:49:00:d0:79:95:98:00:08:00 SRC=90.193.1
82.62 DST=128.40.2.35 LEN=72 TOS=0x00 PREC=0x00 TTL=51 ID=17706 DF
PROTO=TCP SPT=34948 DPT=22 WINDOW=3555 RES=0x00 ACK FIN U
RGP=0

Of course, these two commands, which should equate to only two
connections, should not be triggering the limit of (3,60) to be
kicking in. This behaviour is reproducible.

Post by Tom Eastep
b) You are getting a lot of INVALID state packets. You might try adding
echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

Next I did this (in combination with the change above), and everything
works just as it should.

I tried one other thing - I reverted this last change (echo 0 >
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal) and turned
off tcp window scaling on the server (i.e. echo 0 >
/proc/sys/net/ipv4/tcp_window_scaling). That also made everything work
as it should, with no stalling.

So, Tom, it seems you're correct in your analysis.

[Aside: Just for kicks, I reverted to the original rate limited ssh
rule, and that too works as expected if I do echo 1 >
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal.]

So if I understand you (Tom) correctly, this is an issue with the
network the server is on, or the firewall it is behind that is causing
problems with certain server firewall configurations? [Just in case
that's a confusing sentence, here we have been debugging a shorewall
generated firewall, but the machine that firewall is on is behind a
LAN firewall allowing ssh connections through to the server in
question]

I realize at this point it is probably clear that this is no bug with
shorewall, but I wonder if you might have any suggestion about how I
would go about finding what is causing all of those INVALID packets.

Thanks to Tom, Andrew, Brian, Roberto and Simon for your suggestions,
it really has been much appreciated.

jonathan.

Post by Tom Eastep
-Tom
--
Tom Eastep \ Nothing is foolproof to a sufficiently talented fool
Shoreline, \ http://shorewall.net
PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users

[Tom Eastep]

Post by Jonathan Underwood
So if I understand you (Tom) correctly, this is an issue with the
network the server is on, or the firewall it is behind that is causing
problems with certain server firewall configurations? [Just in case
that's a confusing sentence, here we have been debugging a shorewall
generated firewall, but the machine that firewall is on is behind a
LAN firewall allowing ssh connections through to the server in
question]
I realize at this point it is probably clear that this is no bug with
shorewall, but I wonder if you might have any suggestion about how I
would go about finding what is causing all of those INVALID packets.

The problem is caused by 'out-of-window' packets. So to totally analyze
the problem, you may have to capture:

a) The SCP stream on the outer interface of the other firewall.
b) The SCP stream on the outer interface of the Shorewall box.
c) Invalid connection state packets (I sent instructions earlier).

Once you find out which packets are being dropped (c), then you can
compare those packets in (a) and (b) to see if the other firewall is
mangling them. If not, then you need to send (b) and (c) to the
netfilter developers for analysis.

tcp_be_liberal (which you are setting) turns off Netfilter
window-tracking except for RST packets so you will need to turn of
tcp_be_liberal while tracking this down.

-Tom

[Jonathan Underwood]

Post by Tom Eastep
The problem is caused by 'out-of-window' packets. So to totally analyze
a) The SCP stream on the outer interface of the other firewall.
b) The SCP stream on the outer interface of the Shorewall box.
c) Invalid connection state packets (I sent instructions earlier).
Once you find out which packets are being dropped (c), then you can
compare those packets in (a) and (b) to see if the other firewall is
mangling them. If not, then you need to send (b) and (c) to the
netfilter developers for analysis.

OK, thanks for the info. Alas, (a) isn't accesible to me, but I'll
spend some time seeing if I can see anything suspect in (b).

Post by Tom Eastep
tcp_be_liberal (which you are setting) turns off Netfilter
window-tracking except for RST packets so you will need to turn of
tcp_be_liberal while tracking this down.

OK.

Thanks once again for your help and patience.

Jonathan.

[Tom Eastep]

Post by Jonathan Underwood

Post by Tom Eastep
The problem is caused by 'out-of-window' packets. So to totally analyze
a) The SCP stream on the outer interface of the other firewall.
b) The SCP stream on the outer interface of the Shorewall box.
c) Invalid connection state packets (I sent instructions earlier).
Once you find out which packets are being dropped (c), then you can
compare those packets in (a) and (b) to see if the other firewall is
mangling them. If not, then you need to send (b) and (c) to the
netfilter developers for analysis.

OK, thanks for the info. Alas, (a) isn't accesible to me

You could also capture on the client system.

-Tom

[Roberto C. Sánchez]

Post by Jonathan Underwood
220107.tar.bz2 0% 9464KB 492.9KB/s - stalled -
When it stalls, it periodically has bursts of a few KB/s and then
returns to zero. Normally on this link without that line in the rules
file, I can get 5 MB/s.
I'm still not sure I answered your questions though.

That helps. If it is stalled, that means that scp (ssh, in fact) still
thinks that the connection is open. That must mean that shorewall is in
fact stopping the packets. Of course, this is strange, since I also
have ssh rate limited on my machine and have not seen such a thing
before. What happens if you use sftp and get the file instead of scp?

Also, what version of ssh are you running on the server and also on the
client?

Regards,

-Roberto

[Jonathan Underwood]

Post by Roberto C. Sánchez
That helps. If it is stalled, that means that scp (ssh, in fact) still
thinks that the connection is open. That must mean that shorewall is in
fact stopping the packets. Of course, this is strange, since I also
have ssh rate limited on my machine and have not seen such a thing
before. What happens if you use sftp and get the file instead of scp?

sftp seems to work fine.

Post by Roberto C. Sánchez
Also, what version of ssh are you running on the server and also on the
client?

On both the server and the client I have openssh 4.3p2

J.

[Jonathan Underwood]

Post by Jonathan Underwood

Post by Roberto C. Sánchez
That helps. If it is stalled, that means that scp (ssh, in fact) still
thinks that the connection is open. That must mean that shorewall is in
fact stopping the packets. Of course, this is strange, since I also
have ssh rate limited on my machine and have not seen such a thing
before. What happens if you use sftp and get the file instead of scp?

sftp seems to work fine.

Oh, spoke too soon. I am now seeing sftp stalling as well.

[Roberto C. Sánchez]

Post by Jonathan Underwood

Post by Roberto C. Sánchez
That helps. If it is stalled, that means that scp (ssh, in fact) still
thinks that the connection is open. That must mean that shorewall is in
fact stopping the packets. Of course, this is strange, since I also
have ssh rate limited on my machine and have not seen such a thing
before. What happens if you use sftp and get the file instead of scp?

sftp seems to work fine.

Post by Roberto C. Sánchez
Also, what version of ssh are you running on the server and also on the
client?

On both the server and the client I have openssh 4.3p2

I forget exactly how it is configured, but you could you investigate
whether persistent or cached connections (I forget the exact term that
is used by the ssh folks) have been enabled/disabled?

Regards,

-Roberto

[Jonathan Underwood]

Post by Roberto C. Sánchez

Post by Jonathan Underwood

Post by Roberto C. Sánchez
That helps. If it is stalled, that means that scp (ssh, in fact) still
thinks that the connection is open. That must mean that shorewall is in
fact stopping the packets. Of course, this is strange, since I also
have ssh rate limited on my machine and have not seen such a thing
before. What happens if you use sftp and get the file instead of scp?

sftp seems to work fine.

Post by Roberto C. Sánchez
Also, what version of ssh are you running on the server and also on the
client?

On both the server and the client I have openssh 4.3p2

I forget exactly how it is configured, but you could you investigate
whether persistent or cached connections (I forget the exact term that
is used by the ssh folks) have been enabled/disabled?

Do you mean the ControlMaster stuff?
(http://www.torchbox.com/blog/ssh_tips_2.html).

I'm not making any use of that, there are no configuration entries in
/etc/ssh_config or my user ssh settings in .ssh/config so I don't
think that stuff is involved.

J.

Post by Roberto C. Sánchez
Regards,
-Roberto
--
Roberto C. Sánchez
http://people.connexer.com/~roberto
http://www.connexer.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFGVyI45SXWIKfIlGQRAi8HAKCzeTY2C1sr0SHetC7z0qx+6DylUQCcDfJ/
jXJNqclAWzoWOh2F6YXKawk=
=2ZfJ
-----END PGP SIGNATURE-----
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users

[Felix Erkinger]

Hi,

i have a setup where the firewall creates four pptp tunnels (austrian
provider setup for multiple ip addresses) and i have no idea how to
setup traffic shaping in this configuration

if i shape the encapsulated packets on the link to the modem, the
firewall shapes only GRE packets, so this doesnt work.

is there a possibility like to put all pptp tunnels to a bridge and
reroute them via traffic shaping or can i use a common limit on a bunch
of interfaces so the count together, or am i stuck ?

Thanks,
Felix

[Tom Eastep]

Post by Felix Erkinger
Hi,
i have a setup where the firewall creates four pptp tunnels (austrian
provider setup for multiple ip addresses) and i have no idea how to
setup traffic shaping in this configuration
if i shape the encapsulated packets on the link to the modem, the
firewall shapes only GRE packets, so this doesnt work.
is there a possibility like to put all pptp tunnels to a bridge and
reroute them via traffic shaping or can i use a common limit on a bunch
of interfaces so the count together, or am i stuck ?

I believe you are stuck.

-Tom

[Felix Erkinger]

Post by Tom Eastep

Post by Felix Erkinger
Hi,
i have a setup where the firewall creates four pptp tunnels (austrian
provider setup for multiple ip addresses) and i have no idea how to
setup traffic shaping in this configuration
if i shape the encapsulated packets on the link to the modem, the
firewall shapes only GRE packets, so this doesnt work.
is there a possibility like to put all pptp tunnels to a bridge and
reroute them via traffic shaping or can i use a common limit on a bunch
of interfaces so the count together, or am i stuck ?

I believe you are stuck.

Hmm, what about setting up a bridge (eg. "tonet"),
adding a private ip address to that bridge,
setting the default route to the ip address of that bridge,
mark all traffic for the corresponding pptp tunnels with fwmark x
create four ip rules that sorts packets with fwmark x to table ppp-x if
coming from dev tonet,
add four ip routes for the default gw of dev ppp-x and table ppp-x

Could this work ?

[Tom Eastep]

Post by Felix Erkinger

Post by Tom Eastep

Post by Felix Erkinger
Hi,
i have a setup where the firewall creates four pptp tunnels (austrian
provider setup for multiple ip addresses) and i have no idea how to
setup traffic shaping in this configuration
if i shape the encapsulated packets on the link to the modem, the
firewall shapes only GRE packets, so this doesnt work.
is there a possibility like to put all pptp tunnels to a bridge and
reroute them via traffic shaping or can i use a common limit on a bunch
of interfaces so the count together, or am i stuck ?

I believe you are stuck.

Hmm, what about setting up a bridge (eg. "tonet"),
adding a private ip address to that bridge,
setting the default route to the ip address of that bridge,
mark all traffic for the corresponding pptp tunnels with fwmark x
create four ip rules that sorts packets with fwmark x to table ppp-x if
coming from dev tonet,
add four ip routes for the default gw of dev ppp-x and table ppp-x
Could this work ?

I don't understand your proposal. What devices would you add to the
bridge (can't be PPtP devices)?

-Tom

[Felix Erkinger]

Post by Tom Eastep

Post by Felix Erkinger

Post by Tom Eastep

Post by Felix Erkinger
Hi,
i have a setup where the firewall creates four pptp tunnels (austrian
provider setup for multiple ip addresses) and i have no idea how to
setup traffic shaping in this configuration
if i shape the encapsulated packets on the link to the modem, the
firewall shapes only GRE packets, so this doesnt work.
is there a possibility like to put all pptp tunnels to a bridge and
reroute them via traffic shaping or can i use a common limit on a bunch
of interfaces so the count together, or am i stuck ?

I believe you are stuck.

Hmm, what about setting up a bridge (eg. "tonet"),
adding a private ip address to that bridge,
setting the default route to the ip address of that bridge,
mark all traffic for the corresponding pptp tunnels with fwmark x
create four ip rules that sorts packets with fwmark x to table ppp-x if
coming from dev tonet,
add four ip routes for the default gw of dev ppp-x and table ppp-x
Could this work ?

I don't understand your proposal. What devices would you add to the
bridge (can't be PPtP devices)?

hmm, none, i just poked arround, and added a bridge, with
"brctl add tonet"
and assigned an ip address to it with
"ifconfig tonet 10.10.0.1"
after that i can ping it, and assign the default gw route to it, so
there must be a ip stack answering the ping, so there could be a
possibility to reroute ?

if everything that want to get outside, gets rerouted via the bridge,
so my thinking, i could do traffic shapping on that bridge ?




and while poking arround

Post by Tom Eastep
-Tom
------------------------------------------------------------------------
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
------------------------------------------------------------------------
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users

[Tom Eastep]

Post by Felix Erkinger

Post by Tom Eastep

Post by Felix Erkinger

Post by Tom Eastep

Post by Felix Erkinger
Hi,
i have a setup where the firewall creates four pptp tunnels (austrian
provider setup for multiple ip addresses) and i have no idea how to
setup traffic shaping in this configuration
if i shape the encapsulated packets on the link to the modem, the
firewall shapes only GRE packets, so this doesnt work.
is there a possibility like to put all pptp tunnels to a bridge and
reroute them via traffic shaping or can i use a common limit on a bunch
of interfaces so the count together, or am i stuck ?

I believe you are stuck.

Hmm, what about setting up a bridge (eg. "tonet"),
adding a private ip address to that bridge,
setting the default route to the ip address of that bridge,
mark all traffic for the corresponding pptp tunnels with fwmark x
create four ip rules that sorts packets with fwmark x to table ppp-x if
coming from dev tonet,
add four ip routes for the default gw of dev ppp-x and table ppp-x
Could this work ?

I don't understand your proposal. What devices would you add to the
bridge (can't be PPtP devices)?

hmm, none, i just poked arround, and added a bridge, with
"brctl add tonet"
and assigned an ip address to it with
"ifconfig tonet 10.10.0.1"
after that i can ping it, and assign the default gw route to it, so
there must be a ip stack answering the ping, so there could be a
possibility to reroute ?

Not that I can see.

What this calls for is an Intermediate Queuing Device (IQD) attached to
your internal interface. Unfortunately, I've never been able to make one
work (but I haven't tried very hard either). Warning: It requires kernel
patching. See the LARTC Howto.

YMMV

-Tom

[Felix Erkinger]

Post by Tom Eastep

Post by Felix Erkinger

Post by Tom Eastep

Post by Felix Erkinger

Post by Tom Eastep

Post by Felix Erkinger
Hi,
i have a setup where the firewall creates four pptp tunnels (austrian
provider setup for multiple ip addresses) and i have no idea how to
setup traffic shaping in this configuration
if i shape the encapsulated packets on the link to the modem, the
firewall shapes only GRE packets, so this doesnt work.
is there a possibility like to put all pptp tunnels to a bridge and
reroute them via traffic shaping or can i use a common limit on a bunch
of interfaces so the count together, or am i stuck ?

I believe you are stuck.

Hmm, what about setting up a bridge (eg. "tonet"),
adding a private ip address to that bridge,
setting the default route to the ip address of that bridge,
mark all traffic for the corresponding pptp tunnels with fwmark x
create four ip rules that sorts packets with fwmark x to table ppp-x if
coming from dev tonet,
add four ip routes for the default gw of dev ppp-x and table ppp-x
Could this work ?

I don't understand your proposal. What devices would you add to the
bridge (can't be PPtP devices)?

hmm, none, i just poked arround, and added a bridge, with
"brctl add tonet"
and assigned an ip address to it with
"ifconfig tonet 10.10.0.1"
after that i can ping it, and assign the default gw route to it, so
there must be a ip stack answering the ping, so there could be a
possibility to reroute ?

Not that I can see.
What this calls for is an Intermediate Queuing Device (IQD) attached to
your internal interface. Unfortunately, I've never been able to make one
work (but I haven't tried very hard either). Warning: It requires kernel
patching. See the LARTC Howto.

i will investigate a little bit further regarding your information,
thank you for your help and trying to make sense out of my ideas :-) ,

Felix

[Felix Erkinger]

Hi,

i have a setup with a xen server and to interfaces, One to the public
net, and one a direct link to a other xen server used for intra domain
communication.

dom0 is setup with to bridges, netbr and clubr (cluster bridge), where
the dom0 only has a ip address on eth1 on the clubr bridge.

There is a domu with two virtual interfaces bound to both clubr and
netbr, which makes the firewalling, routing and natting.

Every domu has a interfaces connected to the clubr, so intra cluster
communication works, and outside access goes via the gateway domu.

There is a mailserver which gets the smtp/ssmtp ports from the public
interfaces natted to his internal address.
Im using shorewall 3.0.5.
Alltough a fancy setup, it works.

The problem with that is, if a internal domu (except the firewall domu
where it is working) wants to access the public mailserver via his
official address it doesnt get routed back to the internal address.

"picture":
web domu (1.2.3.3 -> smtp mail.mysetup.org (external)
->via-> gateway domu (1.2.3.1) ->via-> mail domu (1.2.3.2)
(doesnt work)

gateway domu rules:
# Route external firewall address:25 to internal mailserver
DNAT net clu:1.2.3.2:25 tcp 25
# experimental rule (doesnt work, currently disabled and makes strange
things)
DNAT clu clu:1.2.3.2:25 tcp 25

gateway domu interfaces:
net eth0 detect tcpflags,norfc1918
clu eth1 detect routeback

Thanks for any idea,

Felix

[Tom Eastep]

Post by Felix Erkinger
Hi,
i have a setup with a xen server and to interfaces, One to the public
net, and one a direct link to a other xen server used for intra domain
communication.
dom0 is setup with to bridges, netbr and clubr (cluster bridge), where
the dom0 only has a ip address on eth1 on the clubr bridge.
There is a domu with two virtual interfaces bound to both clubr and
netbr, which makes the firewalling, routing and natting.
Every domu has a interfaces connected to the clubr, so intra cluster
communication works, and outside access goes via the gateway domu.
There is a mailserver which gets the smtp/ssmtp ports from the public
interfaces natted to his internal address.
Im using shorewall 3.0.5.
Alltough a fancy setup, it works.
The problem with that is, if a internal domu (except the firewall domu
where it is working) wants to access the public mailserver via his
official address it doesnt get routed back to the internal address.
web domu (1.2.3.3 -> smtp mail.mysetup.org (external)
->via-> gateway domu (1.2.3.1) ->via-> mail domu (1.2.3.2)
(doesnt work)
# Route external firewall address:25 to internal mailserver
DNAT net clu:1.2.3.2:25 tcp 25
# experimental rule (doesnt work, currently disabled and makes strange
things)
DNAT clu clu:1.2.3.2:25 tcp 25
net eth0 detect tcpflags,norfc1918
clu eth1 detect routeback

This sounds like Shorewall FAQ #2.

-Tom

[Felix Erkinger]

Post by Tom Eastep
This sounds like Shorewall FAQ #2.

Aehhm, sorry, i think this solves it ....
(shame on me, for not searching enough)

Thank you,
Felix