[Shorewall-users] traffic does not flow through firewall/router

[Simon Matter]

Hi,
I'm trying to update to shorewall 5.1 with a config that is *supposedly*
working with 5.0.
In any case, I'm trying to ping from a host in lan zone with IP addr.
10.215.144.48 to a host in IBS zone with IP addr. 10.215.9.172.
ICMP traffic should be allowed but the client isn't receiving any replies.
I'm attaching the shorewall dump.
Jun 15 07:52:10 inf-fw2 root[32520]: Shorewall Stopped
Jun 15 07:52:11 inf-fw2 root[900]: Shorewall started
/var/log/shorewall-init.log doesn't seem to contain any error messages.
Please note that this shorewall box was supposed to replace another one
with the same IP address (it's the default gateway/firewall).
So I merely unplugged the ethernet cables from the "old" shorewall box and
plugged them into the new one.
It didn't occurr to me to try and ping $FW from a lan host or connect via
ssh.
However, from within the $FW console I could ping to any host IP addresses
in all "zones".
The switch happened at 07:45:05 and had to revert to the old FW at
07:52:11 because the users were already complaining.
Could there be an arp cache issue?

Exactly, what about the rest of the network, switches/routers, how do they
know about the FW change? (I guess the easiest solution would be to simply
reboot those devices after the FW change)

Simon

[Simon Matter]

________________________________

Post by Simon Matter
Exactly, what about the rest of the network, switches/routers, how do they
know about the FW change? (I guess the easiest solution would be to
simply> reboot those devices after the FW change)

Note that I've kept the new FW online for more than 5 minutes.
I'm not sure yet when an ARP entry times out in my network devices (I'll
need to check on each and every switch firmware), but in Linux it should
/proc/sys/net/ipv4/neigh/default/gc_stale_time
I'm only assuming the other network devices have similar settings, but I
guess I'll need to check thoroughly.

I remember a case with an externally controlled Cisco router where are
timeout was 1h. So better check.

BTW, any chance you are using proxy ARP on the shorewall FW? This can also
lead to such issues IIRC.

Simon

[Philip Le Riche]

We have Shorewall 4 protecting the school network from a group of
Raspberry Pis, which we operate from PCs on the school network using VNC
running through Shorewall. For some weeks we've had frequent problems
with VNC sessions hanging for around 30 seconds. I've been trying to
track it down with increasingly focussed Wireshark captures, and this is
what seems to be happening on one fairly typical hang:

Two Pis are being controlled from separate PCs. I have ping running from
the firewall to one of the Pis and also from the firewall to the default
gateway on the school network.

Hundreds of packets are passing through the firewall from one of the Pis
to the PC controlling it, containing VNC screen update data. These are
interspersed every second by a ping/reply to one of the Pis and a
ping/reply to the default gateway.

Suddenly TCP retransmissions of VNC traffic start appearing. Often at
this point you see one or two other packets, such as an ntp or a VNC
from the other Pi, but this may only be because they're no longer being
hidden amongst a mass of VNC.

More retransmissions from the Pi(s) but nothing on the school network
NIC, and in particular, no pings to the default gateway.

After around 10 seconds, the Pi network NIC sends ICMP network
unreachable to both Pis.

Sometimes I've seen ICMP host unreachable, I think from the school
network NIC back to a Pi. Other times I've seen RST, ACK packets from
one of the VNC client PCs - I don't see RST, ACK in the standard TCP
state diagram.

After a total of around 30 seconds, everything seems to recover, and
pings reappear on the school network, though VNC generally has to open
a new TCP connection.

Only fairly recently have we regularly run more than one Pi at the same
time. Maybe we're just running out of kernel buffers? Or we need a more
powerful machine to run Shorewall? (It's an unremarkable desktop machine
maybe 5 years old.) Or maybe I've just got something misconfigured.
Ideas please?

Regards - Philip

[Tom Eastep]

Post by Philip Le Riche
We have Shorewall 4 protecting the school network from a group of
Raspberry Pis, which we operate from PCs on the school network
using VNC running through Shorewall. For some weeks we've had
frequent problems with VNC sessions hanging for around 30 seconds.
I've been trying to track it down with increasingly focussed
Wireshark captures, and this is what seems to be happening on one
Two Pis are being controlled from separate PCs. I have ping running
from the firewall to one of the Pis and also from the firewall to
the default gateway on the school network.
Hundreds of packets are passing through the firewall from one of
the Pis to the PC controlling it, containing VNC screen update
data. These are interspersed every second by a ping/reply to one of
the Pis and a ping/reply to the default gateway.
Suddenly TCP retransmissions of VNC traffic start appearing. Often
at this point you see one or two other packets, such as an ntp or a
VNC from the other Pi, but this may only be because they're no
longer being hidden amongst a mass of VNC.
More retransmissions from the Pi(s) but nothing on the school
network NIC, and in particular, no pings to the default gateway.
After around 10 seconds, the Pi network NIC sends ICMP network
unreachable to both Pis.
Sometimes I've seen ICMP host unreachable, I think from the school
network NIC back to a Pi. Other times I've seen RST, ACK packets
from one of the VNC client PCs - I don't see RST, ACK in the
standard TCP state diagram.
After a total of around 30 seconds, everything seems to recover,
and pings reappear on the school network, though VNC generally has
to open a new TCP connection.
Only fairly recently have we regularly run more than one Pi at the
same time. Maybe we're just running out of kernel buffers? Or we
need a more powerful machine to run Shorewall? (It's an
unremarkable desktop machine maybe 5 years old.) Or maybe I've just
got something misconfigured. Ideas please?

Are you monitoring ARP traffic between the Shorewall box and the
School network?

- -Tom
- --
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________

[Philip Le Riche]

Post by Tom Eastep

Post by Philip Le Riche
We have Shorewall 4 protecting the school network from a group of
Raspberry Pis, which we operate from PCs on the school network
using VNC running through Shorewall. For some weeks we've had
frequent problems with VNC sessions hanging for around 30 seconds.
I've been trying to track it down with increasingly focussed
Wireshark captures, and this is what seems to be happening on one

... snip...

Post by Tom Eastep
Are you monitoring ARP traffic between the Shorewall box and the
School network?
-Tom

ARP was about the first thing that I filtered out in my capture filter
as there was so much of it on the school network. Is this significant?

Regards - Philip

[Tom Eastep]

Post by Philip Le Riche

Post by Tom Eastep

Post by Philip Le Riche
We have Shorewall 4 protecting the school network from a group
of Raspberry Pis, which we operate from PCs on the school
network using VNC running through Shorewall. For some weeks
we've had frequent problems with VNC sessions hanging for
around 30 seconds. I've been trying to track it down with
increasingly focussed Wireshark captures, and this is what

... snip...

Post by Tom Eastep
Are you monitoring ARP traffic between the Shorewall box and the
School network?
-Tom

ARP was about the first thing that I filtered out in my capture
filter as there was so much of it on the school network. Is this
significant?

It is the first thing that I would want to look at when traffic
suddenly stops then later starts again.

Also, are you seeing any errors (ip -s link ls) on the school network NIC?

- -Tom
- --
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________

[Tom Eastep]

Hi,
I'm trying to update to shorewall 5.1 with a config that is
*supposedly* working with 5.0.
In any case, I'm trying to ping from a host in lan zone with IP
addr. 10.215.144.48 to a host in IBS zone with IP addr.
10.215.9.172. ICMP traffic should be allowed but the client isn't
receiving any replies. I'm attaching the shorewall dump.
/var/log/shorewall/info.log only has messages of this kind when
Jun 15 07:52:10 inf-fw2 root[32520]: Shorewall Stopped Jun 15
07:52:11 inf-fw2 root[900]: Shorewall started
/var/log/shorewall-init.log doesn't seem to contain any error
messages.
Please note that this shorewall box was supposed to replace another
one with the same IP address (it's the default gateway/firewall).
So I merely unplugged the ethernet cables from the "old" shorewall
box and plugged them into the new one. It didn't occurr to me to
try and ping $FW from a lan host or connect via ssh. However, from
within the $FW console I could ping to any host IP addresses in all
"zones".
The switch happened at 07:45:05 and had to revert to the old FW at
07:52:11 because the users were already complaining.
Could there be an arp cache issue?

Definitely.

- -Tom
- --
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________