Discussion:
[Shorewall-users] Shorewall behavior with flaky links
Norman Henderson
2017-06-20 06:01:32 UTC
Permalink
We have three internet connections at the moment, one of which is stable
but costly per Gb, and the other two cheaper and uncapped but highly prone
to failures - usually somewhere upstream in their networks. These can
manifest as slowdowns, high lost packet rates, or complete failures.
(Welcome to Africa...)

We have complex route_rules which prefer the cheaper providers but allow
(some users) access to the costly provider if the normal ones are disabled
in Shorewall.

1) Sometimes a particular user (usually me!) needs to use the costly
provider for a while, although other users have to suffer the normal
providers. Since it's a pain to change the route_rules file and restart
Shorewall (with implications for tunnels and other things too), I have just
been deleting the relevant "ip rule" to give myself access to the costly
provider. The trouble is that these rules come back again all by
themselves, after a random delay but sometimes just a few minutes.

Question: what Shorewall features / behaviors trigger the ip rules to be
reinitialized? Is there something I could do (or stop doing) to prevent
this happening?

2) FOOLSM suffers from the limitation that it will only monitor an on-net
address i.e. if the interface address is 192.168.0.1/24, it will only
monitor a remote address in that same subnet. Question: does anyone know of
a different monitoring solution that will enable and disable providers
based on an end-to-end path (such as to my virtual server in the USA)?

3) I have experimented with monitoring scripts of my own but there is a
conundrum:
- if a provider is flaky it needs to be disabled, otherwise it remains the
chosen route for users and they don't get any Internet.
- Meaningful testing of the status of a path require the interface to be
enabled. For example, pings over an interface that is up, but for which the
provider is marked "down" in Shorewall alternately succeed or report
"Operation not permitted".
- therefore, testing the status of a path requires frequent enabling and
disabling (if it is found to still be flaky) which in turn introduces brief
but annoying route changes that mess with users' ability to access the 'net.
Question: Any ideas for how to disable a provider and the corresponding
route_rules (for users) and yet still allow the firewall itself to reliably
use that interface? At least for pings and maybe for more complex tests
e.g. speedtest.net.

Thanks in advance!
Tom Eastep
2017-06-20 15:35:03 UTC
Permalink
Post by Norman Henderson
We have three internet connections at the moment, one of which is stable
but costly per Gb, and the other two cheaper and uncapped but highly
prone to failures - usually somewhere upstream in their networks. These
can manifest as slowdowns, high lost packet rates, or complete failures.
(Welcome to Africa...)
We have complex route_rules which prefer the cheaper providers but allow
(some users) access to the costly provider if the normal ones are
disabled in Shorewall.
1) Sometimes a particular user (usually me!) needs to use the costly
provider for a while, although other users have to suffer the normal
providers. Since it's a pain to change the route_rules file and restart
Shorewall (with implications for tunnels and other things too), I have
just been deleting the relevant "ip rule" to give myself access to the
costly provider. The trouble is that these rules come back again all by
themselves, after a random delay but sometimes just a few minutes.
Question: what Shorewall features / behaviors trigger the ip rules to be
reinitialized? Is there something I could do (or stop doing) to prevent
this happening?
2) FOOLSM suffers from the limitation that it will only monitor an
on-net address i.e. if the interface address is 192.168.0.1/24
<http://192.168.0.1/24>, it will only monitor a remote address in that
same subnet. Question: does anyone know of a different monitoring
solution that will enable and disable providers based on an end-to-end
path (such as to my virtual server in the USA)?
FOOLSM does not have such a limitation -- it can monitor any address,
provided that you add the appropriate route for that address.

In my own configuration, I have this in /etc/network/interfaces:

post-up /sbin/ip route add 96.120.100.113/32 via 10.2.10.1 dev eth0
post-up /sbin/ip route add 76.28.230.188/32 via 10.1.10.1 dev eth1

Then, in FOOLSM, I monitor 96.120.100.113 through eth0 and 76.28.230.188
through eth1. This requires the appropriate TTL to be configured for
each in FOOLSM and it requires that both providers be specified as
'persistent' in /etc/shorewall/providers
Post by Norman Henderson
3) I have experimented with monitoring scripts of my own but there is a
- if a provider is flaky it needs to be disabled, otherwise it remains
the chosen route for users and they don't get any Internet.
- Meaningful testing of the status of a path require the interface to be
enabled. For example, pings over an interface that is up, but for which
the provider is marked "down" in Shorewall alternately succeed or report
"Operation not permitted".
Use 'persistent' in /etc/shorewall/providers.
Post by Norman Henderson
- therefore, testing the status of a path requires frequent enabling and
disabling (if it is found to still be flaky) which in turn introduces
brief but annoying route changes that mess with users' ability to access
the 'net.
Question: Any ideas for how to disable a provider and the corresponding
route_rules (for users) and yet still allow the firewall itself to
reliably use that interface? At least for pings and maybe for more
complex tests e.g. speedtest.net <http://speedtest.net>.
Again, use 'persistent'.

-Tom
--
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________
Norman Henderson
2017-06-21 06:21:57 UTC
Permalink
Thank you for this Tom. The "persistent" option is a good help, although it
has the side effect that on reload etc. Shorewall will try to re-enable the
provider, which is not desirable if it is flaky e.g. can be enabled but
won't pass traffic.

The change of name from lsm to foolsm has created some issues, in my
installation (just upgraded to 5.1.4.3) shorewall is still referring for
example to start_lsm rather than start_foolsm. Worked around for now using
start_lsm() { start_foolsm; } in lib.private.

Yes I see that I was incorrect, foolsm can in fact ping any address
however, there is a statement somewhere that the source address for pings
is always autodiscovered. If that is true, it isn't possible to ping from a
determined interface except to an on-net address.

Adding a route for a specific target address doesn't seem like a very clean
solution and creates other problems because the target address is not only
for monitoring, it is also used to establish tunnels etc. and for those
uses I need to be able to choose the "best" provider that is up.
Furthermore that implies that I need a different target address for each
provider, again not so good.

On the other hand, I see there is a "sourceip=" option in foolsm.conf, if
that works then I should be OK. I am trying that but will have to wait for
at least one of the flaky providers to come up again (!)

Finally: the suggested eventscript for foolsm uses argument ${4} as the
${DEVICE} which is used for firewall enable / disable. I am seeing that
when the script gets called that argument is null. I am using the ${NAME}
instead for now, but any idea what might cause that?
Post by Tom Eastep
Post by Norman Henderson
We have three internet connections at the moment, one of which is stable
but costly per Gb, and the other two cheaper and uncapped but highly
prone to failures - usually somewhere upstream in their networks. These
can manifest as slowdowns, high lost packet rates, or complete failures.
(Welcome to Africa...)
We have complex route_rules which prefer the cheaper providers but allow
(some users) access to the costly provider if the normal ones are
disabled in Shorewall.
1) Sometimes a particular user (usually me!) needs to use the costly
provider for a while, although other users have to suffer the normal
providers. Since it's a pain to change the route_rules file and restart
Shorewall (with implications for tunnels and other things too), I have
just been deleting the relevant "ip rule" to give myself access to the
costly provider. The trouble is that these rules come back again all by
themselves, after a random delay but sometimes just a few minutes.
Question: what Shorewall features / behaviors trigger the ip rules to be
reinitialized? Is there something I could do (or stop doing) to prevent
this happening?
2) FOOLSM suffers from the limitation that it will only monitor an
on-net address i.e. if the interface address is 192.168.0.1/24
<http://192.168.0.1/24>, it will only monitor a remote address in that
same subnet. Question: does anyone know of a different monitoring
solution that will enable and disable providers based on an end-to-end
path (such as to my virtual server in the USA)?
FOOLSM does not have such a limitation -- it can monitor any address,
provided that you add the appropriate route for that address.
post-up /sbin/ip route add 96.120.100.113/32 via 10.2.10.1 dev eth0
post-up /sbin/ip route add 76.28.230.188/32 via 10.1.10.1 dev eth1
Then, in FOOLSM, I monitor 96.120.100.113 through eth0 and 76.28.230.188
through eth1. This requires the appropriate TTL to be configured for
each in FOOLSM and it requires that both providers be specified as
'persistent' in /etc/shorewall/providers
Post by Norman Henderson
3) I have experimented with monitoring scripts of my own but there is a
- if a provider is flaky it needs to be disabled, otherwise it remains
the chosen route for users and they don't get any Internet.
- Meaningful testing of the status of a path require the interface to be
enabled. For example, pings over an interface that is up, but for which
the provider is marked "down" in Shorewall alternately succeed or report
"Operation not permitted".
Use 'persistent' in /etc/shorewall/providers.
Post by Norman Henderson
- therefore, testing the status of a path requires frequent enabling and
disabling (if it is found to still be flaky) which in turn introduces
brief but annoying route changes that mess with users' ability to access
the 'net.
Question: Any ideas for how to disable a provider and the corresponding
route_rules (for users) and yet still allow the firewall itself to
reliably use that interface? At least for pings and maybe for more
complex tests e.g. speedtest.net <http://speedtest.net>.
Again, use 'persistent'.
-Tom
--
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users
Tom Eastep
2017-06-21 15:15:07 UTC
Permalink
Post by Norman Henderson
Thank you for this Tom. The "persistent" option is a good help, although
it has the side effect that on reload etc. Shorewall will try to
re-enable the provider, which is not desirable if it is flaky e.g. can
be enabled but won't pass traffic.
That isn't intentional. I'll take a look.
Post by Norman Henderson
The change of name from lsm to foolsm has created some issues, in my
installation (just upgraded to 5.1.4.3) shorewall is still referring for
example to start_lsm rather than start_foolsm. Worked around for now
using start_lsm() { start_foolsm; } in lib.private.
lib.private is a user-supplied library. So, yes, you needed to change it
when lsm was renamed.
Post by Norman Henderson
Yes I see that I was incorrect, foolsm can in fact ping any address
however, there is a statement somewhere that the source address for
pings is always autodiscovered. If that is true, it isn't possible to
ping from a determined interface except to an on-net address.
Not true -- you specify the interface, and the address is then
autodiscovered ON THAT INTERFACE.
Post by Norman Henderson
Adding a route for a specific target address doesn't seem like a very
clean solution and creates other problems because the target address is
not only for monitoring, it is also used to establish tunnels etc. and
for those uses I need to be able to choose the "best" provider that is
up. Furthermore that implies that I need a different target address for
each provider, again not so good.
Any monitor that you use will have those restrictions.
Post by Norman Henderson
On the other hand, I see there is a "sourceip=" option in foolsm.conf,
if that works then I should be OK. I am trying that but will have to
wait for at least one of the flaky providers to come up again (!)
Finally: the suggested eventscript for foolsm uses argument ${4} as the
${DEVICE} which is used for firewall enable / disable. I am seeing that
when the script gets called that argument is null. I am using the
${NAME} instead for now, but any idea what might cause that?
Afraid not.

-Tom
--
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________
Tom Eastep
2017-06-21 15:19:27 UTC
Permalink
Post by Tom Eastep
Post by Norman Henderson
Thank you for this Tom. The "persistent" option is a good help, although
it has the side effect that on reload etc. Shorewall will try to
re-enable the provider, which is not desirable if it is flaky e.g. can
be enabled but won't pass traffic.
That isn't intentional. I'll take a look.
I'm not seeing that behavior:

***@gateway:~# shorewall disable eth0
Provider IPv6Beta (1) stopped
***@gateway:~# shorewall reload
Reloading Shorewall....
Initializing...
Setting up ARP filtering...
Setting up Route Filtering...
Setting up Martian Logging...
Setting up Accept Source Routing...
Setting up log backend
Setting up Proxy ARP...
Adding Providers...
Null Routing the RFC 1918 subnets
WARNING: Interface eth0 is not usable -- Provider IPv6Beta (1) not
Started
WARNING: No Default route added (all 'balance' providers are down)
Preparing iptables-restore input...
Running /sbin/iptables-restore ...
IPv4 Forwarding Enabled
Processing /etc/shorewall/started ...
done.
***@gateway:~# shorewall enable eth0
Provider IPv6Beta (1) Started
***@gateway:~#

-Tom
--
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________
Norman Henderson
2017-06-21 16:37:17 UTC
Permalink
This post might be inappropriate. Click to display it.
Tom Eastep
2017-06-21 21:19:41 UTC
Permalink
Post by Norman Henderson
Thanks very much Tom, see further comments below.
Post by Norman Henderson
Thank you for this Tom. The "persistent" option is a good help, although
it has the side effect that on reload etc. Shorewall will try to
re-enable the provider, which is not desirable if it is flaky e.g. can
be enabled but won't pass traffic.
That isn't intentional. I'll take a look.
I was basing the comment on the following
from shorewall.net/manpages/shorewall-providers.html
Note
The generated script will attempt to reenable a disabled persistent
provider during execution of
the *start*, *restart* and *reload* commands. When |persistent| is not
specified, only the *enable* and *reenable* commands can reenable the
provider.
That behavior is overridden by the isusable extension script shown in
the Multi-ISP documentation.

-Tom
--
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________
Loading...