[Shorewall-users] Question - ipsets in route

Discussion:

[Shorewall-users] Question - ipsets in route_rules

Norman Henderson

2017-05-05 16:52:38 UTC

Hi, I'm experimenting with this, trying to simplify a situation with
several unreliable providers. Basically, we have two uncapped providers
that tend to go down (upstream failures) or suffer poor performance
(delayed and dropped packets) at unpredictable times; and another provider
that is stable but, capped, and quite expensive per Gb. So we want to use
the uncapped providers for everyone when they are good, and switch some
"special" users to the per-Gb provider when the uncapped ones are bad.

One solution I have tried is to simply delete some of the rules that
shorewall creates (ip rule del pref ...). Even though I'm not aware of any
trigger for shorewall to have been restarted, periodically these rules
reappear.

I put in route_rules:
+unitelusers - uni01 1295
(uni01 being a high-cost but more reliable provider)

The ipset unitelusers was defined with
ipset create unitelusers hash:net
ipset add unitelusers 10.1.0.0/24
...and some other internal addresses and subnets.

The resulting ip rule is:
1295: from all iif +unitelusers [detached] lookup uni01
That doesn't look right, because my ipset contains nets, not interfaces...

I also tried, in route_rules:
vlan4:+unitelusers - uni01 1295
That returns:
ERROR: An ipset name (+unitelusers) is not allowed in this context

Are ipsets not supposed to work with route_rules, or am I missing something?

Thanks, Norm

Tom Eastep

2017-05-06 00:23:35 UTC

Permalink

Post by Norman Henderson
Are ipsets not supposed to work with route_rules, or am I missing something?

Bill Shirley

2017-05-07 05:12:12 UTC

Permalink

I don't have a providers file but I do have two internet providers and use ipsets. Perhaps this may help.
(Note I use variables defined in Shorewall params):

Shorewall mangle:
MARK($COMCAST_MARK1/$CONNMASK):P - +$COMCAST1_IPSET

ip rule:
.
.
10101: from all fwmark 0x4000/0x3ff00 lookup Comcast_ip1

If you run a Red Hat distro, you can create a file /etc/sysconfig/network-scripts/rule-eth1 which will add
the rule when the interface comes up:
fwmark 0x4000/0x3f000 lookup Comcast_ip1 pri 10101
and /etc/sysconfig/network-scripts/route-eth1:
default via 173.xxx.y.254 dev ccast proto static src 173.xxx.y.249
and of course, your table name(Comcast_ip1) has to be defined in /etc/iproute2/rt_tables.

So in the mangle rule instead of +$COMCAST1_IPSET, you would use +unitelusers. Perhaps you
can translate this into provider marks. Set the rule priority appropriately also.

One additional thought: you might mark your low priority services to use your 2nd ISP and just wait
until it comes back up:
Shorewall mangle:
MARK($BUDGET_ISP/$CONNMASK) $FW - tcp smtp

HTH,
Bill

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Post by Norman Henderson
Are ipsets not supposed to work with route_rules, or am I missing something?

Ipsets are not supported in rtrules -- this is a Linux networking
restriction, independent of Shorewall.
- -Tom
- --
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.net \________________________________________________
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJZDReHAAoJEJbms/JCOk0QIXMP/RpLh6Dl5fjOw9AwaN0nqlvY
NUw6OOpc3gJJoH+yvNFVIs8d5jl/+kGVLJuWE4qBz2Br59T5upFn9AUtocX31H0K
N7zpc4OU9trx2arnPVVdvR8xksPi0ZtTF7hvkz0B3ce2cgKOh2SeSR3xMRxQkOCc
VMUSckhQ0niz/9txk1BxKV1rG3+5x+pbpPNdI4GN0HHICafTBihJiauJ1gxz54qj
00k3PhdNIZWCdiwdi8Z/Y3OuSzIXuPK6paET6LtfFI9GpwkQ+7kz2NE7QSyUX8Xc
hKeKzWw7nQSsKLdhRwcZBkU0xFhBHdCqZkespBTtpzVnnlJSfJ1cyrBqTz4ExP+2
L3oBc0RNi0iSv5nPnf3ri7kJMBiJfuNVJc6yEnPx+Sr8n+BezMIudW9Q3F/zZqRI
YWDm/OyhYmiUSpMXta4VwJlF1g2V1xvt/e4pAhXdYUJKLxjlRI5k5WdzDyMKxfoP
3NuwBPZe5M4D5vRbgcmb95YMrZO5FPWqJADuQWppi3QEfHRm7qEWWFH1vZBAjsl6
DpsSYh2GzwRXJaLZ7M4eHILWceKhNtfxJ3uqMiW0aQr8LnSFh/lsTukTSDz1IrEd
cqtrX2MdV6Iu37bjM/FnTXnlmfZu/jR2wzj6x3/9YNa5qFJW6EAEopzolWqF2I0Y
ABWSWQ+a9bJfni7AgqtV
=Vgt3
-----END PGP SIGNATURE-----
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users

Norman Henderson

2017-05-07 16:53:34 UTC

Permalink

Thank you Bill, that's immensely helpful. Just curious why you are using a
mask for the marks? And why in particular, 0x3f000?
- Norm

On Sun, May 7, 2017 at 6:12 AM, Bill Shirley <

Post by Bill Shirley
I don't have a providers file but I do have two internet providers and use
ipsets. Perhaps this may help.
MARK($COMCAST_MARK1/$CONNMASK):P - +$COMCAST1_IPSET
.
.
10101: from all fwmark 0x4000/0x3ff00 lookup Comcast_ip1
If you run a Red Hat distro, you can create a file /etc/sysconfig/network-scripts/rule-eth1
which will add
fwmark 0x4000/0x3f000 lookup Comcast_ip1 pri 10101
default via 173.xxx.y.254 dev ccast proto static src 173.xxx.y.249
and of course, your table name(Comcast_ip1) has to be defined in /etc/iproute2/rt_tables.
So in the mangle rule instead of +$COMCAST1_IPSET, you would use +unitelusers. Perhaps you
can translate this into provider marks. Set the rule priority
appropriately also.
One additional thought: you might mark your low priority services to use
your 2nd ISP and just wait
MARK($BUDGET_ISP/$CONNMASK) $FW - tcp smtp
HTH,
Bill

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Post by Norman Henderson
Are ipsets not supposed to work with route_rules, or am I missing something?

------------------

Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users

Bill Shirley

2017-05-08 22:09:23 UTC

Permalink

From my shorewall.conf:
TC_BITS=8
MASK_BITS=8
PROVIDER_OFFSET=24
PROVIDER_BITS=0
ZONE_BITS=5
# m Shorewall event mark
# t Shorewall tproxy
# x Shorewall exclusion
# z Shorewall zone bits
# j JUNK_MARK for dianostics
# o NEW_SERVER_OUTPUT - server's output chain
# u unsed bits
# p network-daemon ping mark
# i IPSEC VPN
# c connection bits
# s traffic shaping
# ------------------------------------------------------------------------------
# 2 2 2 1
# 9 4 0 6 8 0
# mtxzzzzzjouuuupiccccccccssssssss
# ------------------------------------------------------------------------------
The connection bits are broken down into 4 bits for the interface and 4 for the IP address.

0x4000/0x3ff00 is interface #4, IP address 0.
0x4100/0x3ff00 is interface #4, IP address 1.

Shorewall params:
CONNMASK=0x3ff00 # change this - change network-daemon
NO_ND_CONNMASK=0x1ff00 # mask without ND ping bit

IF_MASK=0x3f000 # interface mask (any IP address)

IPSEC_MARK=0x10000 # 65536
# { test=$IPSET_NO_GRP_MARK/$IPSEC_NO_GRP_MASK:C} means IPSEC but no group
IPSET_NO_GRP_MARK=0x10000
IPSEC_NO_GRP_MASK=0x18000

ND_PING_MASK=0x20000 # 131072

NEW_SERVER_OUTPUT=0x400000 # 2097152

JUNK_MARK=0x800000 # 8388608

INET2_MARK1=0x4000 # 16384
INET2_IP1=173.xxx.yyy.249
INET2_MARK2=0x4100 # 16640
INET2_IP2=173.xxx.yyy.250

I have a network daemon that pings each ISP connection and VPN connection. There is a rule
that is always available for ping with a mark (-m 16384, ping requires decimal):
5101: from all fwmark 0x24000/0x3ff00 lookup ping_Comcast
and rules that get removed if an interface goes down:
10101: from all fwmark 0x4000/0x3ff00 lookup Comcast_ip1
10102: from all fwmark 0x4100/0x3ff00 lookup Comcast_ip2
32101: from all lookup Comcast_ip1

HTH,
Bill

Thank you Bill, that's immensely helpful. Just curious why you are using a mask for the marks? And why in particular, 0x3f000?
- Norm
I don't have a providers file but I do have two internet providers and use ipsets. Perhaps this may help.
MARK($COMCAST_MARK1/$CONNMASK):P - +$COMCAST1_IPSET
.
.
10101: from all fwmark 0x4000/0x3ff00 lookup Comcast_ip1
If you run a Red Hat distro, you can create a file /etc/sysconfig/network-scripts/rule-eth1 which will add
fwmark 0x4000/0x3f000 lookup Comcast_ip1 pri 10101
default via 173.xxx.y.254 dev ccast proto static src 173.xxx.y.249
and of course, your table name(Comcast_ip1) has to be defined in /etc/iproute2/rt_tables.
So in the mangle rule instead of +$COMCAST1_IPSET, you would use +unitelusers. Perhaps you
can translate this into provider marks. Set the rule priority appropriately also.
One additional thought: you might mark your low priority services to use your 2nd ISP and just wait
MARK($BUDGET_ISP/$CONNMASK) $FW - tcp smtp
HTH,
Bill

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Post by Norman Henderson
Are ipsets not supposed to work with route_rules, or am I missing something?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users <https://lists.sourceforge.net/lists/listinfo/shorewall-users>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Shorewall-users mailing list
https://lists.sourceforge.net/lists/listinfo/shorewall-users

Bill Shirley

2017-05-08 22:30:10 UTC

Permalink

Sorry, that should read:
is always available for ping with a mark (-m 147456, ping requires decimal)

which is 0x20000 + 0x4000 which matches:

5101: from all fwmark 0x24000/0x3ff00 lookup ping_Comcast

Bill

Post by Bill Shirley
TC_BITS=8
MASK_BITS=8
PROVIDER_OFFSET=24
PROVIDER_BITS=0
ZONE_BITS=5
# m Shorewall event mark
# t Shorewall tproxy
# x Shorewall exclusion
# z Shorewall zone bits
# j JUNK_MARK for dianostics
# o NEW_SERVER_OUTPUT - server's output chain
# u unsed bits
# p network-daemon ping mark
# i IPSEC VPN
# c connection bits
# s traffic shaping
# ------------------------------------------------------------------------------
# 2 2 2 1
# 9 4 0 6 8 0
# mtxzzzzzjouuuupiccccccccssssssss
# ------------------------------------------------------------------------------
The connection bits are broken down into 4 bits for the interface and 4 for the IP address.
0x4000/0x3ff00 is interface #4, IP address 0.
0x4100/0x3ff00 is interface #4, IP address 1.
CONNMASK=0x3ff00 # change this - change network-daemon
NO_ND_CONNMASK=0x1ff00 # mask without ND ping bit
IF_MASK=0x3f000 # interface mask (any IP address)
IPSEC_MARK=0x10000 # 65536
# { test=$IPSET_NO_GRP_MARK/$IPSEC_NO_GRP_MASK:C} means IPSEC but no group
IPSET_NO_GRP_MARK=0x10000
IPSEC_NO_GRP_MASK=0x18000
ND_PING_MASK=0x20000 # 131072
NEW_SERVER_OUTPUT=0x400000 # 2097152
JUNK_MARK=0x800000 # 8388608
INET2_MARK1=0x4000 # 16384
INET2_IP1=173.xxx.yyy.249
INET2_MARK2=0x4100 # 16640
INET2_IP2=173.xxx.yyy.250
I have a network daemon that pings each ISP connection and VPN connection. There is a rule
5101: from all fwmark 0x24000/0x3ff00 lookup ping_Comcast
10101: from all fwmark 0x4000/0x3ff00 lookup Comcast_ip1
10102: from all fwmark 0x4100/0x3ff00 lookup Comcast_ip2
32101: from all lookup Comcast_ip1
HTH,
Bill

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Post by Norman Henderson
Are ipsets not supposed to work with route_rules, or am I missing
something?