[Shorewall-users] Distributing outgoing packets over multiple interfaces

[Mark Morgan Lloyd]

Apologies if this turns out to be an FAQ, but I'm having trouble getting
to grips with things.

I've got a Raspberry Pi (little ARM box) here running Debian "Jessie"
with the as-supplied Shorewall 4.6. As well as eth0 (192.168.1.5) as the
"internal" side of the router, it's using VLANs on eth1 to provide 2x
upstream interfaces (172.27.200.5 and 172.27.201.5) plus some more
reserved for DMZ systems (90.155.84.x).

The upstream systems are configured with 4G wireless (carrier-grade NAT)
and L2TP endpoints. Incoming traffic on the L2TP can see the DMZ. Custom
iptables and tc stuff on these systems ensures that traffic originating
on a routable address (90.155.84.x) goes out over the tunnel, but
anything on an RFC-1918 address goes over carrier-grade NAT on the 4G.
Setting this up was "fun", but these aren't causing me significant problems.

At our ISP, the L2TP tunnels are bonded on a Firebrick router. I think
it's reasonable to assume that they know what they're doing, since they
design and manufacture the things :-)

How do I tell Shorewall on the Raspberry Pi to do this:

* Anything originating at an internal RFC-1918 address is to be routed
over one or the other 172.27.x.x paths, hence over 4G with carrier-grade
NAT. Am I correct that this falls into the "multiple provider" basket?

* Anything originating at a DMZ routable address is to be split
proportionally onto both tunnels, i.e. a 50:50 bandwidth split.

* Incoming traffic over both tunnels is to be merged for the DMZ (SMTP
etc.).

I believe that our ISP monitors the performance of the tunnels and
balances traffic accordingly. I expect I'll have to do something similar
for this end at some point, but that's the least of my problems right now.

I've read the FAQ, support guide etc. but I think that what I need is an
initial kick in the right direction: I have a few years experience with
iptables etc. but am new to Shorewall.

I'm subscribed, CC not necessary.

[Simon Hobson]

Apologies if this turns out to be an FAQ, but I'm having trouble getting to grips with things.
I've got a Raspberry Pi (little ARM box) here running Debian "Jessie" with the as-supplied Shorewall 4.6. As well as eth0 (192.168.1.5) as the "internal" side of the router, it's using VLANs on eth1 to provide 2x upstream interfaces (172.27.200.5 and 172.27.201.5) plus some more reserved for DMZ systems (90.155.84.x).
The upstream systems are configured with 4G wireless (carrier-grade NAT) and L2TP endpoints. Incoming traffic on the L2TP can see the DMZ. Custom iptables and tc stuff on these systems ensures that traffic originating on a routable address (90.155.84.x) goes out over the tunnel, but anything on an RFC-1918 address goes over carrier-grade NAT on the 4G. Setting this up was "fun", but these aren't causing me significant problems.
At our ISP, the L2TP tunnels are bonded on a Firebrick router. I think it's reasonable to assume that they know what they're doing, since they design and manufacture the things :-)
* Anything originating at an internal RFC-1918 address is to be routed over one or the other 172.27.x.x paths, hence over 4G with carrier-grade NAT. Am I correct that this falls into the "multiple provider" basket?

Correct it's done in the routing rules.

* Anything originating at a DMZ routable address is to be split proportionally onto both tunnels, i.e. a 50:50 bandwidth split.

Not too sure about the 50:50 split - never done it that way. But I think the way to do it will be to define both providers with the balance option, and set at the same priority.

* Incoming traffic over both tunnels is to be merged for the DMZ (SMTP etc.).

Just put both interfaces into the one zone.
Merging of the traffic isn't a Shorewall function. The network stack does that, and it doesn't really care which interface a packet came in on, by default it just routes on destination - except for the source routing rules setup for the outbound traffic that is.

You probably want to remove your custom iptables and tc stuff and let Shorewall do it all. Otherwise you run real risks of having problems that are "interesting" to diagnose.

My thoughts are that you'll want 4 providers - 2 using the native 4G, 2 using the tunnels.
As I read it, you won't be doing any NAT/Masq yourself - that's being done by the carrier.
Then configure your routing rules alon the lines of :
source=RFC1918, route via 4G-1 and 4G-2
source=DMZ, route via Tunnel-1 and Tunnel-2

Configure policies and rules as required.

[Mark Morgan Lloyd]

Post by Simon Hobson

Apologies if this turns out to be an FAQ, but I'm having trouble getting to grips with things.
I've got a Raspberry Pi (little ARM box) here running Debian "Jessie" with the as-supplied Shorewall 4.6. As well as eth0 (192.168.1.5) as the "internal" side of the router, it's using VLANs on eth1 to provide 2x upstream interfaces (172.27.200.5 and 172.27.201.5) plus some more reserved for DMZ systems (90.155.84.x).
The upstream systems are configured with 4G wireless (carrier-grade NAT) and L2TP endpoints. Incoming traffic on the L2TP can see the DMZ. Custom iptables and tc stuff on these systems ensures that traffic originating on a routable address (90.155.84.x) goes out over the tunnel, but anything on an RFC-1918 address goes over carrier-grade NAT on the 4G. Setting this up was "fun", but these aren't causing me significant problems.
At our ISP, the L2TP tunnels are bonded on a Firebrick router. I think it's reasonable to assume that they know what they're doing, since they design and manufacture the things :-)
* Anything originating at an internal RFC-1918 address is to be routed over one or the other 172.27.x.x paths, hence over 4G with carrier-grade NAT. Am I correct that this falls into the "multiple provider" basket?

Correct it's done in the routing rules.

* Anything originating at a DMZ routable address is to be split proportionally onto both tunnels, i.e. a 50:50 bandwidth split.

Not too sure about the 50:50 split - never done it that way. But I think the way to do it will be to define both providers with the balance option, and set at the same priority.

The problem there is that the balance facility- if I understand it
correctly- works at the level of TCP sessions. What I need is something
that kicks stuff around on a per-packet basis, i.e. if an e.g. SMTP
session sends 1000 packets half go out over one interface and half over
the other.

Post by Simon Hobson

* Incoming traffic over both tunnels is to be merged for the DMZ (SMTP etc.).

Just put both interfaces into the one zone.
Merging of the traffic isn't a Shorewall function. The network stack does that, and it doesn't really care which interface a packet came in on, by default it just routes on destination - except for the source routing rules setup for the outbound traffic that is.

Thanks, I put that in more for completeness than anything else.

Post by Simon Hobson
You probably want to remove your custom iptables and tc stuff and let Shorewall do it all. Otherwise you run real risks of having problems that are "interesting" to diagnose.

It really does add rather a lot of value- tracking the state of
different 4G devices, making sure that they present a uniform IP
address, handling fallback to alternative media and so on. But I do hear
what you're saying, and will bear it in mind.

Post by Simon Hobson
My thoughts are that you'll want 4 providers - 2 using the native 4G, 2 using the tunnels.

So effectively each upstream device presents two different IP addresses
and Shorewall distributes based on that, rather than taking into account
whether the originator is DMZ or RFC-1918.

Post by Simon Hobson
As I read it, you won't be doing any NAT/Masq yourself - that's being done by the carrier.

Although at the moment I've got a layer of NAT in there to make sure
that whatever address the 4G device presents when it's plugged in- and
potentially there can be more than one presenting the same address- it's
changed to a controlled one.

So I've currently got two upstream "slices", each with a 4G stick which
claims to be 192.168.9.1- not changeable- or falling back to copper, one
of which always presents itself as 172.27.200.1 and the other as
172.27.201.1.

Each of those also has an L2TP tunnel daemon, with IP addresses
allocated by our ISP. Our ISP is also routing stuff for the DMZ onto
both tunnels, which is being routed towards the common Raspberry Pi.

Post by Simon Hobson
source=RFC1918, route via 4G-1 and 4G-2
source=DMZ, route via Tunnel-1 and Tunnel-2
Configure policies and rules as required.