[Shorewall-users] Accounting problem - am I missing something simple ?

Post by Simon Hobson
I've had accounting (counting traffic by IP) running for ages on other routers (ethernet interfaces), but I'm struggling to to get it going on a newer one with a PPPoE interface. Everything looks OK in terms of the iptables rules setup - but I'm just not getting reasonable figures. Does anyone know if there's anything special about PPP interfaces for this ?
System is Debian Wheezy, running as a VM under Xen, and with Shorewall 4.5.5.3
Anyone see something silly I've overlooked (that's my usual problem, too close, can't see things right in front of me) ?

Ah, I think I may have spotted the issue. There is another key difference between this router and the others - this one is doing NAT (combination of masq and DNAT rules). I've observed that if I send traffic (eg sustained pings with large packets) to an address the router answers - then I see sensible amounts of traffic counted. If I DNAT all traffic on an IP to an internal host, then I see no traffic counted.
So it looks like the accounting is done before outbound masq, and after inbound DNAT rules - hence the rules using the outside addresses just don't match.

Is there any way to fix this ?

Tom Eastep

2017-02-16 17:39:36 UTC

Post by Simon Hobson
I've had accounting (counting traffic by IP) running for ages on
other routers (ethernet interfaces), but I'm struggling to to get
it going on a newer one with a PPPoE interface. Everything looks
OK in terms of the iptables rules setup - but I'm just not
getting reasonable figures. Does anyone know if there's anything
special about PPP interfaces for this ?
System is Debian Wheezy, running as a VM under Xen, and with
Shorewall 4.5.5.3
Anyone see something silly I've overlooked (that's my usual
problem, too close, can't see things right in front of me) ?

Ah, I think I may have spotted the issue. There is another key
difference between this router and the others - this one is doing
NAT (combination of masq and DNAT rules). I've observed that if I
send traffic (eg sustained pings with large packets) to an address
the router answers - then I see sensible amounts of traffic
counted. If I DNAT all traffic on an IP to an internal host, then I
see no traffic counted. So it looks like the accounting is done
before outbound masq, and after inbound DNAT rules - hence the
rules using the outside addresses just don't match.
Is there any way to fix this ?

Partially. With ACCOUNTING_TABLE=mangle, rules in the PREROUTING
section of the accounting file are traversed prior to DNAT.
Unfortunately, rules in the POSTROUTING section are still traversed
before SNAT/MASQUERADE. See
http://www.shorewall.org/NetfilterOverview.html.

- -Tom
- --
Tom Eastep \ When I die, I want to go like my Grandfather who
Shoreline, \ died peacefully in his sleep. Not screaming like
Washington, USA \ all of the passengers in his car
http://shorewall.net \________________________________________________

Simon Hobson

2017-02-16 21:38:56 UTC

Post by Simon Hobson
Is there any way to fix this ?

Hmm, that's "inconvenient" :-(

The diagram is useful, but doesn't show where accounting rules fit into it. But your explanation is sufficient, thanks.

Tom Eastep

2017-02-16 22:14:07 UTC

Post by Simon Hobson
Is there any way to fix this ?

Hmm, that's "inconvenient" :-(
The diagram is useful, but doesn't show where accounting rules fit
into it.

It actually does. With ACCOUNTING_TABLE=mangle, all rules are in the
mangle table. When you section the accounting file, the rules in each
section are jumped to from the chain of the same name.

- -Tom

- --
Tom Eastep \ When I die, I want to go like my Grandfather who
Shoreline, \ died peacefully in his sleep. Not screaming like
Washington, USA \ all of the passengers in his car
http://shorewall.net \________________________________________________

Simon Hobson

2017-02-17 08:17:32 UTC

-----BEGIN PGP SIGNED MESSAGE-----

Post by Simon Hobson
The diagram is useful, but doesn't show where accounting rules fit
into it.

It actually does. With ACCOUNTING_TABLE=mangle, all rules are in the
mangle table. When you section the accounting file, the rules in each
section are jumped to from the chain of the same name.

OK, I see it now, and the note at the top of http://shorewall.net/manpages/shorewall-accounting.html about the introduction of the ACCOUNTING_TABLE setting.

Just to clarify, in the prerouting and postrouting boxes, the chains are executed top-down in the list ? So Raw, Mangle, NAT for prerouting; and Mangle, NAT, Rawpost for postrouting ?

I feel some experimentation to see if (manually added) accounting rules will work in the Rawpost chain ...

Simon Hobson

2017-02-17 15:43:34 UTC

Post by Simon Hobson
I feel some experimentation to see if (manually added) accounting rules will work in the Rawpost chain ...

s/chain/table/

Well that didn't take long. Seems the rawpost table isn't installed by default (at least on the Debian systems I work with). it's available in the xtables-addons-dkms package - but that'll pull in a large number of packages I don't really want on this box.

I think I'll need to go and have a rethink about all of this setup :-(

Matt Darfeuille

2017-02-17 16:25:43 UTC

Post by Simon Hobson
I feel some experimentation to see if (manually added) accounting rules will work in the Rawpost chain ...

s/chain/table/
Well that didn't take long. Seems the rawpost table isn't installed by default (at least on the Debian systems I work with). it's available in the xtables-addons-dkms package - but that'll pull in a large number of packages I don't really want on this box.
I think I'll need to go and have a rethink about all of this setup :-(

You need less packages if you do it yourself!

At least that was so the last time I did it:

https://packages.debian.org/jessie/admin/xtables-addons-source

-Matt

--
Matt Darfeuille

Tom Eastep

2017-02-17 17:47:30 UTC

-----BEGIN PGP SIGNED MESSAGE-----

Post by Simon Hobson
The diagram is useful, but doesn't show where accounting rules
fit into it.

It actually does. With ACCOUNTING_TABLE=mangle, all rules are in
the mangle table. When you section the accounting file, the rules
in each section are jumped to from the chain of the same name.

OK, I see it now, and the note at the top of
http://shorewall.net/manpages/shorewall-accounting.html about the
introduction of the ACCOUNTING_TABLE setting.
Just to clarify, in the prerouting and postrouting boxes, the
chains are executed top-down in the list ? So Raw, Mangle, NAT for
prerouting; and Mangle, NAT, Rawpost for postrouting ?

Correct.

Post by Simon Hobson
I feel some experimentation to see if (manually added) accounting
rules will work in the Rawpost chain ...

Beware that the rawpost table has been removed in recent kernels. It
was used for stateless SNAT which is now done in the mangle table.

- -Tom
- --
Tom Eastep \ When I die, I want to go like my Grandfather who
Shoreline, \ died peacefully in his sleep. Not screaming like
Washington, USA \ all of the passengers in his car
http://shorewall.net \________________________________________________

Simon Hobson

2017-02-17 20:49:55 UTC

Post by Simon Hobson
I feel some experimentation to see if (manually added) accounting
rules will work in the Rawpost chain ...

Beware that the rawpost table has been removed in recent kernels. It
was used for stateless SNAT which is now done in the mangle table.

Pity, it would have done nicely for me.

And to add a bit of a chuckle to your Friday afternoon. I looked at that diagram again, and (bear in mind I've come down with a cold, and it's Friday) thought ... "hang on, if I insert a rule at the start of nat:prerouting and append another to the nat:postrouting chain, then that gets me the rules where I want them".
Then I could see the packet/byte counts going up, but not by the right amount. And then the penny dropped about the nat table only getting the first packet in each connection.

I've been toying with the idea of splitting the single router appliance into several. One that does all the NAT and stuff; and another in front of each connection that doesn't do NAT, but is two-port and can easily do traffic shaping - and traffic data collection.
That'll be fun to do without killing anything :-)