[Shorewall-users] mtr and ICMP type 11 (Time exceeded)

[Ob Noxious]

Hi,

When I use mtr-tiny (text mode version) to check on a destination, the
firewall logs get flooded a lot! No matter if I mtr from an inside host
(ie: desktop) or the firewall itself.

ex: on the firewall itself, "mtr 1.2.3.4" and suppose there are 6 hops to
reach it from A.A.A.A to F.F.F.F

I get something :
Shorewall:+net-fw:ACCEPT:IN=eth0 OUT= SRC=A.A.A.A DST=$FW LEN=56 TOS=0x00
PREC=0x00 TTL=255 ID=34268 PROTO=ICMP TYPE=11 CODE=0 [SRC=$FW DST=1.2.3.4
LEN=64 TOS=0x00 PREC=0x00 TTL=1 ID=54513 PROTO=ICMP TYPE=8 CODE=0 ID=65151
SEQ=59520 ]

At start, every hop from A.A.A.A to F.F.F.F generates one of these lines
and after each hop have been reached, there's a variation: Only some of the
hops on the way generate one of these lines every second.

Is this normal behaviour or is there something wrong somewhere?

Note: mtr's "Loss%" column stays at 0.0% all the time, no issue there.

[Tom Eastep]

Post by Ob Noxious
Hi,
When I use mtr-tiny (text mode version) to check on a destination, the
firewall logs get flooded a lot! No matter if I mtr from an inside host
(ie: desktop) or the firewall itself.
ex: on the firewall itself, "mtr 1.2.3.4" and suppose there are 6 hops
to reach it from A.A.A.A to F.F.F.F
Shorewall:+net-fw:ACCEPT:IN=eth0 OUT= SRC=A.A.A.A DST=$FW LEN=56
TOS=0x00 PREC=0x00 TTL=255 ID=34268 PROTO=ICMP TYPE=11 CODE=0 [SRC=$FW
DST=1.2.3.4 LEN=64 TOS=0x00 PREC=0x00 TTL=1 ID=54513 PROTO=ICMP TYPE=8
CODE=0 ID=65151 SEQ=59520 ]
At start, every hop from A.A.A.A to F.F.F.F generates one of these lines
and after each hop have been reached, there's a variation: Only some of
the hops on the way generate one of these lines every second.
Is this normal behaviour or is there something wrong somewhere?
Note: mtr's "Loss%" column stays at 0.0% all the time, no issue there.

You are logging related ACCEPT decisions -- do you expect those
connections to not be accepted???

-Tomo

[Ob Noxious]

On Wed, Sep 28, 2016 at 4:50 AM, Tom Eastep <***@shorewall.net> wrote:

You are logging related ACCEPT decisions -- do you expect those

Post by Tom Eastep
connections to not be accepted???

I use RELATED_LOG_LEVEL=info to get informed when "something goes wrong" or
"something unexpected happens".

I get (untracked) related ACCEPT log entries from time to time which is not
unusual. What I wasn't expecting is the amount of these log entries by
merely using a somewhat "normal" tool such as "mtr-tiny".

To my knowledge, no other tool generates so much log entries in Shorewall
in normal usage, hence my question about something I did wrong.

Do you experience the same behaviour?

[Simon Hobson]

What I wasn't expecting is the amount of these log entries by merely using a somewhat "normal" tool such as "mtr-tiny".
From the network traffic PoV, MTR is not "normal" - the traffic it generates is far from normal.

"Normal" traffic will rarely generate TTL Exceeded responses. With it's default settings, MTR will generate one TTL Exceeded packet per second per hop down(up?)stream of the router.


------------------------------------------------------------------------------

[Ob Noxious]

From the network traffic PoV, MTR is not "normal" - the traffic it
generates is far from normal.
"Normal" traffic will rarely generate TTL Exceeded responses. With it's
default settings, MTR will generate one TTL Exceeded packet per second per
hop down(up?)stream of the router.

Ok this is normal behaviour for MTR :-) Thanks

[Simon Hobson]

Post by Ob Noxious
Ok this is normal behaviour for MTR :-) Thanks

And traceroute.
It occurs to me, you might not know how traceroute works - it's fairly simple and obvious once yo see it.

When you send a packet, one of the header fields is Time To Live (TTL). The defaults vary between OSs, but are typically in the order of several tens - 30 or up. Every time the packet passes through a router, the TTL value is decremented - and if it reaches zero then the router doesn't forward it, but sends back an ICMP TTL Exceeded message to the source.
The main purpose of this TTL is to stop a routing loop creating a network killing packet storm. Without it, a routing problem which results in a packet returning to a router that's already handled it would result in that packet going round and round forever.
Eg, suppose router A has a route for the destination via router B, but router B has a route to the destination via router A. A will pass the packet to B, B will pass it back to A, which passes it back to B, which ... it may be more than two routers, so the packet goes via A, B, C, A, B, C, A ... and so on.

So that's why TTL Exceeded messages are not too common - they only really happen if there is a routing issue.

So where is this leading ?
How both traceroute, and MTR (which I believe was originally called Matt's TraceRoute), map the network is to send packets out with artificially low TTL values. If you set the TTL to 1 then the first router will respond with TTL Exceeded. With a TTL of 2, the second router will give TTL Exceeded, and so on. MTR defaults to using ICMP Ping packets for this, traceroute defaults to using UDP packets to some obscure (and high) range of ports expecting to get a (I think) ICMP Destination Unreachable, Port Unreachable response from the target host. These days with firewalls, it's more likely that the destination host will simply ignore the packet and traceroute will continue showing "* * *" for ever increasing hop counts until it reaches it's upper limit. Some modern traceroutes can use ICMP Ping instead.

So now you should be able to see why you get so many ICMP TTL Exceeded packets when running traceroutes. With traceroute, by default you'll get three from the first router, 3 from the second router, and so on.
With MTR, by default you'll get one from each hop as it sends one packet per second per hop up to the highest hop count it's seen till now + 1. SO if there's (say) 5 hops (including the target), you'll see 5 packets per second - one from each hop.

Obviously you'll have seen hops that don't respond with TTL Exceeded and these will appear as "invisible" hops in the results.
There are also techniques for hiding routers by fiddling with the TTL value. This is something that I've seen with an MPLS WAN I used to have at a previous employer. Normally there were only the two endpoint routers (those on my sites) visible - but one time while they were debugging they stopped fiddling with the TTL values, and their intermediate routers showed up for a while.


------------------------------------------------------------------------------

[Ob Noxious]

On Thu, Sep 29, 2016 at 9:32 PM, Simon Hobson <***@thehobsons.co.uk>
wrote:

And traceroute.

Post by Simon Hobson
It occurs to me, you might not know how traceroute works - it's fairly
simple and obvious once yo see it.

First of all, thank you very much for this detailed and very nice
explanation. This should be kept for reference as it is very well written,
very didactic.

Reading through it made me facepalm myself for not remembering most of this
content.

These days with firewalls, it's more likely that the destination host will

Post by Simon Hobson
simply ignore the packet and traceroute will continue showing "* * *" for
ever increasing hop counts until it reaches it's upper limit. Some modern
traceroutes can use ICMP Ping instead.

Yep, on Unixes, TraceRoute uses UDP. On Windows, "tracert" uses ICMP though.

Post by Simon Hobson
So now you should be able to see why you get so many ICMP TTL Exceeded
packets when running traceroutes. With traceroute, by default you'll get
three from the first router, 3 from the second router, and so on.
With MTR, by default you'll get one from each hop as it sends one packet
per second per hop up to the highest hop count it's seen till now + 1. SO
if there's (say) 5 hops (including the target), you'll see 5 packets per
second - one from each hop.

My initial trouble was with the Shorewall messages in fact. I wasn't really
surprised by the ICMP messages per se. Shorewall logs zone-to-zone messages
prefixed with a "+" sign when they happen to be UNKNOWN by NetFilter's
connection tracker. So as I stated in the initial post, I was surprised by
the *flood* of messages. I may not have expressed that in a clear way
though.

How come all the ICMP messages happen to be unkwown to the connection
tracker?

I know MOST of the inner workings of MTR and TraceRoute, I didn't remember
the TimeExceeded messages though which is a good memory refresher. But now
it makes sense, especially this part.

Post by Simon Hobson
Obviously you'll have seen hops that don't respond with TTL Exceeded and
these will appear as "invisible" hops in the results.

Most of the time, not that invisible. To my knowledge, many routers have
several IP adresses and according to their route table, the paths differ
whether the packet is "incoming" or "outgoing". That explains why I was
receiving some packets where the IP source wasn't listed in the MTR's hops
list but "close enough" (read: sharing a big part of CIDR) to think that
this was indeed "this" hop but through another interface. I haven't done
any serious routing (BGP or OSPF) for more than a decade, I must pull the
memories out of the "swap space" :-)

All that made sense after I sent the message as I studied the logs with
more carefully. Still, I wonder why NetFilter connection tracker does not
track packets sent to known hops (those hops in the MTR list). How to
explain the flood of "+net-fw" logs?

Post by Simon Hobson
There are also techniques for hiding routers by fiddling with the TTL
value.

NetFilter/IPTables have such a option with --set-ttl 128 :-)

Post by Simon Hobson
This is something that I've seen with an MPLS WAN I used to have at a
previous employer. Normally there were only the two endpoint routers (those
on my sites) visible - but one time while they were debugging they stopped
fiddling with the TTL values, and their intermediate routers showed up for
a while.

Debugging always forces us to disable every measure we take to hide from
the outside world :-) Even back in my days I never played with MPLS and I
only have a very vague idea of how it works.