Date: Fri, 3 Feb 2017 12:47:47 +0500 From: "Eugene M. Zheganin" <emz@norma.perm.ru> To: "freebsd-pf@freebsd.org" <freebsd-pf@freebsd.org> Subject: PF artifacts in NAT and ICMP exceeded replies Message-ID: <589435A3.3040603@norma.perm.ru>
next in thread | raw e-mail | index | archive | help
Hi.
I found a weird artifacts on one of my FreeBSD installations. Here comes
the traceroute and tcpdumps outputs, mostly self-explanatory.
Problem: some of the ICMP exceeded in transit replies have source IP
translated to the original traceroute destination IP (i.e. I traceroute
host A, and some of the packets on the third hop returns with the source
IP of the host A, which is impossible).
As you can see below, originating host receives a traceroute picture
that is really weird. In the same time the border passes clearly valid
packets. Something bad happens on the NAT itself. All the three hosts
run FreeBSD with pf, different releases, mostly 10.x branch.
I have a border configuration with two borders in CARP. First I thought
that this could be explaining if the traceroute session is somehow split
between borders, but, as you can see below, the session is handled by
only one border, from first to the last packet.
All the outputs are captured during the same traceroute pass/
Host one - ICMP originator:
traceroute -P icmp 153.92.28.82
traceroute to 153.92.28.82 (153.92.28.82), 64 hops max, 48 byte packets
1 192.168.7.7 (192.168.7.7) 0.129 ms 0.227 ms 0.116 ms
2 wizard.hq.norma.perm.ru (128.127.144.1) 0.379 ms
153.92.28.82 (153.92.28.82) 0.313 ms
wizard.hq.norma.perm.ru (128.127.144.1) 0.246 ms
3 153.92.28.82 (153.92.28.82) 1.153 ms 0.999 ms
prm01.prm28.transtelecom.net (188.43.17.174) 0.923 ms
4 153.92.28.82 (153.92.28.82) 69.619 ms
rtr01.da-rz.net (80.81.194.157) 64.087 ms
153.92.28.82 (153.92.28.82) 60.011 ms
5 153.92.28.82 (153.92.28.82) 60.124 ms 60.004 ms 59.983 ms
it's tcpdump:
# tcpdump -npi re0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decod=
e
listening on re0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:08:49.703343 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 1, length 28
12:08:49.703434 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-tran=
sit, length 36
12:08:49.712355 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 2, length 28
12:08:49.712505 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-tran=
sit, length 36
12:08:49.712548 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 3, length 28
12:08:49.712644 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-tran=
sit, length 36
12:08:49.712668 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 4, length 28
12:08:49.713032 IP 128.127.144.1 > 192.168.7.96: ICMP time exceeded in-tr=
ansit, length 36
12:08:49.713552 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 5, length 28
12:08:49.713818 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.714239 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 6, length 28
12:08:49.714468 IP 128.127.144.1 > 192.168.7.96: ICMP time exceeded in-tr=
ansit, length 36
12:08:49.714948 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 7, length 28
12:08:49.716088 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.716716 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 8, length 28
12:08:49.717654 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.717718 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 9, length 28
12:08:49.718581 IP 188.43.17.174 > 192.168.7.96: ICMP time exceeded in-tr=
ansit, length 36
12:08:49.718982 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 10, length 28
12:08:49.788448 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.789403 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 11, length 28
12:08:49.853330 IP 80.81.194.157 > 192.168.7.96: ICMP time exceeded in-tr=
ansit, length 36
12:08:49.854609 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 12, length 28
12:08:49.914486 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.915685 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 13, length 28
12:08:49.975603 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602=
, seq 13, length 28
12:08:49.976377 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 14, length 28
12:08:50.036233 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602=
, seq 14, length 28
12:08:50.036381 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 15, length 28
12:08:50.096203 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602=
, seq 15, length 28
Host with NAT - tcpdump on the LAN interface (facing ICMP originator; spo=
lier: some ICMP replies are translated to the IP of the destination host)=
:
# tcpdump -npi vlan15 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decod=
e
listening on vlan15, link-type EN10MB (Ethernet), capture size 65535 byte=
s
12:08:42.108537 IP 192.168.7.253 > 192.168.3.9: ICMP 192.168.7.253 udp po=
rt 623 unreachable, length 57
12:08:42.200953 IP 192.168.7.138 > 192.168.142.220: ICMP echo request, id=
512, seq 61241, length 19
12:08:44.501117 IP 192.168.7.123 > 192.168.7.6: ICMP echo request, id 392=
00, seq 0, length 64
12:08:44.501132 IP 192.168.7.6 > 192.168.7.123: ICMP echo reply, id 39200=
, seq 0, length 64
12:08:47.108923 IP 192.168.7.253 > 192.168.3.9: ICMP 192.168.7.253 udp po=
rt 623 unreachable, length 57
12:08:47.684410 IP 192.168.7.138 > 192.168.142.220: ICMP echo request, id=
512, seq 61497, length 19
12:08:49.694248 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 1, length 28
12:08:49.694267 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-tran=
sit, length 36
12:08:49.703258 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 2, length 28
12:08:49.703266 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-tran=
sit, length 36
12:08:49.703454 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 3, length 28
12:08:49.703461 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-tran=
sit, length 36
12:08:49.703573 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 4, length 28
12:08:49.703874 IP 128.127.144.1 > 192.168.7.96: ICMP time exceeded in-tr=
ansit, length 36
12:08:49.704453 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 5, length 28
12:08:49.704659 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.705141 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 6, length 28
12:08:49.705309 IP 128.127.144.1 > 192.168.7.96: ICMP time exceeded in-tr=
ansit, length 36
12:08:49.705864 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 7, length 28
12:08:49.706929 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.707656 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 8, length 28
12:08:49.708495 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.708625 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 9, length 28
12:08:49.709421 IP 188.43.17.174 > 192.168.7.96: ICMP time exceeded in-tr=
ansit, length 36
12:08:49.709884 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 10, length 28
12:08:49.779249 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.780345 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 11, length 28
12:08:49.844153 IP 80.81.194.157 > 192.168.7.96: ICMP time exceeded in-tr=
ansit, length 36
12:08:49.845512 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 12, length 28
12:08:49.905325 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-tra=
nsit, length 36
12:08:49.906601 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 13, length 28
12:08:49.966390 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602=
, seq 13, length 28
12:08:49.967282 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 14, length 28
12:08:50.027041 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602=
, seq 14, length 28
12:08:50.027284 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 466=
02, seq 15, length 28
12:08:50.086991 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602=
, seq 15, length 28
border interface facing host with NAT (spolier: everything is normal ):
# tcpdump -npi vlan23 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decod=
e
listening on vlan23, link-type EN10MB (Ethernet), capture size 65535 byte=
s
12:08:49.704074 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 4, length 28
12:08:49.704086 IP 128.127.144.1 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.704879 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 5, length 28
12:08:49.704887 IP 128.127.144.1 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.705523 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 6, length 28
12:08:49.705532 IP 128.127.144.1 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.706324 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 7, length 28
12:08:49.707132 IP 188.43.17.174 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.708122 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 8, length 28
12:08:49.708660 IP 188.43.17.174 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.709110 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 9, length 28
12:08:49.709571 IP 188.43.17.174 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.710234 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 10, length 28
12:08:49.779444 IP 80.81.194.157 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.780816 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 11, length 28
12:08:49.844373 IP 80.81.194.157 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.845883 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 12, length 28
12:08:49.905547 IP 80.81.194.157 > 128.127.144.3: ICMP time exceeded in-t=
ransit, length 36
12:08:49.906993 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 13, length 28
12:08:49.966620 IP 153.92.28.82 > 128.127.144.3: ICMP echo reply, id 4455=
0, seq 13, length 28
12:08:49.967736 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 14, length 28
12:08:50.027266 IP 153.92.28.82 > 128.127.144.3: ICMP echo reply, id 4455=
0, seq 14, length 28
12:08:50.027746 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44=
550, seq 15, length 28
12:08:50.087208 IP 153.92.28.82 > 128.127.144.3: ICMP echo reply, id 4455=
0, seq 15, length 28
Eugene.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?589435A3.3040603>