From owner-freebsd-net@freebsd.org Sat Dec 14 19:54:42 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7138C1D5B12 for ; Sat, 14 Dec 2019 19:54:42 +0000 (UTC) (envelope-from john@saltant.com) Received: from twaddle.saltant.net (twaddle.saltant.net [72.78.188.147]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47Zyt9446cz3CZH for ; Sat, 14 Dec 2019 19:54:41 +0000 (UTC) (envelope-from john@saltant.com) Received: from statler.priv.n.saltant.net (unknown [IPv6:2001:470:8d6f:0:e89c:86ea:6d81:3ec7]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by twaddle.saltant.net (Postfix) with ESMTPSA id 236B32ABAA for ; Sat, 14 Dec 2019 14:54:35 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=saltant.com; s=twaddle; t=1576353275; bh=jq3z6JeBubj38Py5YHk2v4IuQKJ7dDuiriftCHX1z/A=; h=To:From:Subject:Date; b=rm9F4zVKrWiiwLET4/r4p8ii6mtLDfhcCjog20jgPDDFJO/ViyzHTPmLf09R1GNZF v4+HRoujhv7hGmQl7cDf2PKlpzmsb+0w7woxdhW77Z+Q2ROTKTlGa7GdbyWwivuT+u equzpKbsfXF/WAkAyq4ueXi2MPWzvMKRP6YZH9Q+Ebnq96tzl3N9NmsKou/L4nt4oD j3ytyt55U2KQqbRf9RxFHfjuGgk4+H4whuXq9rtv6YNiaHwqnfdWGcpiQUQt74QxR7 VJPJytie8pn4z1QwB3hZ5B1mCzaqB/Lv3+Zs/T4qd/ZXBYRRcQajApvwghRzTEMNUJ KC8qjF1lFw5WA== To: FreeBSD Networking From: "John W. O'Brien" Subject: NAT64 return traffic vanishes after successful de-alias Autocrypt: addr=john@saltant.com; prefer-encrypt=mutual; keydata= mQINBFpcMG0BEACeAEQ0ZTUEH+6B8XIBid2H8g1yY+niHxVphqz8JwnQtYX+bS+Kl3vr783F HH81DEbfPtYgHY53NF9FjSzCyj13lXVnEGQOdxXzZVKsN1nyuXCN2hDOFH7Yc5yQ8h85T4Hv sqPIGIXOztu4MX14iUAcTgLhfibNQBeKDeNI+BBeaE9lPuNVeiM+xsI4JYcjmDbjFzAHRpBo ull0koUFh6RZAKE7u17yLej1pTIQQVjQpWdK37BAq4hdkLwjGDY8mDGo3ZwGdNibxIAxv/wi KU6u2DfUg8+kLHIhOqk/+kFQ/uK5YA1azsyD5eIbNAs4W7LglA6SkiGBglTwkP0VCrkPdD14 6sx3U7uFgexDWbVuhLIkcPQ0SRmnjgUKHgk7px/jMvAPKSKoL0JQNdP/+pnO9CDLGmoHx9gE 5kVr5dQK8c/WauEfimAdE9qLuN6vb0Iei73q3e3OOHAUusR5wC5SwXt4iilbaK4r04NKXyfb SB3+qWST07F9cmMscfEStSBhpez3awB+1jz8gr40tkEGsFZGvD2KKAgZdKpoxv6IrZepclWz HpqHF01SRFORYMsd1d83XlEu/S1/Z9YJ87RoCdZuYCkjnoRPtpTi9d+JD/u3ZiQFwLUz/Ne3 VqiGKvY66EGcO3tvANMg6GWD9sqlnBDp9Lls0ChEY3dgDYd6DQARAQABtCJKb2huIFcuIE8n QnJpZW4gPGpvaG5Ac2FsdGFudC5jb20+iQJUBBMBCAA+AhsDBQsJCAcCBhUKCQgLAgQWAgMB Ah4BAheAFiEENPkbBr3zmPAVSH2HM8TWS4ldvzsFAlpcMTMFCQX2qcYACgkQM8TWS4ldvztT xQ//eHb1mgd40Z0fN2GnJti6/9uJ771IO6slFQ02GZcXZI+FIQo8Yd1dHe0e0Codu78qvJNr ggUtqdxH6SVp7K1AWHeLH5S0PF6iG5B+YUux080wEv/Mr8PPMgAD8gS3wiPDDgB/kUXO52bn DC3Fc0dUrFE/JAOByVEEDL5nLF6SQNpAtIUnaAIIuhKxi0d40LMcLUwuJ6jExynw8Iu7OVtu Y1PRAH5ESt6wYZq8ro8ukh4rMOxiWtT1yNEgHgnq3N4jKErVo87YJijHSSj80IKxUiKb/T6K tGTEBTKiSUV3OFj0ZoPxcbUmhIg2sBCNHaUCiI0KabqN1NyK2glKtcK6NpWy3JIHvtr3+VL1 /tvQTwlVUIacmsuxkGzm5vJPs/i2RtwsJXEXPmIRNgJ1EwZgpg5VqqEUDlmSyRLb48QcDrdv utKLA1MKLib1fD+0XmxZTbCMlFMlvJjAoBlVq60mvB/Jnv1TTnZ2eN6DKMWoxHKmPICh5F1q esmT/aJRIUoCiAgcChi4Ol4XmW3dM7ypjKCGHzyr6emCky5pjqSQZyFzg0RN5UjUQBISAGmJ E8hCFZIy7tf8meqIDbtkONh+JShN6u3t02JrnzSOQjZCh5WQW9Pnu7unJlIsYB10aZ6rvuAK YjghT8QLG8QVgJj/U9oeVG1Ag60fmLZdOFjRGmm5Ag0EWlwwbQEQANebvidw1D5SKSmG3Ut8 p9vngBi5HjYe4FSYcfz0NgYa893RiScQ6yjOwuEf/fEoBgvpVnhcbu0JsaYvDNNzFGzPQcj0 CFhkr5s7REWNLGmmFCxCaGieTxIQdYsLxwn72mops8bsrL0a++8NDE+l7X4K3EUyp9GP7pIq 4l9jeIJ/RnX3yySRlXxcM3P+DV9ltXsnQ9pC/qEVVyK18C1zoiskhxmAY9cv9TJOaANHtA7R 7+hM5TyppIz7kqiwiCf6XfVFqKH0I0srdamb0KTnAZpmyx7iNKYl60PdIfEwkwck8fcGwOSA lwE9CLkHLwKMjx/gF3xRag5xjOdP/Out0cQ/pXv8DWnKblWbiGZheB4xUqhOT9Cj/8u/tKtC 51C9wID26hsrhtSAMJPUwQoo/SwLNEd1JpkqUP1njOdlV8FmM1EozHLPSvwlTm6oWwubkkY6 QkUHqXuO+2VdNhyDfx23fQhd0UPhQ0ceDRnjaSB9ycWqpktBP5iNQajYbx5Ktt8fC2Y+Ztjo u1KY7wJSUzqh7uZgR1TqIOVZp7bdPLBGHW5eNEf0Awq17utGe6d9i4hPmeNqELUz71hjmABm bIQJ+VgqYcQ0T/PrjwhzHv5g3jn67/ftW91nlTNpbhwm8suIdPA1hF6vgnZ3B4+JsevnevLG yU6YCb0OOKleP6pZABEBAAGJAjwEGAEIACYCGwwWIQQ0+RsGvfOY8BVIfYczxNZLiV2/OwUC WlwxTQUJBBV2YAAKCRAzxNZLiV2/O2PnD/wMKz/rzYbf0SaTvgae4jqryrcWRta56dcnVe7W KPuUu4Q/WBGhXKeCfPrlr399bILxZGw5TXuGMjS8gEoMd81PEMcWaMpgg3F569Cxd9GN6AZd LXXrZa0aM7dvZkz98ymILEnqHMpF74sLvZY2PrsOwo2gKXNqhtCJ2ph8OUKhG+NHvAomjMu9 lPQMkXJ4HRV0OljawqAe4y+IFu2K4abWwZw1mdniTCb5al8V2umzf26QL0DgeFp3banlfjYW Dn5cRuDBQqIoR/6cQaKdFKTJYiTVK3p3WRWiJQniYi39S8CR646w+zVi7ax1shSB0r0lxIFo CZu285HcMd7HsHH+T2ZI45ilayUoyoZvxPPlwhiRzyYZ6qqAAXKDihhda7uNApUqLwoSn5FW njmx6KdlVPF9ycCdf+in5k6nVlHWG15ogF/Y96K+/Q1Iuod9rzWqT4bz9a5olY8r++QE3V1b H3z803wXEUAJg+WGTkYXFNw7w6RhSSEhBRzupDoCROSkRhe3vQGy5FLG+BMV9n9nevhj5sBx CM1BbNBdB5H/2RcXh0wSb6zjewgs3UAbBvCQOdMAMo8XpYM5SLBqtaY7oalBElTxtFnwSNJm hMbahYE/wHbkmMqalrzGyQxbSUdrmE64CIX8xmv47fnjRoTZMzKim/02MRH+Ss1M+rLzp7kB DQRaXDCyAQgAyaQWiyazOcbV1JVndXG3JbeWom0Ros4RgjliRNLTm4rLefgk4mtvQpsGvTX7 bsiNRkxu2KdDo8zEG95e7FqbftxOFlptaEnJlrfrod6a5GX7E4cW74RgMHU9yj0IYijInENP FDf5yok1NvQ4IdS7Wqetta8X3hb2+iAXVkwDOhC9HTxEKZSWpsuZSs3eh2B2ypowa/12B4Dj ZXZ0ImUeLXqjL/ze5HmwcrQ1wqvo1pxc5NTA8vmwP4d9bnuKV6C7OIqw1Bw/VCxmNjX31gL3 a8K1eTMWu6TBkZ8z798eidmpU6gHB4zqE7NhBpHvNPePbQodXsMH40b5W82B3CRNDwARAQAB iQNyBBgBCAAmAhsCFiEENPkbBr3zmPAVSH2HM8TWS4ldvzsFAlpcMU0FCQQVdhsBQMB0IAQZ AQgAHRYhBCqRB5JEaEg4iCZEDlj7SueqT/5uBQJaXDCyAAoJEFj7SueqT/5u3SEH/21Wd0DD DVDx9jW6j7AlYSaJI9FZQVBZq0AakK3DgzWoyppb0NgNIWCRkghYmeni7ZyufmJg8mqzoWJT E8SeS9CYBhtmT3VO2N+w6x988GBplC69nhqoQBvHf81REZlWC72k5DIxfHJHWLI/9/aWc3ND wwifSdIjuGwfytqDp1RcAlCgx79ej8oodEII+PIBsLV6C7S9QV6kfJ1OXHE/lqbBV62Ywu/Y xHhvWgCOR8mz41NMrDz/K0otILUVwoDcE5tMOx5j6GFQEItFi/GFKogssV+4Tk9COmPS8ka7 ZFEnjjdoCiL3OveN2P4mBqG2Mh/0HAA/0v2DP6jqKHmaINkJEDPE1kuJXb872swP/3Ftis9+ 285gWUT7sKMbHkLxwwc/4Ga0vkBFyp9xRprlkvd7ivq2DP1gWvVds/V28BGFQ7SoRA5rLO+K BP7a2JJCk0025W4M8D6rp2mYj7iHLoxCNb5bScPYmBMnhKH4fg9QJWZozHik7wXrQNmrRb3A e+L0XfQ83tviuQhQsi+JtupQgf9d2a2Yza5bppdPYKialrJre3LIh/T4g4kJeoa4IQPwkXe8 httQa48571xINK2vtNkIjc4iG7mM4bAFCjZLx7AM8Dc3vVcZNbd21o5mhxe0WN9nICG8oKk5 9KwJKu6ul6TR0BxzvzpgcQyZGsDfhETsI/z0G7TVUXnRbZIgJHYH7DOVycjZLHAxQ5KweHkA bincQlaI0HMFf7FGtYnrUy3voTZ70xYQoYH1Gh/MeuELnscsTNBvYgOI2xYPOYilcFA4D3ZP p7p7ou7eZRkBLD6HHnrTgZB/Hn6FIklwll8jev3KBYWjSGKKcJQMK38OvJHDwHe1Wue+xpPl tFGoX7KCLFxe+VDmFjhfcgmoPJYBBq6D2s5AUj7cjTZUhb727ROSsK6KFCQhW25j8MJF+qGT RcRcWqgTQZoxWNqr5Foyeu3KoUY5ywBcPjqBMyqod27wOS8iQmHskLf7v9UrOR3/zLWASFyX MaAD/5Af9kIDAmJcwLvO0Mz9HDQB Organization: Saltant Solutions Message-ID: <9f3ee846-1357-0b73-cc0f-e001ea74b15c@saltant.com> Date: Sat, 14 Dec 2019 14:54:26 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:68.0) Gecko/20100101 Thunderbird/68.3.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ixsIqOhOcFNBiAgDzVdOO5k1eN6b0wqF0" X-Rspamd-Queue-Id: 47Zyt9446cz3CZH X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=saltant.com header.s=twaddle header.b=rm9F4zVK; dmarc=none; spf=pass (mx1.freebsd.org: domain of john@saltant.com designates 72.78.188.147 as permitted sender) smtp.mailfrom=john@saltant.com X-Spamd-Result: default: False [-4.44 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[saltant.com:s=twaddle]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:72.78.188.144/29]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_ATTACHMENT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[saltant.com]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[saltant.com:+]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; IP_SCORE(0.16)[asn: 701(0.84), country: US(-0.05)]; ASN(0.00)[asn:701, ipnet:72.78.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Dec 2019 19:54:42 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ixsIqOhOcFNBiAgDzVdOO5k1eN6b0wqF0 Content-Type: multipart/mixed; boundary="07s5fTw6F1s33hZB5CDOM160yvzi0NO5q" --07s5fTw6F1s33hZB5CDOM160yvzi0NO5q Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hello FreeBSD Networking, As the subject summarizes, I have a mostly-working NAT64 rig, but return traffic is disappearing, and I haven't been able to figure out why. I observe the post-translation (4-to-6) packets via ipfwlog0, but a simple ipfw counter rule ipfw matches nothing. My attempt to develop a minimum reproducible example failed in the sense that I did not reproduce the problem. Of course, this implies that one of the many differences between the simplified test (EC2 instance, two jails) and the problem rig (physical server, lagg, vlans, other things going on) is the cause. What I am hoping this list can help me with is being smart about what I try next. Otherwise, I would probably just try to brute force a solution by thinking of ways to permute the config that would rule each possible difference in or out. So far my main troubleshooting tools have been ipfw for its rule counters and nat64lsn stats output, netstat to look at fibs, and tcpdump pointed at real and diagnostic interfaces. What debugging tools and techniques should I employ to do better than brute force? If it would help, I would gladly share the working, EC2/jail demo configs on the list. Sharing the non-working configs I would prefer to do privately or not at all. This is on 12.1-RELEASE. Thank you, --=20 John W. O'Brien OpenPGP keys: 0x33C4D64B895DBF3B --07s5fTw6F1s33hZB5CDOM160yvzi0NO5q-- --ixsIqOhOcFNBiAgDzVdOO5k1eN6b0wqF0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCgAdFiEEKpEHkkRoSDiIJkQOWPtK56pP/m4FAl31PfIACgkQWPtK56pP /m6TIQf+P7s0gQ1s3lYX1gMki0f1ebjJWe/21DdTCa+MEJuga+oqYiV0EJHWHwor xNtf8/uXvIPOyEO5l/hJ1KwKDXo8A8y/tpEhwDN2Pw00aTvL4SlHfZz+QorgAcWI hSe2YQOwjGN6Y3Udik57YDXs9dsdftD8xCRYiG7IZ/O9wBqSAv7L6B7/TJLpttXt oHu3YdPkKpFUBVQYH7ADvIuDXA3wzuPxMQAbbzXXPxh6DSH+Iz1rXJEPnmM1k0r4 ZNpm13gmvIZHvou9tge03/rSy93BfvcNtdQe6OZNbwz+8Da1iJ4RBxxLV6VinbBa 7A72vcyrch//l19xqToHJLFzPIl5BQ== =aDhU -----END PGP SIGNATURE----- --ixsIqOhOcFNBiAgDzVdOO5k1eN6b0wqF0--