From nobody Fri Oct 8 10:54:31 2021 X-Original-To: freebsd-pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CB65317EF89D for ; Fri, 8 Oct 2021 10:54:42 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [IPv6:2001:678:618::40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "plan-b.pwste.edu.pl", Issuer "GEANT OV RSA CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HQlSd4x6Lz3h3Q; Fri, 8 Oct 2021 10:54:41 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from fomalhaut.potoki.eu (dom.potoki.eu [62.133.140.50]) (authenticated bits=0) by plan-b.pwste.edu.pl (8.17.1/8.17.1) with ESMTPSA id 198AsWYX033969 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 8 Oct 2021 12:54:32 +0200 (CEST) (envelope-from zarychtam@plan-b.pwste.edu.pl) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=plan-b.pwste.edu.pl; s=plan-b-mailer; t=1633690472; bh=fu8+ZAAwZ9a/yU/UWmzLgjKrc6TnDrB9Nb+BL18NLWQ=; h=From:To:Cc:References:Subject:Date:In-Reply-To; b=RBMU7Qc+Y5nlf+JmGoeJ2y6Nzh9jNKaobezuED9cL9o8pwe25t7M+RyU/iMdst44Y hMFGtQ0l1YwyLwuH5zR15u1b6sRf1nnuoCPcOOyxHt/dsIM+HAqHDu9yey8mbUaMLO mJThCqEasUhYdm+sszA/jpXHAx3tHsGdJvsioS0mV/Y+GPyq6gqtMo0A9/3qqh1WIA 81sz1ngQNiGHc0CcFE/aA1jdNCQzDkEQUw/SJaG8WpPnXxUydpkkWOdzhwMCj23KNZ x7TL5291nuK5rg9D4wGV+6c2VEoIELWcci4Dk11CEIvbF+Xds3AiqsQufoK0ZvUCmi V8mUOrgQo+cCA== X-Authentication-Warning: plan-b.pwste.edu.pl: Host dom.potoki.eu [62.133.140.50] claimed to be fomalhaut.potoki.eu From: Marek Zarychta To: Kristof Provost Cc: freebsd-pf@freebsd.org References: <76015004-7980-fb5c-1cf8-60d7d745bdb9@plan-b.pwste.edu.pl> Subject: Re: "set skip on lo" on 12.x and 13.0 Message-ID: <33519ad1-cd22-6c50-a3af-8db6398445d5@plan-b.pwste.edu.pl> Date: Fri, 8 Oct 2021 12:54:31 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="s4cui987dOg2xDeoxv74jG0Z3tOJn4be9" X-Rspamd-Queue-Id: 4HQlSd4x6Lz3h3Q X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=plan-b.pwste.edu.pl header.s=plan-b-mailer header.b=RBMU7Qc+; dmarc=pass (policy=none) header.from=plan-b.pwste.edu.pl; spf=none (mx1.freebsd.org: domain of zarychtam@plan-b.pwste.edu.pl has no SPF policy when checking 2001:678:618::40) smtp.mailfrom=zarychtam@plan-b.pwste.edu.pl X-Spamd-Result: default: False [-5.87 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[plan-b.pwste.edu.pl:s=plan-b-mailer]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; HAS_ATTACHMENT(0.00)[]; DKIM_TRACE(0.00)[plan-b.pwste.edu.pl:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[plan-b.pwste.edu.pl,none]; NEURAL_HAM_SHORT(-0.97)[-0.971]; SIGNED_PGP(-2.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; ASN(0.00)[asn:206006, ipnet:2001:678:618::/48, country:PL]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --s4cui987dOg2xDeoxv74jG0Z3tOJn4be9 Content-Type: multipart/mixed; boundary="FNKmuMJH9Btsh3aDTkoO8h0IBEeUXzj62"; protected-headers="v1" From: Marek Zarychta To: Kristof Provost Cc: freebsd-pf@freebsd.org Message-ID: <33519ad1-cd22-6c50-a3af-8db6398445d5@plan-b.pwste.edu.pl> Subject: Re: "set skip on lo" on 12.x and 13.0 References: <76015004-7980-fb5c-1cf8-60d7d745bdb9@plan-b.pwste.edu.pl> In-Reply-To: --FNKmuMJH9Btsh3aDTkoO8h0IBEeUXzj62 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable W dniu 09.02.2021 o=C2=A016:44, Marek Zarychta pisze: > W dniu 09.02.2021 o=C2=A015:55, Kristof Provost pisze: >> On 9 Feb 2021, at 15:50, Marek Zarychta wrote: >>> Dear list, >>> >>> I am observing changed behaviour of the rule "set skip on lo". This >>> rule previously allowed for communication between the host and the >>> jail no only on loopback interfaces, but also on shared network >>> interfaces, for example, if a host had address x.x.x.x/24 and jail >>> had address x.x.x.y/32 on the same NIC, the rule above allowed for >>> communication between the host and jail using x.x.x.x and x.x.x.y >>> addresses. I am considering jails without VNET enabled and using the >>> same fib number. Now to allow this kind of communication I had to add= >>> "pass quick on lo", but I went out of free states rather quickly, so >>> instead of increasing the state limit, I have changed the method of >>> communication between the host and the jails to utilize only loopback= >>> addresses. >>> >>> It's rather not a regression but a change, some people might consider= >>> it POLA violation, but probably won't if it gets widely announced. >>> >> I=E2=80=99m not aware of the behaviour change you describe. >> >> However, there have been subtle issues around set skip on >> that may be confusing you. >> See #250994 / 0c156a3c32cd0d9168570da5686ddc96abcbbc5a for some of the= >> details. >> >=20 > I have seen this fix, but probably never used on affected machine > 12.2-STABLE after the MFC of this fix, I have transitioned to > 13.0-STABLE instead. Anyway, both: 12.x-STABLE and 11.x-STABLE with "se= t > skip on lo" were allowing for such communication between jail and host > not only on 127.0.0.0/8 addresses but also on shared NIC addresses. >=20 > The behaviour described above was happening with 13.0-STABLE regardless= > of using set skip on the group or individual interfaces, I mean=C2=A0 "= set > skip on lo" and "set skip on {lo0,lo1,lo2,lo3,....}". Now, to work > around this I have transitioned to using 127.0.0.0/8 only, but some > other people might get confused. >=20 The original problem has been solved a long time ago in different way, but the right solution was to remove the rule: "antispoof quick for lo" which followed "set skip on lo". In FreeBSD 13.0 and later this ruleset adds among others: "block drop in quick on ! lo inet from 127.0.0.0/8 to any" that prevented communication between the host and jails. I have neither 12 nor earlier versions to test this, but certainly, it worked different way there. So concluding this 8 months old thread: either "set skip on lo" worked a different way preventing "antispoof quick for lo" load or this erroneous contradiction was worked around a different way. Thank you for help in solving this. Kind regards, --=20 Marek Zarychta --FNKmuMJH9Btsh3aDTkoO8h0IBEeUXzj62-- --s4cui987dOg2xDeoxv74jG0Z3tOJn4be9 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEnjwyTmqn2oNX6C8qHZW8vIFppoIFAmFgI2cFAwAAAAAACgkQHZW8vIFppoIP Xgf/U/eAphKpcbyPPf4F9JjLXBms3vCBbTxTLN9e6v6F93JPHTme1CsYmVcfgUCmYgFx8x1e0VTb vdlqvAQaTHyp9b82PqNV2B2L+M4fu8PwIldVLQbaW4qstJoddkxRjhJJoWt2Mc07O6f6IFUk/CCr qQ7l4YH27ffSZoLxglpVdWm1wk/4+fjbNdxMrdf+AmvAhNi/gKAgp3GZvTWXxPwP+IweXabwHops 62cDO/2Ig56F1cyBlgalIHmcRSPlEV+2Ev4tsMwcmIQzceVIM0JLPQ4LM/JSRx0LdLUy914hryMF etu/2Vq9HJC184Uat2X2KBEE0fUU0/PM8oFBkCHXuQ== =jD2I -----END PGP SIGNATURE----- --s4cui987dOg2xDeoxv74jG0Z3tOJn4be9-- From nobody Sun Oct 10 11:28:47 2021 X-Original-To: freebsd-pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0F04C17F5D04; Sun, 10 Oct 2021 11:27:36 +0000 (UTC) (envelope-from kisscoolandthegangbang@hotmail.fr) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05olkn2047.outbound.protection.outlook.com [40.92.90.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HS05g0by5z4VHH; Sun, 10 Oct 2021 11:27:35 +0000 (UTC) (envelope-from kisscoolandthegangbang@hotmail.fr) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=itEUXAXsgXJGgcRmKziMalGfABG+c50eiEJQ8FpHg2iPxqwoVbd3kfPz7FyZJ/HLWbNTlISFvCmtbdxUShzD1PL/PRCGNMfQuncVM/JjczniqQt/8fW4qEdee2wQJKl/AX8fEkUjikIYmaYzW+l+maLVD4vBktBe+vzX4fm9dJHhdxz0IcMZy2HHx/VNBkdxISCSbYP9xhh/LyWNZjpTwdtwX7n5mmy/o0bsjS42XTlSa5zNttF0U38hzvzJTexO86X89m1wepQgRQ74bhuLls1aFrqEHFXdIGp/rV14SkgguxWpyIBHvHEPygKfCuC7VzoQ3Z2SJuZBfH3FKGRjxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KdHZNr/LhNkGZq1jqQ/SMp0GpCQqc15eLu+ek6loBzE=; b=BlwFhzFZBTZkbrj3lAgXZT1STcS57SUpj+/Q7WdGF0g+OjUWewtNCvKWWC1v6oB/pMD6XSzyqTUcEzrwg6A1gE9XOG2Ah9gMYwpAb4cn+Amc1FiW3zQyTLLP5WBgPPdIPNcw6LzxjbkxfnDgnkyPT2WBK0uPKeAVn4H8lXJ+GTCiq++wt2bajIF5ilNDbJaoICVPNsxt3TxMNakQNxFUpRtflumGWpoiZK9hpWyevQ95NEMEHY2Zpu+X8De1441wh0c5AsTh0k3J7Eann3vH4l+18DPTJ7AjQfEOr6NWLAY8X3fmTeUilyjAajrsej4c4AjNaEaf98zregtPTZUdLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none Received: from AM9PR07MB7956.eurprd07.prod.outlook.com (2603:10a6:20b:30d::20) by AM9PR07MB7763.eurprd07.prod.outlook.com (2603:10a6:20b:304::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.12; Sun, 10 Oct 2021 11:27:27 +0000 Received: from AM9PR07MB7956.eurprd07.prod.outlook.com ([fe80::cde2:f4ca:8325:6a10]) by AM9PR07MB7956.eurprd07.prod.outlook.com ([fe80::cde2:f4ca:8325:6a10%7]) with mapi id 15.20.4608.013; Sun, 10 Oct 2021 11:27:27 +0000 Date: Sun, 10 Oct 2021 13:28:47 +0200 From: kaycee gb To: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Subject: Re: Issue with packets routing/forwarding Message-ID: In-Reply-To: References: X-Mailer: Claws Mail 3.17.6 (GTK+ 2.24.31; x86_64-slackware-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-TMN: [egfTJozRL8SMG7kiuYjgclmFj6QjFLk5] X-ClientProxiedBy: DU2P250CA0012.EURP250.PROD.OUTLOOK.COM (2603:10a6:10:231::17) To AM9PR07MB7956.eurprd07.prod.outlook.com (2603:10a6:20b:30d::20) X-Microsoft-Original-Message-ID: <20211010132847.3d6559d0@slackstro.home.lan> List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mail.lacabanedeladmin.trickip.net (93.1.37.139) by DU2P250CA0012.EURP250.PROD.OUTLOOK.COM (2603:10a6:10:231::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.21 via Frontend Transport; Sun, 10 Oct 2021 11:27:26 +0000 Received: from slackstro.home.lan ([172.16.93.19]) (authenticated bits=0) by mail.lacabanedeladmin.trickip.net (8.15.2/8.15.2) with ESMTPSA id 19ABRMNp059881 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 10 Oct 2021 13:27:23 +0200 (CEST) (envelope-from kisscoolandthegangbang@hotmail.fr) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fb11beb2-4e41-4958-3ec5-08d98be0f02f X-MS-TrafficTypeDiagnostic: AM9PR07MB7763: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2lXHs/vJDGumNuY7U4QvJG+NI25sWnv8L8DZfcWfFMlEYnW9OPqN/4v/805UiRXFZeqG4TW4uZbI5JL57A9f0ef9IMqu0MRtZD/o1hQoTosqecN/Bya9CE04nvjWImYZnMhHt4od+/WGGoBn7AYVDIPnNNp1ZGpGlAF9E4WYfplOVDi+k89emszKlWRAY9MAbtTNTnQl0wAGl8ReQtnwa3gH77gCiU5r/rLShbD5/zc9leZeIUppuEzhkxgP8/u1yaW8x/q9edFL5vUitn0ByXNhVS55ZzHFeF8bL1Tg6xiY5QIBJIuDSE952Dvg7O62BxpD6mTns03Nhxt+R7LJwYXehH0YS2heHWc7NK5oPyI4mRT9zK1xEMFtTToPQ7eWvrW94NwG+gbn3qFR+Dsq2Y1w2b+3SpVYaYQnt2V0heJvkyU3gPScdMFyqTjaAguz X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: swod3BaFa5MU8t6x3geEh+GvmY3XdJGmDVTAq6vKPz1XxO0pw1btD5RSyc7vbJb8ukp0SCM7s6akHiJbXM7nVXKAK0g1jUl+iDD/xJmjBhbc3UpNhxC0F7BcG/9vbuQX3/VOzSMdU+YnUwRnSnpAOA== X-OriginatorOrg: sct-15-20-3174-8-msonline-outlook-466f4.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: fb11beb2-4e41-4958-3ec5-08d98be0f02f X-MS-Exchange-CrossTenant-AuthSource: AM9PR07MB7956.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2021 11:27:27.1892 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR07MB7763 X-Rspamd-Queue-Id: 4HS05g0by5z4VHH X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=hotmail.fr; spf=pass (mx1.freebsd.org: domain of kisscoolandthegangbang@hotmail.fr designates 40.92.90.47 as permitted sender) smtp.mailfrom=kisscoolandthegangbang@hotmail.fr X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; RCVD_COUNT_FIVE(0.00)[5]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.90.47:from]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[hotmail.fr]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/15]; RECEIVED_SPAMHAUS_PBL(0.00)[93.1.37.139:received]; NEURAL_HAM_SHORT(-0.20)[-0.200]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.92.90.47:from]; DMARC_POLICY_ALLOW(-0.50)[hotmail.fr,none]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[hotmail.fr]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-ThisMailContainsUnwantedMimeParts: N Hello, I'm back to this case. It makes me crazy as I don't understand why it does = not works.=20 I'll try to describe the problem in a different manner.=20 My topology: I have FreeBSD 11.4 host configured as a router/fw/nat with different jails= set and multiple FIBs. . ___Host_______________. | | |ifaces: lo0, vsw0, ... | | _______ | 192.168.1.1 | | jail1 | | em0 |------->|adsl modem/router| | | jail2 | | | | jailn | | ue0 |------->| 4g modem/router | | |_______| | 192.168.42.129 | _____ | |____| em1 |____________| | LAN -----| I have a jail available from local lan and configured to attach to vsw0. Jail's conf (relevant part): > ... > interface =3D "vsw0"; > ip4.addr +=3D "vsw0|192.168.1.92/32"; > ... > exec.fib =3D 0; >... vsw0 configuration: ># ifconfig vsw0 >vsw0: flags=3D8049 metric 0 mtu 16384 > options=3D680003 > inet6 fe80::1%vsw0 prefixlen 64 scopeid 0xc > inet 192.168.1.92 netmask 0xffffffff > nd6 options=3D21 > fib: 0 > groups: lo My pf configuration is like this (top lines for translation and packet filtering sections): >#### fib 1 >no nat log on $VSW from $proxout >nat log on ue0 tagged P100 -> ($phone_if ) >nat log on em0 tagged P100 -> $lan_ip >#### fib 1: end > >#### fib 0 >nat log on em0 from $proxout tag PROXOUTNAT -> $lan_ip >nat log on $phone_if from $proxout tag PROXOUTNAT -> ( $phone_if ) >#### fib 0: end ... >#### fib 1 > pass out log quick on $VSW proto tcp from $proxout to port {80, 44= 3} >user 100 tag P100 no state pass in log quick on $VSW tagged P100 rtable 0 > > pass out quick on em0 tagged P100 rtable 0 > pass out quick on ue0 tagged P100 rtable 0 >#### fib 1: end > >#### fib 0 > pass out quick on em0 tagged PROXOUTNAT > pass out quick on ue0 tagged PROXOUTNAT >#### fib 0: end > >block log quick from 109.0.64.169 >block log quick to 109.0.64.169 So, in a setup where everything is configured to use em0/192.168.1.1 as def= ault route, it's a working setup > 2021-10-10 12:06:04.419095 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.55998 > 109.0.64.169.80: Flags [S], seq 817941716, win 6553= 5, > options [mss 1460,nop,wscale 6,sackOK ,TS val 567386157 ecr 0], length 0 > 2021-10-10 12:06:04.745475 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.59942 > 109.0.64.169.80: Flags [S], seq 178756216, win 6553= 5, > options [mss 1460,nop,wscale 6,sackOK ,TS val 1682297331 ecr 0], length = 0 > 2021-10-10 12:06:04.841617 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.58758 > 52.84.228.93.443: Flags [S], seq 3657906762, win 65= 535, > options [mss 1460,nop,wscale 6,sack OK,TS val 498309471 ecr 0], length 0 > 2021-10-10 12:06:06.758062 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.54067 > 69.16.175.42.80: Flags [S], seq 1689219730, win 655= 35, > options [mss 1460,nop,wscale 6,sackO K,TS val 3560654970 ecr 0], length = 0 > 2021-10-10 12:06:07.499880 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.52905 > 93.20.64.1.80: Flags [S], seq 2950807804, win 65535= , > options [mss 1460,nop,wscale 6,sackOK, TS val 1714325115 ecr 0], length = 0 > 2021-10-10 12:06:07.500191 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.54440 > 93.20.64.1.80: Flags [S], seq 2349377925, win 65535= , > options [mss 1460,nop,wscale 6,sackOK, TS val 176674686 ecr 0], length 0 Then, I want to simulate an adsl failure, so I route wanted traffic via ue0= /4g. > # A ; route add 109.0.64.169 $defr && route add 69.16.175.42 $defr && rou= te > add 69.16.175.10 $defr && route add 93.20.64.1 $defr=20 > add host 109.0.64.169: gateway 192.168.42.129 fib 0=20 > add host 69.16.175.42: gateway 192.168.42.129 fib 0=20 > add host 69.16.175.10: gateway 192.168.42.129 fib 0=20 > add host 93.20.64.1: gateway 192.168.42.129 fib 0 Here, my setup is working also correctly: > 2021-10-10 12:19:03.752618 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.61024 > 109.0.64.169.80: Flags [S], seq 3895230720, win 65= 535, > options [mss 1460,nop,wscale 6,sack OK,TS val 434432976 ecr 0], length 0 > 2021-10-10 12:19:07.217710 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.63790 > 109.0.64.169.80: Flags [S], seq 1094822977, win 65= 535, > options [mss 1460,nop,wscale 6,sack OK,TS val 3161827496 ecr 0], length = 0 > 2021-10-10 12:19:07.219786 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.64426 > 69.16.175.10.80: Flags [S], seq 3835135791, win 65= 535, > options [mss 1460,nop,wscale 6,sack OK,TS val 2326859600 ecr 0], length = 0 > 2021-10-10 12:19:07.221208 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.62982 > 109.0.64.169.80: Flags [S], seq 2534996192, win 65= 535, > options [mss 1460,nop,wscale 6,sack OK,TS val 3218952486 ecr 0], length = 0 > 2021-10-10 12:19:07.765953 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.55202 > 93.20.64.1.80: Flags [S], seq 2614725609, win 6553= 5, > options [mss 1460,nop,wscale 6,sackOK ,TS val 370574760 ecr 0], length 0 > 2021-10-10 12:19:08.057355 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.58890 > 93.20.64.1.80: Flags [S], seq 3356530101, win 6553= 5, > options [mss 1460,nop,wscale 6,sackOK ,TS val 1210627653 ecr 0], length = 0 Then, I want to isolate this service and make it usable only when pf is enabled. So, I stop this jail, change his configuration to: > ... > exec.fib =3D 1; >... Change vsw0 configuration to: > # ifconfig vsw0 fib 1 And start the jail again.=20 At this point my routes via ue0 are still active, and the setup is working. > 2021-10-10 12:32:35.790102 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17138 > 109.0.64.169.80: Flags [S], seq 3903052933, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0 > 2021-10-10 12:32:35.790155 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17138 > 109.0.64.169.80: Flags [S], seq 3903052933, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0 > 2021-10-10 12:32:35.790211 rule 1/0(match): nat out on ue0: > 192.168.42.93.56331 > 109.0.64.169.80: Flags [S], seq 3903052933, win 65= 535, > options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0 > 2021-10-10 12:32:40.997496 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17141 > 109.0.64.169.80: Flags [S], seq 3383832228, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length = 0 > 2021-10-10 12:32:40.997555 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17141 > 109.0.64.169.80: Flags [S], seq 3383832228, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length = 0 > 2021-10-10 12:32:40.997619 rule 1/0(match): nat out on ue0: > 192.168.42.93.57246 > 109.0.64.169.80: Flags [S], seq 3383832228, win 65= 535, > options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length = 0 > 2021-10-10 12:32:41.002776 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17142 > 109.0.64.169.80: Flags [S], seq 1628607117, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length = 0 > 2021-10-10 12:32:41.002835 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17142 > 109.0.64.169.80: Flags [S], seq 1628607117, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length = 0 > 2021-10-10 12:32:41.002911 rule 1/0(match): nat out on ue0: > 192.168.42.93.60146 > 109.0.64.169.80: Flags [S], seq 1628607117, win 65= 535, > options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length = 0 > 2021-10-10 12:32:41.003179 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17143 > 69.16.175.10.80: Flags [S], seq 3985646764, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length = 0 > 2021-10-10 12:32:41.003224 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17143 > 69.16.175.10.80: Flags [S], seq 3985646764, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length = 0 > 2021-10-10 12:32:41.003276 rule 1/0(match): nat out on ue0: > 192.168.42.93.60290 > 69.16.175.10.80: Flags [S], seq 3985646764, win 65= 535, > options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length = 0 > 2021-10-10 12:32:44.224200 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17149 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0 > 2021-10-10 12:32:44.224257 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17149 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0 > 2021-10-10 12:32:44.224319 rule 1/0(match): nat out on ue0: > 192.168.42.93.57313 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535= , > options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0 > 2021-10-10 12:32:44.297253 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17150 > 93.20.64.1.80: Flags [S], seq 3451366591, win 65535= , > options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0 > 2021-10-10 12:32:44.297313 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17150 > 93.20.64.1.80: Flags [S], seq 3451366591, win 65535= , > options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0 > 2021-10-10 12:32:44.297368 rule 1/0(match): nat out on ue0: > 192.168.42.93.64517 > 93.20.64.1.80: Flags [S], seq 3451366591, win 6553= 5, > options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0 > 2021-10-10 12:32:44.900273 rule 1/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17151 > 54.148.181.117.443: Flags [S], seq 1363702430, win > 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], l= ength > 0=20 > 2021-10-10 12:32:44.900326 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17151 > 54.148.181.117.443: Flags [S], seq 1363702430, win > 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], l= ength > 0=20 > 2021-10-10 12:32:44.900385 rule 2/0(match): nat out on em0: > 192.168.1.50.51385 > 54.148.181.117.443: Flags [S], seq 1363702430, win > 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], l= ength > 0 The problem begins when I want to stop adsl failure simulation and delete t= he routes I added before. > # D ; route del 109.0.64.169 ; route del 69.16.175.42 ; route del 69.16.1= 75.10 > ; route del 93.20.64.1=20 > del host 109.0.64.169 fib 0 > del host 69.16.175.42 fib 0 > del host 69.16.175.10 fib 0 > del host 93.20.64.1 fib 0 >From there, the service have no connectivity to the outside anymore. What I= see on screen is that it tries to connect until it timeouts. > 2021-10-10 13:04:18.446492 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.53781 > 109.0.64.169.80: Flags [S], seq 2140928614, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length = 0 > 2021-10-10 13:04:18.446543 rule 2/0(match): pass in on vsw0: > 192.168.1.92.53781 > 109.0.64.169.80: Flags [S], seq 2140928614, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length = 0 > 2021-10-10 13:04:18.446604 rule 2/0(match): nat out on em0: > 192.168.1.50.51537 > 109.0.64.169.80: Flags [S], seq 2140928614, win 655= 35, > options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length = 0 And tcpdump show that he tries again and again without success. > 13:04:18.446626 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:21.451833 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:21.460066 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:21.460221 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:21.460635 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:21.967652 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:22.778014 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:22.982230 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:22.982328 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:24.197788 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:24.982527 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:24.982620 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:26.862180 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:28.982095 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:28.982191 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:31.948459 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:36.982090 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:36.982207 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:40.728729 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:45.415470 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:45.423316 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:58.085578 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:58.099480 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 1448 > 13:04:58.099594 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:58.100358 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 1168 > 13:04:58.100391 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 As we can see, traffic from the address I want to join comes back to host a= nd it seems it disapears somewhere.=20 The strange part is that routing via 4g/ue0 is working and 4g/ue0 is simila= r to adsl/em0 in physical terms. What could I check else to see what's happening ? Is there a better place m= aybe to ask questions like this ? Are there other informations needed ?=20 Thanks, K. From nobody Sun Oct 10 21:00:36 2021 X-Original-To: pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6A15C17E3CE8 for ; Sun, 10 Oct 2021 21:00:37 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HSDps09yYz4lTY for ; Sun, 10 Oct 2021 21:00:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 57B5E14952 for ; Sun, 10 Oct 2021 21:00:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 19AL0a9w052326 for ; Sun, 10 Oct 2021 21:00:36 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 19AL0aoC052324 for pf@FreeBSD.org; Sun, 10 Oct 2021 21:00:36 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202110102100.19AL0aoC052324@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: pf@FreeBSD.org Subject: Problem reports for pf@FreeBSD.org that need special attention Date: Sun, 10 Oct 2021 21:00:36 +0000 List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16338996361.6E9ca1dE.51299" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: Y --16338996361.6E9ca1dE.51299 Date: Sun, 10 Oct 2021 21:00:36 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 237973 | pf: implement egress keyword to simplify rules ac 1 problems total for which you should take action. --16338996361.6E9ca1dE.51299--