From owner-freebsd-pf@freebsd.org Tue Jan 10 08:29:02 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51C25CA9F32 for ; Tue, 10 Jan 2017 08:29:02 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [88.199.43.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "plan-b.pwste.edu.pl", Issuer "plan-b.pwste.edu.pl" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D189611AD for ; Tue, 10 Jan 2017 08:29:01 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (zarychtam@localhost [127.0.0.1]) by plan-b.pwste.edu.pl (8.15.2/8.15.2) with ESMTPS id v0A8SuPA094345 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 10 Jan 2017 09:28:56 +0100 (CET) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: (from zarychtam@localhost) by plan-b.pwste.edu.pl (8.15.2/8.15.2/Submit) id v0A8Stqk094340; Tue, 10 Jan 2017 09:28:55 +0100 (CET) (envelope-from zarychtam) Date: Tue, 10 Jan 2017 09:28:55 +0100 From: Marek Zarychta To: Ian FREISLICH Cc: freebsd-pf@freebsd.org Subject: Re: udp - weird behavior of reply-to Message-ID: <20170110082855.GA90712@plan-b.pwste.edu.pl> References: <20170108145532.GA17695@plan-b.pwste.edu.pl> <20170109172519.GA62580@plan-b.pwste.edu.pl> <20170109221712.GA49594@plan-b.pwste.edu.pl> <1397ba00-3ebf-28fb-4f06-f026899865bb@capeaugusta.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="NzB8fVQJ5HfG6fxh" Content-Disposition: inline In-Reply-To: <1397ba00-3ebf-28fb-4f06-f026899865bb@capeaugusta.com> User-Agent: Mutt/1.7.2 (2016-11-26) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jan 2017 08:29:02 -0000 --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 09, 2017 at 10:01:21PM -0500, Ian FREISLICH wrote: > On 01/09/17 17:17, Marek Zarychta wrote: > > On Mon, Jan 09, 2017 at 09:58:38PM +0100, Kristof Provost wrote: > >> On 9 Jan 2017, at 18:25, Marek Zarychta wrote: > >>> On Sun, Jan 08, 2017 at 07:08:10PM +0100, Kristof Provost wrote: > >>>> On 8 Jan 2017, at 15:55, Marek Zarychta wrote: > >>>> The problem description doesn=E2=80=99t ring any bells with me, but = I=E2=80=99m > >>>> also > >>>> not sure > >> I=E2=80=99ve fully understood it. Can you document a = minimal reproduction > >>>> scenario, > >>>> with a pf.conf and perhaps network captures documenting the problem? > >>>> > >>>> There=E2=80=99s certainly not been a conscious decision to break UDP > >>>> reply-to. > >>>> > >>> Let me apologize, the problem wasn't previously properly identified. > >>> It > >>> seems to be more problem of UDP protocol implementation than PF issue. > >>> UDP sockets are opened and bound to address of the outgoing interface > >>> (interface which has a route to the client). Because the socket is not > >>> bound to the incoming interface, the PF reply-to rules couldn't be > >>> evaluated. By the way, TCP sockets are bound to the interface where > >>> the > >>> traffic arrives and everything works fine. > >>> This machine is i386 running 11.0-STABLE r311772 > >>> > >>> The problem remains unresolved. Are there any corresponding sysctls > >>> correcting this behavior and enabling the opportunity to use PF > >>> assisted > >>> symmetric routing scenario again? > >>> > >> How are your UDP listen sockets set up? > >> Are they bound to a specific interface, or are they listening on > >> 0.0.0.0? > > Yes, socket is listening on 0.0.0.0, the client from outside network is > > initiating connection and initiating packet arrives on interface B, > > which is supposed only to communicate with devices on its own network > > (no additional routes go via this interface), so normally the reply > > would be passed via interface A having default gateway in scope and > > communication would fail. > > With the assistance of PF reply-to rule, TCP services are able to pass > > reply from interface B via other, second gateway: reply-to (B GW2). >=20 > Are you saying that your network looks approximately like this and there= =20 > is no route to the client network where X resides on your server: >=20 > iface-A----GW1 > iface-B--_local network_--GW2--_client X_ >=20 > Client X originates a UDP "connection" to B and that return traffic to X= =20 > leaves interface A despite your reply-to rule. >=20 > I would be very interested to know: >=20 > 1. whether the reply-to rule actually matches on the inbound traffic. =20 > You can make the rule log and tcpdump on the pflog0 interface. I=20 > believe the -e option to tcpdump will show the rule that matched. >=20 > 2. the output of "pfctl -s sta |grep IP_of_X" >=20 > 3. what software you're using for your UDP server. I can try to=20 > reproduce your issue. >=20 > Ian Both interfaces have public IP addresses, server has a route to client via GW1, client originates connection to B via GW2. TCP replies originate from socket associated with (bound to) interface B so reply-to works as expected and client is able to connect to both interfaces simultaneously. Corresponding PF states are shown below all tcp ADDR_A:22 <- CLIENT_ADDR:57598 ESTABLISHED:ESTABLISHED all tcp ADDR_B:22 <- CLIENT_ADDRR:58856 ESTABLISHED:ESTABLISHED UDP datagrams never originate from socket B ignoring client effort to use socket associated with interface B, what was investigated using sockstat. Server always tries to respond via interface A and simply create new PF state as shown below.=20 all udp ADDR_B:1194 <- CLIENT_ADDRR:35607 NO_TRAFFIC:SINGLE all udp ADDR_A:1194 -> CLIENT_ADDRR:35607 SINGLE:NO_TRAFFIC This is also seen by pflog: 06:38:52.195266 rule 487/0(match): pass in on B: CLIENT_ADDRR.35607 > ADDR_B.1194: UDP, length 42 06:38:52.196970 rule 488/0(match): pass out on A: ADDR_A.1194 > CLIENT_ADDRR.35607: UDP, length 54 Rules 487 and 488 are shown below: pass in log quick on $B \ reply-to ( $B $GW2 ) \ inet proto udp \ from any \ to ($ext_if_2:0) port $udp_services \ keep state pass out log quick on $A proto udp \ from any to any \ keep state =20 Successful setup has been previously used to allow client X use address of interface B as OpenVPN remote address. The scenario could be reproduced simply using netcat as a server and client: nc -u -l 1194, nc -u ADDR_B 1194. =20 >=20 > > This functionality is currently broken for any UDP service, because UDP > > sockets are always opened on supposed_to_be_outgoing interface A and > > bound to the address of this interface, which is considered good from > > routing table perspective, but silently breaks PF reply-to for UDP. > > > > When the machine was running 9-STABLE reply-to had successfully been > > used to assist both TCP and UDP driven services. > > > > Is anyone reading this list still using reply-to rule for routing UDP > > traffic back via incoming interface? > > > > Maybe currently, the better place to discuss this questions would be > > freebsd-net, but the thread was initiated here. > > > >> I=E2=80=99m afraid I=E2=80=99m still struggling to understand what you= r setup is, > >> what you=E2=80=99re > >> expecting to see and what you=E2=80=99re seeing instead. > >> > >> Regards, > >> Kristof > > > > >=20 >=20 > --=20 > =20 >=20 > Cape Augusta Digital Properties, LLC a Cape Augusta Company >=20 > *Breach of confidentiality & accidental breach of confidentiality * >=20 > This email and any files transmitted with it are confidential and intende= d=20 > solely for the use of the individual or entity to whom they are addressed= =2E=20 > If you have received this email in error please notify the system manager= =2E=20 > This message contains confidential information and is intended only for t= he=20 > individual named. If you are not the named addressee you should not=20 > disseminate, distribute or copy this e-mail. Please notify the sender=20 > immediately by e-mail if you have received this e-mail by mistake and=20 > delete this e-mail from your system. If you are not the intended recipien= t=20 > you are notified that disclosing, copying, distributing or taking any=20 > action in reliance on the contents of this information is strictly=20 > prohibited. > _______________________________________________ > freebsd-pf@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" >=20 --=20 Marek Zarychta --NzB8fVQJ5HfG6fxh Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEMOqvKm6wKvS1/ZeCdZ/s//1SjSwFAlh0m0QACgkQdZ/s//1S jSy76AgAtJLxOOy0k6GhvgDggr0a+QzYJEKTnHN6qZFAzwDkGdWrhRziJP2B1hBI 3eNoTRi4yat+YEa33/iktWFeV6oS7ro2IbQtCFucFmjYS5f0QixMk+rI3rVLRG7u ZldQy+9u3eDu53cCJgTbtgwm0yW2l0E9LviH9ICibksmHB3bJqsZxf3HzTqqvVTr A417nNx2MPyS2KR1O/wJO0N32qI53ETO3iZpySvBorsCTowBEFVRyVsvCcRqPPcQ AWufCKS/gDmHFQiqOJw9ToOfwVTSrFzrYdSfdX2JUccOrBaBHkkrkRazzQzSPBON ITRLJ9C94c/erOt8FlD67Cwv9TcAlg== =23De -----END PGP SIGNATURE----- --NzB8fVQJ5HfG6fxh--