Date: Tue, 10 Jan 2017 09:28:55 +0100 From: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> To: Ian FREISLICH <ian.freislich@capeaugusta.com> Cc: freebsd-pf@freebsd.org Subject: Re: udp - weird behavior of reply-to Message-ID: <20170110082855.GA90712@plan-b.pwste.edu.pl> In-Reply-To: <1397ba00-3ebf-28fb-4f06-f026899865bb@capeaugusta.com> References: <20170108145532.GA17695@plan-b.pwste.edu.pl> <E8BB68F1-4784-474A-B5ED-1E861B2975A8@FreeBSD.org> <20170109172519.GA62580@plan-b.pwste.edu.pl> <D90E48F5-F25A-49AC-AD42-F1FD3BAFD4A0@FreeBSD.org> <20170109221712.GA49594@plan-b.pwste.edu.pl> <1397ba00-3ebf-28fb-4f06-f026899865bb@capeaugusta.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--NzB8fVQJ5HfG6fxh
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, Jan 09, 2017 at 10:01:21PM -0500, Ian FREISLICH wrote:
> On 01/09/17 17:17, Marek Zarychta wrote:
> > On Mon, Jan 09, 2017 at 09:58:38PM +0100, Kristof Provost wrote:
> >> On 9 Jan 2017, at 18:25, Marek Zarychta wrote:
> >>> On Sun, Jan 08, 2017 at 07:08:10PM +0100, Kristof Provost wrote:
> >>>> On 8 Jan 2017, at 15:55, Marek Zarychta wrote:
> >>>> The problem description doesn=E2=80=99t ring any bells with me, but =
I=E2=80=99m
> >>>> also
> >>>> not sure > >> I=E2=80=99ve fully understood it. Can you document a =
minimal reproduction
> >>>> scenario,
> >>>> with a pf.conf and perhaps network captures documenting the problem?
> >>>>
> >>>> There=E2=80=99s certainly not been a conscious decision to break UDP
> >>>> reply-to.
> >>>>
> >>> Let me apologize, the problem wasn't previously properly identified.
> >>> It
> >>> seems to be more problem of UDP protocol implementation than PF issue.
> >>> UDP sockets are opened and bound to address of the outgoing interface
> >>> (interface which has a route to the client). Because the socket is not
> >>> bound to the incoming interface, the PF reply-to rules couldn't be
> >>> evaluated. By the way, TCP sockets are bound to the interface where
> >>> the
> >>> traffic arrives and everything works fine.
> >>> This machine is i386 running 11.0-STABLE r311772
> >>>
> >>> The problem remains unresolved. Are there any corresponding sysctls
> >>> correcting this behavior and enabling the opportunity to use PF
> >>> assisted
> >>> symmetric routing scenario again?
> >>>
> >> How are your UDP listen sockets set up?
> >> Are they bound to a specific interface, or are they listening on
> >> 0.0.0.0?
> > Yes, socket is listening on 0.0.0.0, the client from outside network is
> > initiating connection and initiating packet arrives on interface B,
> > which is supposed only to communicate with devices on its own network
> > (no additional routes go via this interface), so normally the reply
> > would be passed via interface A having default gateway in scope and
> > communication would fail.
> > With the assistance of PF reply-to rule, TCP services are able to pass
> > reply from interface B via other, second gateway: reply-to (B GW2).
>=20
> Are you saying that your network looks approximately like this and there=
=20
> is no route to the client network where X resides on your server:
>=20
> iface-A--<default route>--GW1
> iface-B--_local network_--GW2--_client X_
>=20
> Client X originates a UDP "connection" to B and that return traffic to X=
=20
> leaves interface A despite your reply-to rule.
>=20
> I would be very interested to know:
>=20
> 1. whether the reply-to rule actually matches on the inbound traffic. =20
> You can make the rule log and tcpdump on the pflog0 interface. I=20
> believe the -e option to tcpdump will show the rule that matched.
>=20
> 2. the output of "pfctl -s sta |grep IP_of_X"
>=20
> 3. what software you're using for your UDP server. I can try to=20
> reproduce your issue.
>=20
> Ian
Both interfaces have public IP addresses, server has a route to client
via GW1, client originates connection to B via GW2. TCP replies
originate from socket associated with (bound to) interface B so reply-to
works as expected and client is able to connect to both interfaces
simultaneously. Corresponding PF states are shown below
all tcp ADDR_A:22 <- CLIENT_ADDR:57598 ESTABLISHED:ESTABLISHED
all tcp ADDR_B:22 <- CLIENT_ADDRR:58856 ESTABLISHED:ESTABLISHED
UDP datagrams never originate from socket B ignoring client effort to
use socket associated with interface B, what was investigated using
sockstat. Server always tries to respond via interface A and simply
create new PF state as shown below.=20
all udp ADDR_B:1194 <- CLIENT_ADDRR:35607 NO_TRAFFIC:SINGLE
all udp ADDR_A:1194 -> CLIENT_ADDRR:35607 SINGLE:NO_TRAFFIC
This is also seen by pflog:
06:38:52.195266 rule 487/0(match): pass in on B: CLIENT_ADDRR.35607
> ADDR_B.1194: UDP, length 42
06:38:52.196970 rule 488/0(match): pass out on A: ADDR_A.1194 >
CLIENT_ADDRR.35607: UDP, length 54
Rules 487 and 488 are shown below:
pass in log quick on $B \
reply-to ( $B $GW2 ) \
inet proto udp \
from any \
to ($ext_if_2:0) port $udp_services \
keep state
pass out log quick on $A proto
udp \
from any to any \
keep state
=20
Successful setup has been previously used to allow client X use address
of interface B as OpenVPN remote address. The scenario could be
reproduced simply using netcat as a server and client: nc -u -l 1194,
nc -u ADDR_B 1194. =20
>=20
> > This functionality is currently broken for any UDP service, because UDP
> > sockets are always opened on supposed_to_be_outgoing interface A and
> > bound to the address of this interface, which is considered good from
> > routing table perspective, but silently breaks PF reply-to for UDP.
> >
> > When the machine was running 9-STABLE reply-to had successfully been
> > used to assist both TCP and UDP driven services.
> >
> > Is anyone reading this list still using reply-to rule for routing UDP
> > traffic back via incoming interface?
> >
> > Maybe currently, the better place to discuss this questions would be
> > freebsd-net, but the thread was initiated here.
> >
> >> I=E2=80=99m afraid I=E2=80=99m still struggling to understand what you=
r setup is,
> >> what you=E2=80=99re
> >> expecting to see and what you=E2=80=99re seeing instead.
> >>
> >> Regards,
> >> Kristof
> >
> >
>=20
>=20
> --=20
> =20
>=20
> Cape Augusta Digital Properties, LLC a Cape Augusta Company
>=20
> *Breach of confidentiality & accidental breach of confidentiality *
>=20
> This email and any files transmitted with it are confidential and intende=
d=20
> solely for the use of the individual or entity to whom they are addressed=
=2E=20
> If you have received this email in error please notify the system manager=
=2E=20
> This message contains confidential information and is intended only for t=
he=20
> individual named. If you are not the named addressee you should not=20
> disseminate, distribute or copy this e-mail. Please notify the sender=20
> immediately by e-mail if you have received this e-mail by mistake and=20
> delete this e-mail from your system. If you are not the intended recipien=
t=20
> you are notified that disclosing, copying, distributing or taking any=20
> action in reliance on the contents of this information is strictly=20
> prohibited.
> _______________________________________________
> freebsd-pf@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org"
>=20
--=20
Marek Zarychta
--NzB8fVQJ5HfG6fxh
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEMOqvKm6wKvS1/ZeCdZ/s//1SjSwFAlh0m0QACgkQdZ/s//1S
jSy76AgAtJLxOOy0k6GhvgDggr0a+QzYJEKTnHN6qZFAzwDkGdWrhRziJP2B1hBI
3eNoTRi4yat+YEa33/iktWFeV6oS7ro2IbQtCFucFmjYS5f0QixMk+rI3rVLRG7u
ZldQy+9u3eDu53cCJgTbtgwm0yW2l0E9LviH9ICibksmHB3bJqsZxf3HzTqqvVTr
A417nNx2MPyS2KR1O/wJO0N32qI53ETO3iZpySvBorsCTowBEFVRyVsvCcRqPPcQ
AWufCKS/gDmHFQiqOJw9ToOfwVTSrFzrYdSfdX2JUccOrBaBHkkrkRazzQzSPBON
ITRLJ9C94c/erOt8FlD67Cwv9TcAlg==
=23De
-----END PGP SIGNATURE-----
--NzB8fVQJ5HfG6fxh--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170110082855.GA90712>