Date: Sat, 13 Mar 2021 15:55:54 +0530 From: Ashish SHUKLA <ashish@FreeBSD.org> To: freebsd-stable@freebsd.org Subject: TCP BBR in 13.0-RC2 not playing well with pf firewall Message-ID: <E7AF40EB-3C9A-49EE-8152-5C10914C9CB7@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
[sorry for some reason, the previous message is showing as empty in = archives :(] Hi, I just wanted to try TCP BBR functionality available in FreeBSD 13.0-RC2 = and noticed that it=E2=80=99s not usable, if FreeBSD pf is enabled. I=E2=80=99m using following pf rules: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D block drop all block drop in quick on em0 from <badguys> to any block drop in quick on em0 from <sshguard> to any label "ssh bruteforce" pass in on em0 proto udp from any to any port =3D domain keep state pass in on em0 proto udp from any to any port =3D 2015 keep state pass in on em0 proto udp from any to any port =3D 5001 keep state pass in on em0 proto udp from any to any port =3D 6881 keep state pass in on em0 proto udp from any to any port =3D 51234 keep state pass in on em0 proto udp from any to any port 54000:54322 keep state pass in on em0 proto udp from any to any port 60000:61000 keep state pass in on em0 proto tcp from any to any port =3D ssh flags S/SA keep = state pass in on em0 proto tcp from any to any port =3D xmpp-client flags S/SA = keep state pass in on em0 proto tcp from any to any port =3D xmpp-server flags S/SA = keep state pass in on em0 proto tcp from any to any port 54000:54322 flags S/SA = keep state pass in on em0 proto icmp all keep state pass in on em0 proto ipv6-icmp all keep state pass out on em0 all flags S/SA keep state =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Following is how I tried to reproduce (same with IPv6): =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D % wget -4 -O /dev/null = https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz --2021-03-13 09:26:23-- = https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.113.176 Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... = connected. HTTP request sent, awaiting response... 200 OK Length: 117629864 (112M) [application/x-xz] Saving to: =E2=80=98/dev/null=E2=80=99 /dev/null 0%[ = ] 137.44K --.-KB/s in 10s =20 2021-03-13 09:26:33 (13.8 KB/s) - Read error at byte 140737/117629864 = (Network is down). Retrying. --2021-03-13 09:26:34-- (try: 2) = https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... = connected. HTTP request sent, awaiting response... 206 Partial Content Length: 117629864 (112M), 117489127 (112M) remaining [application/x-xz] Saving to: =E2=80=98/dev/null=E2=80=99 /dev/null 0%[ = ] 243.21K --.-KB/s in 10s =20 2021-03-13 09:26:44 (10.6 KB/s) - Read error at byte 249046/117629864 = (Network is down). Retrying. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D A copy of tcpdump output (tcpdump -i em0 -nv port 443 and host = 151.101.113.176) is uploaded[1]. Since TCP BBR requires custom kernel, following is the diff: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- GENERIC 2021-02-23 10:55:03.397376000 +0000 +++ MYKERNEL 2021-02-23 10:58:39.442981000 +0000 @@ -19,7 +19,7 @@ # $FreeBSD$ cpu HAMMER -ident GENERIC +ident MYKERNEL makeoptions DEBUG=3D-g # Build kernel with gdb(1) = debug symbols makeoptions WITH_CTF=3D1 # Run ctfconvert(1) for DTrace = support @@ -382,3 +382,6 @@ options HID_DEBUG # enable debug msgs device hid # Generic HID support options IICHID_SAMPLING # Workaround missing GPIO INTR = support + +options ROUTETABLES=3D12 +options TCPHPTS =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D And following src.conf: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D WITH_EXTRA_TCP_STACKS=3D yes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D sysctl.conf(5) is empty, except following sysctl knob set to switch to = BBR stack: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D net.inet.tcp.functions_default=3Dbbr =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Following are the contents of loader.conf(5): =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D kern.geom.label.disk_ident.enable=3D"0" kern.geom.label.gptid.enable=3D"0" opensolaris_load=3D"YES" zfs_load=3D"YES" debug.acpi.disabled=3D"thermal" coretemp_load=3DYES vm.pmap.pti=3D0 cpu_microcode_load=3D"YES" cpu_microcode_name=3D"/boot/firmware/intel-ucode.bin" aesni_load=3DYES nullfs_load=3D"YES" vfs.zfs.prefetch_disable=3D1 vfs.zfs.write_limit_override=3D1073741824 vfs.zfs.min_auto_ashift=3D12 vfs.zfs.arc_max=3D"17179869184" vfs.zfs.compressed_arc_enabled=3D1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Following is the output of kldstat: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Id Refs Address Size Name 1 33 0xffffffff80200000 1f12e50 kernel 2 1 0xffffffff82113000 b7b8 opensolaris.ko 3 1 0xffffffff8211f000 9ac0 nullfs.ko 4 1 0xffffffff8249d000 67fbc8 zfs.ko 5 1 0xffffffff82b1d000 4128 coretemp.ko 6 1 0xffffffff82d20000 3378 acpi_wmi.ko 7 1 0xffffffff82d24000 3250 ichsmb.ko 8 1 0xffffffff82d28000 2180 smbus.ko 9 1 0xffffffff82d2b000 2110 pchtherm.ko 10 1 0xffffffff82d2e000 2a08 mac_ntpd.ko 11 1 0xffffffff82d31000 29bc8 tcp_bbr.ko 12 1 0xffffffff82d5b000 36f70 pf.ko =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Host is running kernel from git revision = =E2=80=9C13c22f7495305f5b92874128b088ab47d9512c20=E2=80=9D, while rest = of the userland is from "freebsd-update upgrade -r 13.0-RC2=E2=80=9D. After disabling pf firewall, it becomes usable again: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --2021-03-13 09:42:50-- = https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.113.176 Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... = connected. HTTP request sent, awaiting response... 200 OK Length: 117629864 (112M) [application/x-xz] Saving to: =E2=80=98/dev/null=E2=80=99 /dev/null = 100%[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D>] 112.18M 20.2MB/s in 6.9s =20 2021-03-13 09:42:57 (16.3 MB/s) - =E2=80=98/dev/null=E2=80=99 saved = [117629864/117629864] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Please let me know if I overlooked something with TCP BBR configuration. References: [1] = https://people.freebsd.org/~ashish/tcpdump-cdn-kernel-org-20210313.txt Thanks! -- Ashish | GPG: F682 CDCC 39DC 0FEA E116 20B6 C746 CFA9 E74F A4B0 =E2=80=9CSometimes even to live is an act of courage.=E2=80=9D (Seneca)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E7AF40EB-3C9A-49EE-8152-5C10914C9CB7>