From owner-freebsd-current@freebsd.org Fri Apr 24 21:12:09 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 649AA2C260A; Fri, 24 Apr 2020 21:12:09 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "anubis.delphij.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4986Lc39TSz484K; Fri, 24 Apr 2020 21:12:08 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from odin.corp.delphij.net (unknown [IPv6:2601:646:8600:58ba:f8e4:80ce:7871:5fe6]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 94B7322E34; Fri, 24 Apr 2020 14:12:06 -0700 (PDT) Reply-To: d@delphij.net Subject: Re: CFT: if_bridge performance improvements To: Kristof Provost , d@delphij.net Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org References: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> <8e0e2bf1-27cd-1a99-b266-c7223255942f@delphij.net> <8634ec5c-a509-d2dd-8f5c-31efcbd50340@delphij.net> <544E27A6-D799-4AF3-B4B7-1E68D5D50698@FreeBSD.org> From: Xin Li Autocrypt: addr=delphij@delphij.net; prefer-encrypt=mutual; keydata= mQINBFuSR4oBEACvvEgwRIHs6IcSP/yaDtySF78Ji3rP29qdiQsxhMsOtvtffdbS56VApIWO UFb3/iN2gA8HwLvrmjijN0HEoLVX7na1WARmxRYzQMtApsZIUTtx7hnUYlsi2F5odZa6CDW9 a954DLRzYxiUwYDcu5Zjl9bglK1H8e/N9uC0Vuigr4teWfh86brzOyf819QzwFVYfMIK4ihw QGwMvTzbyVuCFy+LENkmcVYni70oQy6rZ5ktSuYbuOFvu7inRRfhSWPHziV7k+bW88sJ7xhv lBlegcnhkSudWX2M8tZ3MO1PJOcyys0CJlsBY5Weiog2lIPi05h/E9pZ9mc1Vud17iqDaL6w RaggOUhuPfDGCdO5ro82W4BZGeQMRnRF5Ntk+t2ShIH4nn3xRLV0E5nziCiKlgiMqOrz/ZTL QTVbHrCuiwD+fSK14y0oHbkOLYTYLlgh1JbwfY2Ty7elOYiWzyeJ7sJh2dF91NSEneWIOys3 mBpuvtU3nSzzTvAB48VV+Nbg1CpIOgNlPjj7uhIum/Z/VjUaJEyaLpTIRh0MVJVcbP7hXSqZ NA35EEZZVnWEOYdycm4CmEdeNPWkrAf2Ya77iR5VLGypwMlsUMQPh+sKVWDD38M8stFGBBNm d01Hi74Bsq5hKan654dOqMt5eYklrVj0ucMzFQtus7oE502UswARAQABtBxYaW4gTEkgPGRl bHBoaWpAZGVscGhpai5uZXQ+iQJUBBMBCgA+FiEEceNg5NEMZIki80nQQHl/fJX0g08FAluS R/YCGwMFCQmuhAAFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQQHl/fJX0g0+2Og//bWpE F2V5/M5l6YW1T8oLcT9rIOH6oq9M0LMNRgFeiNNnilGIeeIgtOGBRueG4CZiZAvsRPJkrO70 1R2SrdkCIvwGUzUAxx1NfBWb+vgm4fgkW/MotGonceM5v0qfSKKXasWvDctkK28aG+IoQzmi FjXNW4+ju4zeQFYwD4ZDWqw9MqO0hVb24uW3dxtQhbfmOLgJ/PEDMQaFuANbW1c+iR0BQA3D Go/EeMY4kpN8on6Aqt/S/4JVltudfQ9OXdjQsC7netSaB9K3mHGt9aKAAB7RzlRY00DKkYS/ /eQwLzGPmK7yX13M68mMDjBs6mIR8t/E1S5OdBNhHRPNPlEbwugR4KaiCsN5yqzJoSV99fKY z2VyxjWPaG8yhHE+jmKUgIBKTfFUQEfkriQR4EASoeJ+soaMTiFDBij1Zw5n3ndLRFMB1ZCl fZLER36mAgW4m4kP83TWnDiJLxOxSOxifV8HpTFjff902H85cybg9KMwrfPDr6W19GGk5Vo1 fkza5krRMGbKWb7+74Evusi0ZxJLIOFwp5Y8eVqUMZaAD3f1ZX1M3pgXOp20QgAy+2KvMHij rLa4q+tMGRzYYD1BnFVSVdXAX5VOoTmHBcDz67DkuRwk2Byp1sgd407oEOmSwrNJlKS0TPCm xUJ2fdSQF+1/MMSRfee49vtMvz7cOrC5Ag0EW5JHigEQANiBmIFAfRNH3nzYNWC0yC+tfx3z sUwAsH1VaBM/cTib+yKtbBOSIlXWjJZWX3MHwoI/1LeGghB2mxkkX1L0pJ/vj1eXNR+sFZ32 0pYcl61Fxg/5fioG4QDTM4i3i7NR5PxDnc6UVaynSlII93DedRhZ1ROtdn4vyMgzsDiqhbL7 BthDOt5KxjqdRk4qRPSw7BovEqZLOcG5IJtf/zZUzRbM7SBljEbOAfekDGx1Br+RrYSD7/Ef Pwwzou9T8315IpBpIHyQF/dZNk3iFiB9Ed5CA71ZRYV5YoLWE9lL0j9kxOLQ5vHnX3mVq7QZ Bc7nzwZ6UhQgYmrG5+RWvuiPpGwvDRIsugJUGXucYkAQh5kuNblmkwpv6u9rNMjCNbzAylOa qdogra5EW+RUSbRz0b4iIr8nnZeAlh7BihCe7JjOwbDjoBEEEtSfVc4hD/LENqpcYVrChphf aOLB9YIXhnVDTVvMc9OklWT/81HzAaDQqOQCzEfY92199Ct9/CwRoQ2OpO8TO5+8A7b9Nb33 nmxMn09mb48ruRacMrfHxCWbgU4w9SEfbip4GcS5wGG6yTC+hw55Iwnnwus40NrJ0GEr8a4r cdsLbkvlyoNHB8ZGgyJ4aFCQ1V4qE1BnlTk7Z8BYBUkJM1odPSkVvHpCnMUjVpJ3hEOC+73Z YH1dh7lZABEBAAGJAjwEGAEKACYWIQRx42Dk0QxkiSLzSdBAeX98lfSDTwUCW5JHigIbDAUJ Ca6EAAAKCRBAeX98lfSDTz8DEACMh3poeUb+gWNF4RWFZuLteZVo0+E1JLYXQkmtrRBLXviP +Qy0pXyFAVxLM4hNIBoIDYfK9BcwrBYf7AwSKrH0GiNwFpgHCkbZd6qoZy2gB+adTnCpVCTJ KJetsH/8awkrChJWMK0ckGf3EeWMPvawG7kW7FBz70NYEZ0pOMiaEZNVtzD3wwbYWUiDFYth 83XGglOExg+1ShTW5XjQPRrdyJAO+aUW4o3lVjfyUJXMgI4rmhMiLVm06GuNrbpKIF0s+4Vd jQAjhrDQjfoXi9CkfsA/cONseuHNv1JGj3RqHiqHJq1dbrpodXp925zGDAnUGxCOBPoFopAH gVzR89GTut059GpwqsddZmU6y7rqifuam/ekJ+QRwc16vgt7pHqCrTY8WPxRZr2UpFU1wlTo COdeiFep1gq1F9jzFjJnoMaAdmC6k7bgAA+RQusOgIhJL0jIej7DoAHxmxFFCfRy+lDtpXwF gQ8HMvzHI65QWmQnMo7s6SQH/ZH5s1yR6SJq8+3lDz+dCuT42qJVqIPVvxd10LW0FNN+t7HF eLadU6ekSgD13/EYMYXlvNHkw7dAItSDxIzgRyykLz0bCU9xwNWoS4Z43+ifF9anJ+uR0ltW El1j++h6ZrD3LLuCgJIt1so0m49GzdcSpOI7LCwMlacyvafiEyjUn+tSNDsnfw== Message-ID: <5246fd44-b856-5402-092e-9065d4d4f7ae@delphij.net> Date: Fri, 24 Apr 2020 14:12:02 -0700 User-Agent: Thunderbird MIME-Version: 1.0 In-Reply-To: <544E27A6-D799-4AF3-B4B7-1E68D5D50698@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="OL3HcfXl96oGJZBGGYPqcYw9drokEEWjC" X-Rspamd-Queue-Id: 4986Lc39TSz484K X-Spamd-Bar: -------- X-Spamd-Result: default: False [-8.73 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[delphij.net:s=m7e2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; HAS_REPLYTO(0.00)[d@delphij.net]; TO_DN_SOME(0.00)[]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[delphij.net:+]; DMARC_POLICY_ALLOW(-0.50)[delphij.net,reject]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; IP_SCORE(-3.64)[ip: (-9.91), ipnet: 2001:470::/32(-4.65), asn: 6939(-3.60), country: US(-0.05)]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Apr 2020 21:12:09 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --OL3HcfXl96oGJZBGGYPqcYw9drokEEWjC Content-Type: multipart/mixed; boundary="bBbUDl9mUraOpX4O489N3sYhzSlAUBRFO"; protected-headers="v1" From: Xin Li Reply-To: d@delphij.net To: Kristof Provost , d@delphij.net Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Message-ID: <5246fd44-b856-5402-092e-9065d4d4f7ae@delphij.net> Subject: Re: CFT: if_bridge performance improvements References: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> <8e0e2bf1-27cd-1a99-b266-c7223255942f@delphij.net> <8634ec5c-a509-d2dd-8f5c-31efcbd50340@delphij.net> <544E27A6-D799-4AF3-B4B7-1E68D5D50698@FreeBSD.org> In-Reply-To: <544E27A6-D799-4AF3-B4B7-1E68D5D50698@FreeBSD.org> --bBbUDl9mUraOpX4O489N3sYhzSlAUBRFO Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 4/24/20 06:42, Kristof Provost wrote: > On 22 Apr 2020, at 18:15, Xin Li wrote: >> On 4/22/20 01:45, Kristof Provost wrote: >>> On 22 Apr 2020, at 10:20, Xin Li wrote: >>>> Hi, >>>> >>>> On 4/14/20 02:51, Kristof Provost wrote: >>>>> Hi, >>>>> >>>>> Thanks to support from The FreeBSD Foundation I=E2=80=99ve been abl= e to >>>>> work on >>>>> improving the throughput of if_bridge. >>>>> It changes the (data path) locking to use the NET_EPOCH >>>>> infrastructure. >>>>> Benchmarking shows substantial improvements (x5 in test setups). >>>>> >>>>> This work is ready for wider testing now. >>>>> >>>>> It=E2=80=99s under review here: https://reviews.freebsd.org/D24250 >>>>> >>>>> Patch for CURRENT: https://reviews.freebsd.org/D24250?download=3Dtr= ue >>>>> Patches for stable/12: >>>>> https://people.freebsd.org/~kp/if_bridge/stable_12/ >>>>> >>>>> I=E2=80=99m not currently aware of any panics or issues resulting f= rom these >>>>> patches. >>>> >>>> I have observed the following panic with latest stable/12 after >>>> applying >>>> the stable_12 patchset, it appears like a race condition related NUL= L >>>> pointer deference, but I haven't took a deeper look yet. >>>> >>>> The box have 7 igb(4) NICs, with several bridge and VLAN configured >>>> acting as a router.=C2=A0 Please let me know if you need additional >>>> information; I can try -CURRENT as well, but it would take some time= as >>>> the box is relatively slow (it's a ZFS based system so I can create = a >>>> separate boot environment for -CURRENT if needed, but that would tak= e >>>> some time as I might have to upgrade the packages, should there be a= ny >>>> ABI breakages). >>>> >>> Thanks for the report. I don=E2=80=99t immediately see how this could= happen. >>> >>> Are you running an L2 firewall on that bridge by any chance? An earli= er >>> version of the patch had issues with a stray unlock in that code path= =2E >> >> I don't think I have a L2 firewall (I assume means filtering based on >> MAC address like what can be done with e.g. ipfw?=C2=A0 The bridges we= re >> created on vlan interfaces though, do they count as L2 firewall?), the= >> system is using pf with a few NAT rules: >> >=20 > That backtrace looks identical to the one Peter reported, up to and > including the offset in the bridge_input() function. > Given that there=E2=80=99s no likely way to end up with a NULL mutex ei= ther I > have to assume that it=E2=80=99s a case of trying to unlock a locked mu= tex, and > the most likely reason is that you ran into the same problem Peter ran > into. >=20 > The current version of the patch should resolve it. Thanks, I'd like to report that after applying the patch from Peter the system seems to survive without problem. Cheers, --bBbUDl9mUraOpX4O489N3sYhzSlAUBRFO-- --OL3HcfXl96oGJZBGGYPqcYw9drokEEWjC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.2.20 (Darwin) iQIzBAEBCgAdFiEEceNg5NEMZIki80nQQHl/fJX0g08FAl6jViUACgkQQHl/fJX0 g08YSQ//ehsN8qEaDAOVOCamsRCb4l5MiW/0i9gulOLsRbEN6Rm4ZRWDa+5yCTyq p0bs1RlOhspYOZhVkwtw7gITCrJ29LdHM7VIxVzaHOskuh6zTGcnUG6PZMHKw62H fA54Yzt20pRpYYS+L/LrKsQUaaJRsZNzMnASchANunuEQcaAuGZJIBG+nZXkM1rQ TRKJv6F9sxAkE+CtWB73AB1isV7IREDrcLgi/C7CIrehqD/nvBdpF35+JflFIe0f s2NHZZNyjoFm6Fxbs7tWfbit+hLDY5pVMHyB/ItsIyFToB0gvWWuNAGasLP4Pimn b5yKyGKb1KXPzIrtkvcLFhRHwqPBqiGeOrm/657MHJ8IxvaXzs6/UGuy2uqZU2iP NfPWVkvU6oP3TZSVkedysyk5/UCWhslCfdAnUP/tT8QqKI5lpbmDsvnCBXN0rNbi uzKnlbMmatqNgLLgw8D9h/MvXj9ZdzUKm2Y1wYF/Jzzag0BBTR3IsOV4BxKmvpX2 e5GwHaa6y/fwHdjIO4K9KWg7y1b6XUFTgp1sxm/0c5GnrmSj29wHUdDntse8yAqm PdWpyU/vwfV2Uc1FIJBpWYnKiWZPx4GgULSKLwliZPCaKeoTqStmE00rfLmtnPPd o2n7dv+oH4ybM2qOkVt55N1TGpGT3HlIne5SWqsNG7aaMWk46p4= =Kjcw -----END PGP SIGNATURE----- --OL3HcfXl96oGJZBGGYPqcYw9drokEEWjC--