From owner-freebsd-stable@freebsd.org Wed Apr 22 17:16:55 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 43F172BAFA4; Wed, 22 Apr 2020 17:16:55 +0000 (UTC) (envelope-from peter.blok@bsd4all.org) Received: from smtpq1.tb.mail.iss.as9143.net (smtpq1.tb.mail.iss.as9143.net [212.54.42.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 496nD56qJCz4Wnc; Wed, 22 Apr 2020 17:16:53 +0000 (UTC) (envelope-from peter.blok@bsd4all.org) Received: from [212.54.42.135] (helo=smtp11.tb.mail.iss.as9143.net) by smtpq1.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jRIzS-0005QD-SG; Wed, 22 Apr 2020 19:16:50 +0200 Received: from 94-209-85-88.cable.dynamic.v4.ziggo.nl ([94.209.85.88] helo=wan0.bsd4all.org) by smtp11.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1jRIzS-0002EC-Mb; Wed, 22 Apr 2020 19:16:50 +0200 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id C9627223; Wed, 22 Apr 2020 19:16:49 +0200 (CEST) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sOsf4-Qd-z4U; Wed, 22 Apr 2020 19:16:48 +0200 (CEST) Received: from [192.168.1.65] (unknown [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 87AD94A; Wed, 22 Apr 2020 19:16:48 +0200 (CEST) From: peter.blok@bsd4all.org Message-Id: Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.14\)) Subject: Re: CFT: if_bridge performance improvements Date: Wed, 22 Apr 2020 19:16:48 +0200 In-Reply-To: <8634ec5c-a509-d2dd-8f5c-31efcbd50340@delphij.net> Cc: Kristof Provost , FreeBSD Stable To: d@delphij.net, freebsd-current@freebsd.org References: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> <8e0e2bf1-27cd-1a99-b266-c7223255942f@delphij.net> <8634ec5c-a509-d2dd-8f5c-31efcbd50340@delphij.net> X-Mailer: Apple Mail (2.3445.104.14) X-SourceIP: 94.209.85.88 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.3 cv=du1A92o4 c=1 sm=1 tr=0 a=LYXyOGYQqFYBMgK+Y6iqTg==:17 a=cl8xLZFz6L8A:10 a=SWg00rOMAAAA:8 a=6I5d2MoRAAAA:8 a=7F27aPtjtkppQGVRRocA:9 a=QEXdDO2ut3YA:10 a=mdr11xoXeMGbgq-aR_MA:9 a=Jz0u6kiykQrJQnsa:21 a=_W_S_7VecoQA:10 a=nWvTgx2JuP7DHgfbJPXu:22 a=IjZwj45LgO3ly-622nXo:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 496nD56qJCz4Wnc X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of peter.blok@bsd4all.org designates 212.54.42.164 as permitted sender) smtp.mailfrom=peter.blok@bsd4all.org X-Spamd-Result: default: False [-3.36 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[bsd4all.org]; URI_COUNT_ODD(1.00)[9]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NO_DN(0.00)[]; IP_SCORE(-2.46)[ip: (-5.90), ipnet: 212.54.32.0/20(-4.04), asn: 33915(-2.41), country: NL(0.03)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[164.42.54.212.list.dnswl.org : 127.0.5.1]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[88.85.209.94.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Apr 2020 17:16:55 -0000 Just using pf is enough to provoke this panic. I had the same back = trace. This patch from Kristof fixed it for me. diff --git a/sys/net/if_bridge.c b/sys/net/if_bridge.c index 373fa096d70..83c453090bb 100644 --- a/sys/net/if_bridge.c +++ b/sys/net/if_bridge.c @@ -2529,7 +2529,6 @@ bridge_input(struct ifnet *ifp, struct mbuf *m) OR_PFIL_HOOKED_INET6)) { = \ if (bridge_pfil(&m, NULL, ifp, = \ PFIL_IN) !=3D 0 || m =3D=3D NULL) { = \ - BRIDGE_UNLOCK(sc); = \ return (NULL); = \ } = \ eh =3D mtod(m, struct ether_header *); = \ > On 22 Apr 2020, at 18:15, Xin Li wrote: >=20 > On 4/22/20 01:45, Kristof Provost wrote: >> On 22 Apr 2020, at 10:20, Xin Li wrote: >>> Hi, >>>=20 >>> On 4/14/20 02:51, Kristof Provost wrote: >>>> Hi, >>>>=20 >>>> Thanks to support from The FreeBSD Foundation I=E2=80=99ve been = able to work on >>>> improving the throughput of if_bridge. >>>> It changes the (data path) locking to use the NET_EPOCH = infrastructure. >>>> Benchmarking shows substantial improvements (x5 in test setups). >>>>=20 >>>> This work is ready for wider testing now. >>>>=20 >>>> It=E2=80=99s under review here: https://reviews.freebsd.org/D24250 >>>>=20 >>>> Patch for CURRENT: https://reviews.freebsd.org/D24250?download=3Dtrue= >>>> Patches for stable/12: >>>> https://people.freebsd.org/~kp/if_bridge/stable_12/ >>>>=20 >>>> I=E2=80=99m not currently aware of any panics or issues resulting = from these >>>> patches. >>>=20 >>> I have observed the following panic with latest stable/12 after = applying >>> the stable_12 patchset, it appears like a race condition related = NULL >>> pointer deference, but I haven't took a deeper look yet. >>>=20 >>> The box have 7 igb(4) NICs, with several bridge and VLAN configured >>> acting as a router. Please let me know if you need additional >>> information; I can try -CURRENT as well, but it would take some time = as >>> the box is relatively slow (it's a ZFS based system so I can create = a >>> separate boot environment for -CURRENT if needed, but that would = take >>> some time as I might have to upgrade the packages, should there be = any >>> ABI breakages). >>>=20 >> Thanks for the report. I don=E2=80=99t immediately see how this could = happen. >>=20 >> Are you running an L2 firewall on that bridge by any chance? An = earlier >> version of the patch had issues with a stray unlock in that code = path. >=20 > I don't think I have a L2 firewall (I assume means filtering based on > MAC address like what can be done with e.g. ipfw? The bridges were > created on vlan interfaces though, do they count as L2 firewall?), the > system is using pf with a few NAT rules: >=20 > $ sudo pfctl -s rules > anchor "miniupnpd" all > pass in quick inet6 proto tcp from to any flags S/SA keep state > block drop in quick inet6 proto tcp from ! to flags S/SA > block drop in quick proto tcp from any os "Linux" to any port =3D ssh > pass out on igb6 inet proto tcp from (igb6) to any port =3D domain = flags > S/SA keep state queue dns > pass out on igb6 inet proto udp from (igb6) to any port =3D domain = keep > state queue dns > pass in on igb6 proto tcp from any to (igb6) port =3D http flags S/SA > modulate state queue(web, ack) > pass in on igb6 proto tcp from any to (igb6) port =3D https flags S/SA > modulate state queue(web, ack) > pass out on igb6 inet proto tcp from (igb6) to any flags S/SA modulate > state queue bulk > block drop in quick on igb6 proto tcp from to any port =3D = ssh > label "ssh bruteforce" > block drop in on igb6 from to any >=20 > Cheers,