From owner-freebsd-current@freebsd.org Wed Apr 22 08:21:10 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6ABB72AAA8A; Wed, 22 Apr 2020 08:21:10 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "anubis.delphij.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 496YKx295Kz3PVc; Wed, 22 Apr 2020 08:21:08 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from odin.corp.delphij.net (unknown [IPv6:2601:646:8600:58ba:8449:9abe:e049:bc98]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 4F912C309; Wed, 22 Apr 2020 01:21:02 -0700 (PDT) Reply-To: d@delphij.net To: Kristof Provost , freebsd-current@freebsd.org Cc: freebsd-stable@freebsd.org References: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> From: Xin Li Autocrypt: addr=delphij@delphij.net; prefer-encrypt=mutual; keydata= mQINBFuSR4oBEACvvEgwRIHs6IcSP/yaDtySF78Ji3rP29qdiQsxhMsOtvtffdbS56VApIWO UFb3/iN2gA8HwLvrmjijN0HEoLVX7na1WARmxRYzQMtApsZIUTtx7hnUYlsi2F5odZa6CDW9 a954DLRzYxiUwYDcu5Zjl9bglK1H8e/N9uC0Vuigr4teWfh86brzOyf819QzwFVYfMIK4ihw QGwMvTzbyVuCFy+LENkmcVYni70oQy6rZ5ktSuYbuOFvu7inRRfhSWPHziV7k+bW88sJ7xhv lBlegcnhkSudWX2M8tZ3MO1PJOcyys0CJlsBY5Weiog2lIPi05h/E9pZ9mc1Vud17iqDaL6w RaggOUhuPfDGCdO5ro82W4BZGeQMRnRF5Ntk+t2ShIH4nn3xRLV0E5nziCiKlgiMqOrz/ZTL QTVbHrCuiwD+fSK14y0oHbkOLYTYLlgh1JbwfY2Ty7elOYiWzyeJ7sJh2dF91NSEneWIOys3 mBpuvtU3nSzzTvAB48VV+Nbg1CpIOgNlPjj7uhIum/Z/VjUaJEyaLpTIRh0MVJVcbP7hXSqZ NA35EEZZVnWEOYdycm4CmEdeNPWkrAf2Ya77iR5VLGypwMlsUMQPh+sKVWDD38M8stFGBBNm d01Hi74Bsq5hKan654dOqMt5eYklrVj0ucMzFQtus7oE502UswARAQABtBxYaW4gTEkgPGRl bHBoaWpAZGVscGhpai5uZXQ+iQJUBBMBCgA+FiEEceNg5NEMZIki80nQQHl/fJX0g08FAluS R/YCGwMFCQmuhAAFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQQHl/fJX0g0+2Og//bWpE F2V5/M5l6YW1T8oLcT9rIOH6oq9M0LMNRgFeiNNnilGIeeIgtOGBRueG4CZiZAvsRPJkrO70 1R2SrdkCIvwGUzUAxx1NfBWb+vgm4fgkW/MotGonceM5v0qfSKKXasWvDctkK28aG+IoQzmi FjXNW4+ju4zeQFYwD4ZDWqw9MqO0hVb24uW3dxtQhbfmOLgJ/PEDMQaFuANbW1c+iR0BQA3D Go/EeMY4kpN8on6Aqt/S/4JVltudfQ9OXdjQsC7netSaB9K3mHGt9aKAAB7RzlRY00DKkYS/ /eQwLzGPmK7yX13M68mMDjBs6mIR8t/E1S5OdBNhHRPNPlEbwugR4KaiCsN5yqzJoSV99fKY z2VyxjWPaG8yhHE+jmKUgIBKTfFUQEfkriQR4EASoeJ+soaMTiFDBij1Zw5n3ndLRFMB1ZCl fZLER36mAgW4m4kP83TWnDiJLxOxSOxifV8HpTFjff902H85cybg9KMwrfPDr6W19GGk5Vo1 fkza5krRMGbKWb7+74Evusi0ZxJLIOFwp5Y8eVqUMZaAD3f1ZX1M3pgXOp20QgAy+2KvMHij rLa4q+tMGRzYYD1BnFVSVdXAX5VOoTmHBcDz67DkuRwk2Byp1sgd407oEOmSwrNJlKS0TPCm xUJ2fdSQF+1/MMSRfee49vtMvz7cOrC5Ag0EW5JHigEQANiBmIFAfRNH3nzYNWC0yC+tfx3z sUwAsH1VaBM/cTib+yKtbBOSIlXWjJZWX3MHwoI/1LeGghB2mxkkX1L0pJ/vj1eXNR+sFZ32 0pYcl61Fxg/5fioG4QDTM4i3i7NR5PxDnc6UVaynSlII93DedRhZ1ROtdn4vyMgzsDiqhbL7 BthDOt5KxjqdRk4qRPSw7BovEqZLOcG5IJtf/zZUzRbM7SBljEbOAfekDGx1Br+RrYSD7/Ef Pwwzou9T8315IpBpIHyQF/dZNk3iFiB9Ed5CA71ZRYV5YoLWE9lL0j9kxOLQ5vHnX3mVq7QZ Bc7nzwZ6UhQgYmrG5+RWvuiPpGwvDRIsugJUGXucYkAQh5kuNblmkwpv6u9rNMjCNbzAylOa qdogra5EW+RUSbRz0b4iIr8nnZeAlh7BihCe7JjOwbDjoBEEEtSfVc4hD/LENqpcYVrChphf aOLB9YIXhnVDTVvMc9OklWT/81HzAaDQqOQCzEfY92199Ct9/CwRoQ2OpO8TO5+8A7b9Nb33 nmxMn09mb48ruRacMrfHxCWbgU4w9SEfbip4GcS5wGG6yTC+hw55Iwnnwus40NrJ0GEr8a4r cdsLbkvlyoNHB8ZGgyJ4aFCQ1V4qE1BnlTk7Z8BYBUkJM1odPSkVvHpCnMUjVpJ3hEOC+73Z YH1dh7lZABEBAAGJAjwEGAEKACYWIQRx42Dk0QxkiSLzSdBAeX98lfSDTwUCW5JHigIbDAUJ Ca6EAAAKCRBAeX98lfSDTz8DEACMh3poeUb+gWNF4RWFZuLteZVo0+E1JLYXQkmtrRBLXviP +Qy0pXyFAVxLM4hNIBoIDYfK9BcwrBYf7AwSKrH0GiNwFpgHCkbZd6qoZy2gB+adTnCpVCTJ KJetsH/8awkrChJWMK0ckGf3EeWMPvawG7kW7FBz70NYEZ0pOMiaEZNVtzD3wwbYWUiDFYth 83XGglOExg+1ShTW5XjQPRrdyJAO+aUW4o3lVjfyUJXMgI4rmhMiLVm06GuNrbpKIF0s+4Vd jQAjhrDQjfoXi9CkfsA/cONseuHNv1JGj3RqHiqHJq1dbrpodXp925zGDAnUGxCOBPoFopAH gVzR89GTut059GpwqsddZmU6y7rqifuam/ekJ+QRwc16vgt7pHqCrTY8WPxRZr2UpFU1wlTo COdeiFep1gq1F9jzFjJnoMaAdmC6k7bgAA+RQusOgIhJL0jIej7DoAHxmxFFCfRy+lDtpXwF gQ8HMvzHI65QWmQnMo7s6SQH/ZH5s1yR6SJq8+3lDz+dCuT42qJVqIPVvxd10LW0FNN+t7HF eLadU6ekSgD13/EYMYXlvNHkw7dAItSDxIzgRyykLz0bCU9xwNWoS4Z43+ifF9anJ+uR0ltW El1j++h6ZrD3LLuCgJIt1so0m49GzdcSpOI7LCwMlacyvafiEyjUn+tSNDsnfw== Subject: Re: CFT: if_bridge performance improvements Message-ID: <8e0e2bf1-27cd-1a99-b266-c7223255942f@delphij.net> Date: Wed, 22 Apr 2020 01:20:57 -0700 User-Agent: Thunderbird MIME-Version: 1.0 In-Reply-To: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Eow1ZhByevXoLHJPQxVHdCGIJnld0vFJV" X-Rspamd-Queue-Id: 496YKx295Kz3PVc X-Spamd-Bar: -------- X-Spamd-Result: default: False [-8.73 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[delphij.net:s=m7e2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; HAS_REPLYTO(0.00)[d@delphij.net]; TO_DN_SOME(0.00)[]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; DKIM_TRACE(0.00)[delphij.net:+]; DMARC_POLICY_ALLOW(-0.50)[delphij.net,reject]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; IP_SCORE(-3.64)[ip: (-9.91), ipnet: 2001:470::/32(-4.65), asn: 6939(-3.60), country: US(-0.05)]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Apr 2020 08:21:10 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Eow1ZhByevXoLHJPQxVHdCGIJnld0vFJV Content-Type: multipart/mixed; boundary="o6v1mRSYFjcNeybUBiH78mO6sKJUeyjOa"; protected-headers="v1" From: Xin Li Reply-To: d@delphij.net To: Kristof Provost , freebsd-current@freebsd.org Cc: freebsd-stable@freebsd.org Message-ID: <8e0e2bf1-27cd-1a99-b266-c7223255942f@delphij.net> Subject: Re: CFT: if_bridge performance improvements References: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> In-Reply-To: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> --o6v1mRSYFjcNeybUBiH78mO6sKJUeyjOa Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hi, On 4/14/20 02:51, Kristof Provost wrote: > Hi, >=20 > Thanks to support from The FreeBSD Foundation I=E2=80=99ve been able to= work on > improving the throughput of if_bridge. > It changes the (data path) locking to use the NET_EPOCH infrastructure.= > Benchmarking shows substantial improvements (x5 in test setups). >=20 > This work is ready for wider testing now. >=20 > It=E2=80=99s under review here: https://reviews.freebsd.org/D24250 >=20 > Patch for CURRENT: https://reviews.freebsd.org/D24250?download=3Dtrue > Patches for stable/12: https://people.freebsd.org/~kp/if_bridge/stable_= 12/ >=20 > I=E2=80=99m not currently aware of any panics or issues resulting from = these > patches. I have observed the following panic with latest stable/12 after applying the stable_12 patchset, it appears like a race condition related NULL pointer deference, but I haven't took a deeper look yet. The box have 7 igb(4) NICs, with several bridge and VLAN configured acting as a router. Please let me know if you need additional information; I can try -CURRENT as well, but it would take some time as the box is relatively slow (it's a ZFS based system so I can create a separate boot environment for -CURRENT if needed, but that would take some time as I might have to upgrade the packages, should there be any ABI breakages). =3D=3D=3D Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x20 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80c286d5 stack pointer =3D 0x28:0xffffffff824cb840 frame pointer =3D 0x28:0xffffffff824cb850 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 0 (if_io_tqg_0) trap number =3D 12 panic: page fault cpuid =3D 0 time =3D 1587541913 KDB: stack backtrace: #0 0xffffffff80c117a5 at kdb_backtrace+0x65 #1 0xffffffff80bc588e at vpanic+0x17e #2 0xffffffff80bc5703 at panic+0x43 #3 0xffffffff810d2310 at trap_pfault+0 #4 0xffffffff810d235f at trap_pfault+0x4f #5 0xffffffff810d19b8 at trap+0x288 #6 0xffffffff810aae1c at calltrap+0x8 #7 0xffffffff80ba5c96 at __mtx_unlock_sleep+0xb6 #8 0xffffffff8248f4c7 at bridge_input+0x877 #9 0xffffffff80cd5c47 at ether_nh_input+0x207 #10 0xffffffff80cf1e4a at netisr_dispatch_src+0xca #11 0xffffffff80cd4f0b at ether_input+0x4b #12 0xffffffff80cdf1a3 at vlan_input+0x1f3 #13 0xffffffff80cd4ae1 at ether_demux+0x121 #14 0xffffffff80cd5d7b at ether_nh_input+0x33b #15 0xffffffff80cf1e4a at netisr_dispatch_src+0xca #16 0xffffffff80cd4f0b at ether_input+0x4b #17 0xffffffff80cee41c at iflib_rxeof+0xadc Uptime: 6m6s Dumping 848 out of 16313 MB:..2%..12%..21%..31%..42%..51%..61%..72%..82%..91% Backtrace: (kgdb) #0 doadump () at src/sys/amd64/include/pcpu_aux.h:55 #1 0xffffffff80bc54a5 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:451 #2 0xffffffff80bc58e6 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:880 #3 0xffffffff80bc5703 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:807 #4 0xffffffff810d2310 in trap_fatal (frame=3D, eva=3D) at /usr/src/sys/amd64/amd64/trap.c:925 #5 0xffffffff810d235f in trap_pfault (frame=3D0xffffffff824cb780, usermode=3D, signo=3D, ucode=3D) at src/sys/amd64/include/pcpu_aux.h:55= #6 0xffffffff810d19b8 in trap (frame=3D0xffffffff824cb780) at /usr/src/sys/amd64/amd64/trap.c:407 #7 0xffffffff810aae1c in calltrap () at /usr/src/sys/amd64/amd64/exception.S:289 #8 0xffffffff80c286d5 in turnstile_broadcast (ts=3D0x0, queue=3D0) at /usr/src/sys/kern/subr_turnstile.c:880 #9 0xffffffff80ba5c96 in __mtx_unlock_sleep (c=3D0xfffff80013351430, v=3D= 0) at /usr/src/sys/kern/kern_mutex.c:1041 #10 0xffffffff8248f4c7 in bridge_input (ifp=3D, m=3D) at src/sys/amd64/include/atomic.h:221 #11 0xffffffff80cd5c47 in ether_nh_input (m=3D) at /usr/src/sys/net/if_ethersubr.c:631 #12 0xffffffff80cf1e4a in netisr_dispatch_src (proto=3D5, source=3D, m=3D) at /usr/src/sys/net/netisr.c:1124 #13 0xffffffff80cd4f0b in ether_input (ifp=3D0xfffff800060dc000, m=3D0x0)= at /usr/src/sys/net/if_ethersubr.c:787 #14 0xffffffff80cdf1a3 in vlan_input (ifp=3D0xfffff800036d6800, m=3D0xfffff8001d65fc00) at /usr/src/sys/net/if_vlan.c:1291 #15 0xffffffff80cd4ae1 in ether_demux (ifp=3D0xfffff800036d6800, m=3D) at /usr/src/sys/net/if_ethersubr.c:832 #16 0xffffffff80cd5d7b in ether_nh_input (m=3D) at /usr/src/sys/net/if_ethersubr.c:667 #17 0xffffffff80cf1e4a in netisr_dispatch_src (proto=3D5, source=3D, m=3D) at /usr/src/sys/net/netisr.c:1124 #18 0xffffffff80cd4f0b in ether_input (ifp=3D0xfffff800036d6800, m=3D0xfffff80013939c00) at /usr/src/sys/net/if_ethersubr.c:787 #19 0xffffffff80cee41c in iflib_rxeof (rxq=3D, budget=3D) at /usr/src/sys/net/iflib.c:2873 #20 0xffffffff80ce87b3 in _task_fn_rx (context=3D0xfffff800036d6000) at /usr/src/sys/net/iflib.c:3801 #21 0xffffffff80c100b1 in gtaskqueue_run_locked (queue=3D0xfffff8000306b9= 00) at /usr/src/sys/kern/subr_gtaskqueue.c:363 #22 0xffffffff80c0fd53 in gtaskqueue_thread_loop (arg=3D) at /usr/src/sys/kern/subr_gtaskqueue.c:538 #23 0xffffffff80b86b0e in fork_exit ( callout=3D0xffffffff80c0fc80 , arg=3D0xfffffe00003f4008, frame=3D0xffffffff824cbd40) at /usr/src/sys/kern/kern_fork.c:1079 #24 0xffffffff810abe6e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:1079 #25 0x0000000000000000 in ?? () Current language: auto; currently minimal --o6v1mRSYFjcNeybUBiH78mO6sKJUeyjOa-- --Eow1ZhByevXoLHJPQxVHdCGIJnld0vFJV Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.2.20 (Darwin) iQIzBAEBCgAdFiEEceNg5NEMZIki80nQQHl/fJX0g08FAl6f/m0ACgkQQHl/fJX0 g09bew//RISJvy1mI5phGpP0ydRB8CbMwdlhb1LL9q12bgB4vBYFXOwtJCqir+ne H2/THBx8kqKZxPb//DpvHv8Wty2Hb0dRVbPqJ2seWX4dXtNGLLvRqpt8+iBxmn2B SyudiF36ifNl9OJR6J4nhYB755xhee3Lu2raBuXx2/voWYbSKQm5tNy5PqvnxbML dhUcxlJnMdO62g8v55lE4a2e7/Fg/XNI7Mg0bLZldggVUJIhwcuhV/kX2NDncI9+ l+TU/8MNOqR4f3sg/dITEE3wMklrl7nXFW4QOJ30jrCJlxrwH8zZ2gRPxOdWji9x 9DoA+M5LbuJT7yxGt0AluITZrdsQ9mzb0GlhsPE7k+CNjCFtmotudy0jcv/dEDrb 2YkYXZhTHooBvsdj0/AqiU0eRKZ6/NI6kHDknOeDLcuTQ98P+JeYYTBgO9sLr8fP hCqOPqmqgF2C01Xr7Wc1AnMlKop99qWrxlS7sZ0zJICI88keoXPNuSOzsTIsTws9 UmaDXvADHEltlqEQXo0pGCbFl4VrXHqBLmNg7PI70RzhpN+QHbKPwCKboplhhkWD DZXCMZ6VTkas5+UGA0tY/7CcNnilHaUcjKiXW2wvVquiSqmDTRK324jpNa+Ytr8h wMg77R7bZr+W9nEtUJfs78h/XkafRc/j/8SPkAELjZ5QpCpp91U= =yySD -----END PGP SIGNATURE----- --Eow1ZhByevXoLHJPQxVHdCGIJnld0vFJV--