Date: Wed, 22 Apr 2020 01:20:57 -0700 From: Xin Li <delphij@delphij.net> To: Kristof Provost <kp@FreeBSD.org>, freebsd-current@freebsd.org Cc: freebsd-stable@freebsd.org Subject: Re: CFT: if_bridge performance improvements Message-ID: <8e0e2bf1-27cd-1a99-b266-c7223255942f@delphij.net> In-Reply-To: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> References: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Eow1ZhByevXoLHJPQxVHdCGIJnld0vFJV Content-Type: multipart/mixed; boundary="o6v1mRSYFjcNeybUBiH78mO6sKJUeyjOa"; protected-headers="v1" From: Xin Li <delphij@delphij.net> Reply-To: d@delphij.net To: Kristof Provost <kp@FreeBSD.org>, freebsd-current@freebsd.org Cc: freebsd-stable@freebsd.org Message-ID: <8e0e2bf1-27cd-1a99-b266-c7223255942f@delphij.net> Subject: Re: CFT: if_bridge performance improvements References: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> In-Reply-To: <5377E42E-4C01-4BCC-B934-011AC3448B54@FreeBSD.org> --o6v1mRSYFjcNeybUBiH78mO6sKJUeyjOa Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hi, On 4/14/20 02:51, Kristof Provost wrote: > Hi, >=20 > Thanks to support from The FreeBSD Foundation I=E2=80=99ve been able to= work on > improving the throughput of if_bridge. > It changes the (data path) locking to use the NET_EPOCH infrastructure.= > Benchmarking shows substantial improvements (x5 in test setups). >=20 > This work is ready for wider testing now. >=20 > It=E2=80=99s under review here: https://reviews.freebsd.org/D24250 >=20 > Patch for CURRENT: https://reviews.freebsd.org/D24250?download=3Dtrue > Patches for stable/12: https://people.freebsd.org/~kp/if_bridge/stable_= 12/ >=20 > I=E2=80=99m not currently aware of any panics or issues resulting from = these > patches. I have observed the following panic with latest stable/12 after applying the stable_12 patchset, it appears like a race condition related NULL pointer deference, but I haven't took a deeper look yet. The box have 7 igb(4) NICs, with several bridge and VLAN configured acting as a router. Please let me know if you need additional information; I can try -CURRENT as well, but it would take some time as the box is relatively slow (it's a ZFS based system so I can create a separate boot environment for -CURRENT if needed, but that would take some time as I might have to upgrade the packages, should there be any ABI breakages). =3D=3D=3D Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x20 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80c286d5 stack pointer =3D 0x28:0xffffffff824cb840 frame pointer =3D 0x28:0xffffffff824cb850 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 0 (if_io_tqg_0) trap number =3D 12 panic: page fault cpuid =3D 0 time =3D 1587541913 KDB: stack backtrace: #0 0xffffffff80c117a5 at kdb_backtrace+0x65 #1 0xffffffff80bc588e at vpanic+0x17e #2 0xffffffff80bc5703 at panic+0x43 #3 0xffffffff810d2310 at trap_pfault+0 #4 0xffffffff810d235f at trap_pfault+0x4f #5 0xffffffff810d19b8 at trap+0x288 #6 0xffffffff810aae1c at calltrap+0x8 #7 0xffffffff80ba5c96 at __mtx_unlock_sleep+0xb6 #8 0xffffffff8248f4c7 at bridge_input+0x877 #9 0xffffffff80cd5c47 at ether_nh_input+0x207 #10 0xffffffff80cf1e4a at netisr_dispatch_src+0xca #11 0xffffffff80cd4f0b at ether_input+0x4b #12 0xffffffff80cdf1a3 at vlan_input+0x1f3 #13 0xffffffff80cd4ae1 at ether_demux+0x121 #14 0xffffffff80cd5d7b at ether_nh_input+0x33b #15 0xffffffff80cf1e4a at netisr_dispatch_src+0xca #16 0xffffffff80cd4f0b at ether_input+0x4b #17 0xffffffff80cee41c at iflib_rxeof+0xadc Uptime: 6m6s Dumping 848 out of 16313 MB:..2%..12%..21%..31%..42%..51%..61%..72%..82%..91% Backtrace: (kgdb) #0 doadump () at src/sys/amd64/include/pcpu_aux.h:55 #1 0xffffffff80bc54a5 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:451 #2 0xffffffff80bc58e6 in vpanic (fmt=3D<value optimized out>, ap=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:880 #3 0xffffffff80bc5703 in panic (fmt=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:807 #4 0xffffffff810d2310 in trap_fatal (frame=3D<value optimized out>, eva=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:925 #5 0xffffffff810d235f in trap_pfault (frame=3D0xffffffff824cb780, usermode=3D<value optimized out>, signo=3D<value optimized out>, ucode=3D<value optimized out>) at src/sys/amd64/include/pcpu_aux.h:55= #6 0xffffffff810d19b8 in trap (frame=3D0xffffffff824cb780) at /usr/src/sys/amd64/amd64/trap.c:407 #7 0xffffffff810aae1c in calltrap () at /usr/src/sys/amd64/amd64/exception.S:289 #8 0xffffffff80c286d5 in turnstile_broadcast (ts=3D0x0, queue=3D0) at /usr/src/sys/kern/subr_turnstile.c:880 #9 0xffffffff80ba5c96 in __mtx_unlock_sleep (c=3D0xfffff80013351430, v=3D= 0) at /usr/src/sys/kern/kern_mutex.c:1041 #10 0xffffffff8248f4c7 in bridge_input (ifp=3D<value optimized out>, m=3D<value optimized out>) at src/sys/amd64/include/atomic.h:221 #11 0xffffffff80cd5c47 in ether_nh_input (m=3D<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:631 #12 0xffffffff80cf1e4a in netisr_dispatch_src (proto=3D5, source=3D<value optimized out>, m=3D<value optimized out>) at /usr/src/sys/net/netisr.c:1124 #13 0xffffffff80cd4f0b in ether_input (ifp=3D0xfffff800060dc000, m=3D0x0)= at /usr/src/sys/net/if_ethersubr.c:787 #14 0xffffffff80cdf1a3 in vlan_input (ifp=3D0xfffff800036d6800, m=3D0xfffff8001d65fc00) at /usr/src/sys/net/if_vlan.c:1291 #15 0xffffffff80cd4ae1 in ether_demux (ifp=3D0xfffff800036d6800, m=3D<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:832 #16 0xffffffff80cd5d7b in ether_nh_input (m=3D<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:667 #17 0xffffffff80cf1e4a in netisr_dispatch_src (proto=3D5, source=3D<value optimized out>, m=3D<value optimized out>) at /usr/src/sys/net/netisr.c:1124 #18 0xffffffff80cd4f0b in ether_input (ifp=3D0xfffff800036d6800, m=3D0xfffff80013939c00) at /usr/src/sys/net/if_ethersubr.c:787 #19 0xffffffff80cee41c in iflib_rxeof (rxq=3D<value optimized out>, budget=3D<value optimized out>) at /usr/src/sys/net/iflib.c:2873 #20 0xffffffff80ce87b3 in _task_fn_rx (context=3D0xfffff800036d6000) at /usr/src/sys/net/iflib.c:3801 #21 0xffffffff80c100b1 in gtaskqueue_run_locked (queue=3D0xfffff8000306b9= 00) at /usr/src/sys/kern/subr_gtaskqueue.c:363 #22 0xffffffff80c0fd53 in gtaskqueue_thread_loop (arg=3D<value optimized = out>) at /usr/src/sys/kern/subr_gtaskqueue.c:538 #23 0xffffffff80b86b0e in fork_exit ( callout=3D0xffffffff80c0fc80 <gtaskqueue_thread_loop>, arg=3D0xfffffe00003f4008, frame=3D0xffffffff824cbd40) at /usr/src/sys/kern/kern_fork.c:1079 #24 0xffffffff810abe6e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:1079 #25 0x0000000000000000 in ?? () Current language: auto; currently minimal --o6v1mRSYFjcNeybUBiH78mO6sKJUeyjOa-- --Eow1ZhByevXoLHJPQxVHdCGIJnld0vFJV Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.2.20 (Darwin) iQIzBAEBCgAdFiEEceNg5NEMZIki80nQQHl/fJX0g08FAl6f/m0ACgkQQHl/fJX0 g09bew//RISJvy1mI5phGpP0ydRB8CbMwdlhb1LL9q12bgB4vBYFXOwtJCqir+ne H2/THBx8kqKZxPb//DpvHv8Wty2Hb0dRVbPqJ2seWX4dXtNGLLvRqpt8+iBxmn2B SyudiF36ifNl9OJR6J4nhYB755xhee3Lu2raBuXx2/voWYbSKQm5tNy5PqvnxbML dhUcxlJnMdO62g8v55lE4a2e7/Fg/XNI7Mg0bLZldggVUJIhwcuhV/kX2NDncI9+ l+TU/8MNOqR4f3sg/dITEE3wMklrl7nXFW4QOJ30jrCJlxrwH8zZ2gRPxOdWji9x 9DoA+M5LbuJT7yxGt0AluITZrdsQ9mzb0GlhsPE7k+CNjCFtmotudy0jcv/dEDrb 2YkYXZhTHooBvsdj0/AqiU0eRKZ6/NI6kHDknOeDLcuTQ98P+JeYYTBgO9sLr8fP hCqOPqmqgF2C01Xr7Wc1AnMlKop99qWrxlS7sZ0zJICI88keoXPNuSOzsTIsTws9 UmaDXvADHEltlqEQXo0pGCbFl4VrXHqBLmNg7PI70RzhpN+QHbKPwCKboplhhkWD DZXCMZ6VTkas5+UGA0tY/7CcNnilHaUcjKiXW2wvVquiSqmDTRK324jpNa+Ytr8h wMg77R7bZr+W9nEtUJfs78h/XkafRc/j/8SPkAELjZ5QpCpp91U= =yySD -----END PGP SIGNATURE----- --Eow1ZhByevXoLHJPQxVHdCGIJnld0vFJV--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8e0e2bf1-27cd-1a99-b266-c7223255942f>