Date: Wed, 25 Oct 2023 17:27:25 +0200 From: Kristof Provost <kp@FreeBSD.org> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: fib6_lookup() returning deleted struct ifnet Message-ID: <A5BC4385-AA39-4608-B1BA-0551AFBB49CD@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
--=_MailMate_3433B8C0-4555-448B-877E-ABB0265B45A5_= Content-Type: text/plain; charset=UTF-8; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit Hi, Several pfSense users report IPv6-related panics when an interface is deleted. The relevant bug reports are https://redmine.pfsense.org/issues/14164 and https://redmine.pfsense.org/issues/14431. The latest report is for a build that includes commits up to 1a18383a52bc373e316d224cef1298debf6f7e25 (“libcrypto: link engines and the legacy provider to libcrypto”, September 15th). I believe all reports are for users running PPPoE, via netgraph, but that might be coincidental, as that’s the most likely way for interfaces to be destroyed (when PPP disconnects and reconnects). There are a few different backtraces, but they appear to have the same root cause, so I’ll focus on one of them: db:1:pfs> bt Tracing pid 2 tid 100041 td 0xfffffe0085264560 kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850ad910 vpanic() at vpanic+0x183/frame 0xfffffe00850ad960 panic() at panic+0x43/frame 0xfffffe00850ad9c0 trap_fatal() at trap_fatal+0x409/frame 0xfffffe00850ada20 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850ada80 calltrap() at calltrap+0x8/frame 0xfffffe00850ada80 --- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00850adb50, rbp = 0xfffffe00850adb80 --- in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850adb80 tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00850add70 tcp_timer_rexmt() at tcp_timer_rexmt+0x514/frame 0xfffffe00850addd0 tcp_timer_enter() at tcp_timer_enter+0x102/frame 0xfffffe00850ade10 softclock_call_cc() at softclock_call_cc+0x13c/frame 0xfffffe00850adec0 softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850adef0 fork_exit() at fork_exit+0x7d/frame 0xfffffe00850adf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850adf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- This happens in the TCP output path, where we look up the hop limit for a specific destination. I’ve obtained a core dump for such a crash, and I believe the panic happens on line https://cgit.freebsd.org/src/tree/sys/netinet6/in6_src.c#n861 The call in tcp_default_output() is `in6_selecthlim(int, NULL);`, so we don’t get an ifp from the caller, but instead perform a route lookup, and try to obtain the hop limit through `ND_IFINFO(nh->nh_ifp)`. This panics because the afdata[AF_INET6] pointer is NULL. The core dump shows a deleted structure ifnet: (kgdb) p *(struct ifnet *)0xfffff80203712800 $3 = { if_link = { cstqe_next = 0x0 }, if_clones = { le_next = 0x0, le_prev = 0x0 }, if_groups = { cstqh_first = 0x0, cstqh_last = 0xfffff80203712818 }, if_alloctype = 53 '5', if_numa_domain = 255 '\377', if_softc = 0xfffff80103447a00, if_llsoftc = 0x0, if_l2com = 0x0, if_dname = 0xffffffff81492f70 "ng", if_dunit = 0, if_index = 14, if_idxgen = 2, if_xname = "pppoe0\000\000\000\000\000\000\000\000\000", if_description = 0xfffff8003a5f83d0 "WAN", if_flags = 2132112, if_drv_flags = 0, if_capabilities = 0, if_capabilities2 = 0, … if_afdata = {0x0 <repeats 44 times>}, … if_output = 0xffffffff80e29c60 <ifdead_output>, if_input = 0xffffffff80e29c80 <ifdead_input>, if_bridge_input = 0x0, if_bridge_output = 0x0, if_bridge_linkstate = 0x0, if_start = 0xffffffff80e29c90 <ifdead_start>, if_ioctl = 0xffffffff80e29ca0 <ifdead_ioctl>, … My understanding is that the fib table should get updated whenever we change the routing table (such as during interface cleanup in `if_detach_internal()`). Some quick experimentation with epair and dtrace also shows: 20 20388 sync_algo_end_cb:entry Stage 1 kernel`setup_fd_instance+0x41f kernel`rebuild_fd_flm+0x99 kernel`rebuild_fd+0x136 kernel`rib_notify+0x50 kernel`rt_delete_conditional+0xf1 kernel`rib_del_route+0x1fc kernel`rib_handle_ifaddr_info+0xd9 kernel`nd6_prefix_offlink+0x1ce kernel`nd6_prefix_del+0x94 kernel`if_purgeaddrs+0x148 kernel`if_detach_internal+0x1e8 kernel`if_detach+0x71 if_epair.ko`epair_clone_destroy+0x62 kernel`if_clone_destroyif_flags+0x6a kernel`if_clone_destroy+0x100 kernel`ifioctl+0x8a5 kernel`kern_ioctl+0x286 kernel`sys_ioctl+0x152 kernel`amd64_syscall+0x153 kernel`0xffffffff8102315b In other words, when we delete the interface `if_detach_internal()` purges the interface addresses, which ends up rebuilding the fib (`rebuild_fd()`) via `rib_del_route()`. That ought to ensure that we cannot end up finding this struct ifnet through `fib6_lookup()`, as the purging of the addresses (and thus the rebuilding of the fib) is done before we `if_domdetach()` at the end of `if_detach_internal()`, and the NULL afdata[AF_INET6] demonstrates that we’ve gotten there. We’ve also gone through `if_free()`, as the ifindex_table no longer contains the struct ifnet pointer for the relevant interface. We appear to have not yet called `if_free_deferred()` (and indeed, ifp->if_refcount is 4, so we wouldn’t have called that yet). I’m confused as to how this can happen, and would appreciate hints. Thanks, Kristof --=_MailMate_3433B8C0-4555-448B-877E-ABB0265B45A5_= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE html> <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/xhtml; charset=3Dutf-8"= > </head> <body><div style=3D"font-family: sans-serif;"><div class=3D"markdown" sty= le=3D"white-space: normal;"> <p dir=3D"auto">Hi,</p> <p dir=3D"auto">Several pfSense users report IPv6-related panics when an = interface is deleted.<br> The relevant bug reports are <a href=3D"https://redmine.pfsense.org/issue= s/14164">https://redmine.pfsense.org/issues/14164</a>; and <a href=3D"http= s://redmine.pfsense.org/issues/14431">https://redmine.pfsense.org/issues/= 14431</a>.<br> The latest report is for a build that includes commits up to 1a18383a52bc= 373e316d224cef1298debf6f7e25 (=E2=80=9Clibcrypto: link engines and the le= gacy provider to libcrypto=E2=80=9D, September 15th).</p> <p dir=3D"auto">I believe all reports are for users running PPPoE, via ne= tgraph, but that might be coincidental, as that=E2=80=99s the most likely= way for interfaces to be destroyed (when PPP disconnects and reconnects)= =2E</p> <p dir=3D"auto">There are a few different backtraces, but they appear to = have the same root cause, so I=E2=80=99ll focus on one of them:</p> <pre style=3D"margin-left: 15px; margin-right: 15px; padding: 5px; border= : thin solid gray; overflow-x: auto; max-width: 90vw; background-color: #= E4E4E4;"><code style=3D"padding: 0 0.25em; background-color: #E4E4E4;">db= :1:pfs> bt Tracing pid 2 tid 100041 td 0xfffffe0085264560 kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850ad910 vpanic() at vpanic+0x183/frame 0xfffffe00850ad960 panic() at panic+0x43/frame 0xfffffe00850ad9c0 trap_fatal() at trap_fatal+0x409/frame 0xfffffe00850ada20 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850ada80 calltrap() at calltrap+0x8/frame 0xfffffe00850ada80 --- trap 0xc, rip =3D 0xffffffff80f5a036, rsp =3D 0xfffffe00850adb50, rbp= =3D 0xfffffe00850adb80 --- in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850adb80 tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00850add7= 0 tcp_timer_rexmt() at tcp_timer_rexmt+0x514/frame 0xfffffe00850addd0 tcp_timer_enter() at tcp_timer_enter+0x102/frame 0xfffffe00850ade10 softclock_call_cc() at softclock_call_cc+0x13c/frame 0xfffffe00850adec0 softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850adef0 fork_exit() at fork_exit+0x7d/frame 0xfffffe00850adf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850adf30 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- </code></pre> <p dir=3D"auto">This happens in the TCP output path, where we look up the= hop limit for a specific destination. I=E2=80=99ve obtained a core dump = for such a crash, and I believe the panic happens on line <a href=3D"http= s://cgit.freebsd.org/src/tree/sys/netinet6/in6_src.c#n861">https://cgit.f= reebsd.org/src/tree/sys/netinet6/in6_src.c#n861</a></p> <p dir=3D"auto">The call in tcp_default_output() is <code style=3D"paddin= g: 0 0.25em; background-color: #E4E4E4;">in6_selecthlim(int, NULL);</code= >, so we don=E2=80=99t get an ifp from the caller, but instead perform a = route lookup, and try to obtain the hop limit through <code style=3D"padd= ing: 0 0.25em; background-color: #E4E4E4;">ND_IFINFO(nh->nh_ifp)</code= >. This panics because the afdata[AF_INET6] pointer is NULL. The core dum= p shows a deleted structure ifnet:</p> <pre style=3D"margin-left: 15px; margin-right: 15px; padding: 5px; border= : thin solid gray; overflow-x: auto; max-width: 90vw; background-color: #= E4E4E4;"><code style=3D"padding: 0 0.25em; background-color: #E4E4E4;">(k= gdb) p *(struct ifnet *)0xfffff80203712800 $3 =3D { if_link =3D { cstqe_next =3D 0x0 }, if_clones =3D { le_next =3D 0x0, le_prev =3D 0x0 }, if_groups =3D { cstqh_first =3D 0x0, cstqh_last =3D 0xfffff80203712818 }, if_alloctype =3D 53 '5', if_numa_domain =3D 255 '\377', if_softc =3D 0xfffff80103447a00, if_llsoftc =3D 0x0, if_l2com =3D 0x0, if_dname =3D 0xffffffff81492f70 "ng", if_dunit =3D 0, if_index =3D 14, if_idxgen =3D 2, if_xname =3D "pppoe0\000\000\000\000\000\000\000\000\000", if_description =3D 0xfffff8003a5f83d0 "WAN", if_flags =3D 2132112, if_drv_flags =3D 0, if_capabilities =3D 0, if_capabilities2 =3D 0, =E2=80=A6 if_afdata =3D {0x0 <repeats 44 times>}, =E2=80=A6 if_output =3D 0xffffffff80e29c60 <ifdead_output>, if_input =3D 0xffffffff80e29c80 <ifdead_input>, if_bridge_input =3D 0x0, if_bridge_output =3D 0x0, if_bridge_linkstate =3D 0x0, if_start =3D 0xffffffff80e29c90 <ifdead_start>, if_ioctl =3D 0xffffffff80e29ca0 <ifdead_ioctl>, =E2=80=A6 </code></pre> <p dir=3D"auto">My understanding is that the fib table should get updated= whenever we change the routing table (such as during interface cleanup i= n <code style=3D"padding: 0 0.25em; background-color: #E4E4E4;">if_detach= _internal()</code>). Some quick experimentation with epair and dtrace als= o shows:</p> <pre style=3D"margin-left: 15px; margin-right: 15px; padding: 5px; border= : thin solid gray; overflow-x: auto; max-width: 90vw; background-color: #= E4E4E4;"><code style=3D"padding: 0 0.25em; background-color: #E4E4E4;"> 2= 0 20388 sync_algo_end_cb:entry Stage 1 kernel`setup_fd_instance+0x41f kernel`rebuild_fd_flm+0x99 kernel`rebuild_fd+0x136 kernel`rib_notify+0x50 kernel`rt_delete_conditional+0xf1 kernel`rib_del_route+0x1fc kernel`rib_handle_ifaddr_info+0xd9 kernel`nd6_prefix_offlink+0x1ce kernel`nd6_prefix_del+0x94 kernel`if_purgeaddrs+0x148 kernel`if_detach_internal+0x1e8 kernel`if_detach+0x71 if_epair.ko`epair_clone_destroy+0x62 kernel`if_clone_destroyif_flags+0x6a kernel`if_clone_destroy+0x100 kernel`ifioctl+0x8a5 kernel`kern_ioctl+0x286 kernel`sys_ioctl+0x152 kernel`amd64_syscall+0x153 kernel`0xffffffff8102315b </code></pre> <p dir=3D"auto">In other words, when we delete the interface <code style=3D= "padding: 0 0.25em; background-color: #E4E4E4;">if_detach_internal()</cod= e> purges the interface addresses, which ends up rebuilding the fib (<cod= e style=3D"padding: 0 0.25em; background-color: #E4E4E4;">rebuild_fd()</c= ode>) via <code style=3D"padding: 0 0.25em; background-color: #E4E4E4;">r= ib_del_route()</code>.<br> That ought to ensure that we cannot end up finding this struct ifnet thro= ugh <code style=3D"padding: 0 0.25em; background-color: #E4E4E4;">fib6_lo= okup()</code>, as the purging of the addresses (and thus the rebuilding o= f the fib) is done before we <code style=3D"padding: 0 0.25em; background= -color: #E4E4E4;">if_domdetach()</code> at the end of <code style=3D"padd= ing: 0 0.25em; background-color: #E4E4E4;">if_detach_internal()</code>, a= nd the NULL afdata[AF_INET6] demonstrates that we=E2=80=99ve gotten there= =2E</p> <p dir=3D"auto">We=E2=80=99ve also gone through <code style=3D"padding: 0= 0.25em; background-color: #E4E4E4;">if_free()</code>, as the ifindex_tab= le no longer contains the struct ifnet pointer for the relevant interface= =2E<br> We appear to have not yet called <code style=3D"padding: 0 0.25em; backgr= ound-color: #E4E4E4;">if_free_deferred()</code> (and indeed, ifp->if_r= efcount is 4, so we wouldn=E2=80=99t have called that yet).</p> <p dir=3D"auto">I=E2=80=99m confused as to how this can happen, and would= appreciate hints.</p> <p dir=3D"auto">Thanks,<br> Kristof</p> </div> </div> </body> </html> --=_MailMate_3433B8C0-4555-448B-877E-ABB0265B45A5_=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A5BC4385-AA39-4608-B1BA-0551AFBB49CD>