From nobody Wed Oct 25 15:27:25 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SFt8d1Dv7z4xdwL for ; Wed, 25 Oct 2023 15:27:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SFt8d0jvrz4V6L for ; Wed, 25 Oct 2023 15:27:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698247649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ChOulduti+rbHJNrsMOGSatMwQmrJhQWpCQTdFXWp+0=; b=B0i7EvmceMznGHApd6+H7Fk+7Kzp1x9dm+gJu7KRJFNIpXyaJ01M7tVOsI+t+jVnC6s9Dz 2wWG/eRayIbS0Yool9MTQz3yRprKPuKfwUdj6vSG7GRuVp7a3ERvX6SDj/6ygOgNnIlhj+ n8g4iI9mps3Z4y62PcftFgGgCC2w4Tb0y7vDaAT8Xa2Ki/d9sU2KA93mjQUPeMGcBE3BdT /+FaVOZhhNEgIGXv0G6761pWhjP/0nzFJIXOEPekIlc4Y8lEh0fg24wwUEaWU+maWwdcpE ekc7EFP1Ou/ldyVUghK3VoUzNhWL/0dp4DVYkRvYTwjmpC4neFCWktIxCxyw2A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1698247649; a=rsa-sha256; cv=none; b=aB5Ic8PmR5VMHMgnGTHvADh0maTkBmmcsf+7JC0cZKtBNKp1KAxY74goPEIRpPolHLg+Hy xuUPuZGR77Eylix3yTt76IA9y3UV0Ae+kX/zSxNKho1TYFEfesQnqsQ4E/UWy5KLYriHVZ 0fL+n9dw/QdoKDW+7P/cV1LvGd2cpOSEgG1CMJQ9TuYalhN6/fquOatd8lzVYek1gUCIM2 1VWewmzH+qIVslTOIfrSDauFUXo/VAJnt8HRBmHfodnHAp8xiMhwWkzkJUYaqUPJl8UQDv mPiibUPo1QYauYSAPsln76xVJFffDVzhYA/fkXxnVG4OKr2JjRL1U2uZkolp8w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698247649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ChOulduti+rbHJNrsMOGSatMwQmrJhQWpCQTdFXWp+0=; b=XRfsVSOR6XrhnD8ClyFtCzWJpcMr7i2j+Jmi2PRR4FzPFN4SxgaPYvNfbR6jo8fBakx56B mPpEv58EHOpViSI1JZRrYM1RVMT4w97W9PleCbUq5nRPDwSnpTDMr/kIORfhTVJ7Hgk8VL LOGHcJWZzWYi8ZVV+gtXYN3MWURkGuLqQFS40DiKrtrr1Zgewkvm7Sb7Mc/YiAK5h7imso 1PY+xQ/PUdr4H7NRkMRK9P8lw5elnjbIX0zwonelxCtMWpZ3xxeOIXehVecnwtRpWtlnid oXfH41CUc/cVPNI5ccX5F4S5wN6ZxgFSFbX8VN8QyksXCcc6eNjIHsiAHtLHCg== Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.codepro.be", Issuer "R3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 4SFt8c5bXszv3m for ; Wed, 25 Oct 2023 15:27:28 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id DE76F1B6F7; Wed, 25 Oct 2023 17:27:26 +0200 (CEST) From: Kristof Provost To: FreeBSD Net Subject: fib6_lookup() returning deleted struct ifnet Date: Wed, 25 Oct 2023 17:27:25 +0200 X-Mailer: MailMate (1.14r5937) Message-ID: List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_MailMate_3433B8C0-4555-448B-877E-ABB0265B45A5_=" Content-Transfer-Encoding: 8bit --=_MailMate_3433B8C0-4555-448B-877E-ABB0265B45A5_= Content-Type: text/plain; charset=UTF-8; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit Hi, Several pfSense users report IPv6-related panics when an interface is deleted. The relevant bug reports are https://redmine.pfsense.org/issues/14164 and https://redmine.pfsense.org/issues/14431. The latest report is for a build that includes commits up to 1a18383a52bc373e316d224cef1298debf6f7e25 (“libcrypto: link engines and the legacy provider to libcrypto”, September 15th). I believe all reports are for users running PPPoE, via netgraph, but that might be coincidental, as that’s the most likely way for interfaces to be destroyed (when PPP disconnects and reconnects). There are a few different backtraces, but they appear to have the same root cause, so I’ll focus on one of them: db:1:pfs> bt Tracing pid 2 tid 100041 td 0xfffffe0085264560 kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850ad910 vpanic() at vpanic+0x183/frame 0xfffffe00850ad960 panic() at panic+0x43/frame 0xfffffe00850ad9c0 trap_fatal() at trap_fatal+0x409/frame 0xfffffe00850ada20 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850ada80 calltrap() at calltrap+0x8/frame 0xfffffe00850ada80 --- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00850adb50, rbp = 0xfffffe00850adb80 --- in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850adb80 tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00850add70 tcp_timer_rexmt() at tcp_timer_rexmt+0x514/frame 0xfffffe00850addd0 tcp_timer_enter() at tcp_timer_enter+0x102/frame 0xfffffe00850ade10 softclock_call_cc() at softclock_call_cc+0x13c/frame 0xfffffe00850adec0 softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850adef0 fork_exit() at fork_exit+0x7d/frame 0xfffffe00850adf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850adf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- This happens in the TCP output path, where we look up the hop limit for a specific destination. I’ve obtained a core dump for such a crash, and I believe the panic happens on line https://cgit.freebsd.org/src/tree/sys/netinet6/in6_src.c#n861 The call in tcp_default_output() is `in6_selecthlim(int, NULL);`, so we don’t get an ifp from the caller, but instead perform a route lookup, and try to obtain the hop limit through `ND_IFINFO(nh->nh_ifp)`. This panics because the afdata[AF_INET6] pointer is NULL. The core dump shows a deleted structure ifnet: (kgdb) p *(struct ifnet *)0xfffff80203712800 $3 = { if_link = { cstqe_next = 0x0 }, if_clones = { le_next = 0x0, le_prev = 0x0 }, if_groups = { cstqh_first = 0x0, cstqh_last = 0xfffff80203712818 }, if_alloctype = 53 '5', if_numa_domain = 255 '\377', if_softc = 0xfffff80103447a00, if_llsoftc = 0x0, if_l2com = 0x0, if_dname = 0xffffffff81492f70 "ng", if_dunit = 0, if_index = 14, if_idxgen = 2, if_xname = "pppoe0\000\000\000\000\000\000\000\000\000", if_description = 0xfffff8003a5f83d0 "WAN", if_flags = 2132112, if_drv_flags = 0, if_capabilities = 0, if_capabilities2 = 0, … if_afdata = {0x0 }, … if_output = 0xffffffff80e29c60 , if_input = 0xffffffff80e29c80 , if_bridge_input = 0x0, if_bridge_output = 0x0, if_bridge_linkstate = 0x0, if_start = 0xffffffff80e29c90 , if_ioctl = 0xffffffff80e29ca0 , … My understanding is that the fib table should get updated whenever we change the routing table (such as during interface cleanup in `if_detach_internal()`). Some quick experimentation with epair and dtrace also shows: 20 20388 sync_algo_end_cb:entry Stage 1 kernel`setup_fd_instance+0x41f kernel`rebuild_fd_flm+0x99 kernel`rebuild_fd+0x136 kernel`rib_notify+0x50 kernel`rt_delete_conditional+0xf1 kernel`rib_del_route+0x1fc kernel`rib_handle_ifaddr_info+0xd9 kernel`nd6_prefix_offlink+0x1ce kernel`nd6_prefix_del+0x94 kernel`if_purgeaddrs+0x148 kernel`if_detach_internal+0x1e8 kernel`if_detach+0x71 if_epair.ko`epair_clone_destroy+0x62 kernel`if_clone_destroyif_flags+0x6a kernel`if_clone_destroy+0x100 kernel`ifioctl+0x8a5 kernel`kern_ioctl+0x286 kernel`sys_ioctl+0x152 kernel`amd64_syscall+0x153 kernel`0xffffffff8102315b In other words, when we delete the interface `if_detach_internal()` purges the interface addresses, which ends up rebuilding the fib (`rebuild_fd()`) via `rib_del_route()`. That ought to ensure that we cannot end up finding this struct ifnet through `fib6_lookup()`, as the purging of the addresses (and thus the rebuilding of the fib) is done before we `if_domdetach()` at the end of `if_detach_internal()`, and the NULL afdata[AF_INET6] demonstrates that we’ve gotten there. We’ve also gone through `if_free()`, as the ifindex_table no longer contains the struct ifnet pointer for the relevant interface. We appear to have not yet called `if_free_deferred()` (and indeed, ifp->if_refcount is 4, so we wouldn’t have called that yet). I’m confused as to how this can happen, and would appreciate hints. Thanks, Kristof --=_MailMate_3433B8C0-4555-448B-877E-ABB0265B45A5_= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Hi,

Several pfSense users report IPv6-related panics when an = interface is deleted.
The relevant bug reports are https://redmine.pfsense.org/issues/14164 and https://redmine.pfsense.org/issues/= 14431.
The latest report is for a build that includes commits up to 1a18383a52bc= 373e316d224cef1298debf6f7e25 (=E2=80=9Clibcrypto: link engines and the le= gacy provider to libcrypto=E2=80=9D, September 15th).

I believe all reports are for users running PPPoE, via ne= tgraph, but that might be coincidental, as that=E2=80=99s the most likely= way for interfaces to be destroyed (when PPP disconnects and reconnects)= =2E

There are a few different backtraces, but they appear to = have the same root cause, so I=E2=80=99ll focus on one of them:

db=
:1:pfs> bt
Tracing pid 2 tid 100041 td 0xfffffe0085264560
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850ad910
vpanic() at vpanic+0x183/frame 0xfffffe00850ad960
panic() at panic+0x43/frame 0xfffffe00850ad9c0
trap_fatal() at trap_fatal+0x409/frame 0xfffffe00850ada20
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850ada80
calltrap() at calltrap+0x8/frame 0xfffffe00850ada80
--- trap 0xc, rip =3D 0xffffffff80f5a036, rsp =3D 0xfffffe00850adb50, rbp=
 =3D 0xfffffe00850adb80 ---
in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850adb80
tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00850add7=
0
tcp_timer_rexmt() at tcp_timer_rexmt+0x514/frame 0xfffffe00850addd0
tcp_timer_enter() at tcp_timer_enter+0x102/frame 0xfffffe00850ade10
softclock_call_cc() at softclock_call_cc+0x13c/frame 0xfffffe00850adec0
softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850adef0
fork_exit() at fork_exit+0x7d/frame 0xfffffe00850adf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850adf30
--- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 ---

This happens in the TCP output path, where we look up the= hop limit for a specific destination. I=E2=80=99ve obtained a core dump = for such a crash, and I believe the panic happens on line https://cgit.f= reebsd.org/src/tree/sys/netinet6/in6_src.c#n861

The call in tcp_default_output() is in6_selecthlim(int, NULL);, so we don=E2=80=99t get an ifp from the caller, but instead perform a = route lookup, and try to obtain the hop limit through ND_IFINFO(nh->nh_ifp). This panics because the afdata[AF_INET6] pointer is NULL. The core dum= p shows a deleted structure ifnet:

(k=
gdb) p *(struct ifnet *)0xfffff80203712800
$3 =3D {
  if_link =3D {
    cstqe_next =3D 0x0
  },
  if_clones =3D {
    le_next =3D 0x0,
    le_prev =3D 0x0
  },
  if_groups =3D {
    cstqh_first =3D 0x0,
    cstqh_last =3D 0xfffff80203712818
  },
  if_alloctype =3D 53 '5',
  if_numa_domain =3D 255 '\377',
  if_softc =3D 0xfffff80103447a00,
  if_llsoftc =3D 0x0,
  if_l2com =3D 0x0,
  if_dname =3D 0xffffffff81492f70 "ng",
  if_dunit =3D 0,
  if_index =3D 14,
  if_idxgen =3D 2,
  if_xname =3D "pppoe0\000\000\000\000\000\000\000\000\000",
  if_description =3D 0xfffff8003a5f83d0 "WAN",
  if_flags =3D 2132112,
  if_drv_flags =3D 0,
  if_capabilities =3D 0,
  if_capabilities2 =3D 0,
=E2=80=A6
  if_afdata =3D {0x0 <repeats 44 times>},
=E2=80=A6
  if_output =3D 0xffffffff80e29c60 <ifdead_output>,
  if_input =3D 0xffffffff80e29c80 <ifdead_input>,
  if_bridge_input =3D 0x0,
  if_bridge_output =3D 0x0,
  if_bridge_linkstate =3D 0x0,
  if_start =3D 0xffffffff80e29c90 <ifdead_start>,
  if_ioctl =3D 0xffffffff80e29ca0 <ifdead_ioctl>,
=E2=80=A6

My understanding is that the fib table should get updated= whenever we change the routing table (such as during interface cleanup i= n if_detach= _internal()). Some quick experimentation with epair and dtrace als= o shows:

 2=
0  20388           sync_algo_end_cb:entry Stage 1
              kernel`setup_fd_instance+0x41f
              kernel`rebuild_fd_flm+0x99
              kernel`rebuild_fd+0x136
              kernel`rib_notify+0x50
              kernel`rt_delete_conditional+0xf1
              kernel`rib_del_route+0x1fc
              kernel`rib_handle_ifaddr_info+0xd9
              kernel`nd6_prefix_offlink+0x1ce
              kernel`nd6_prefix_del+0x94
              kernel`if_purgeaddrs+0x148
              kernel`if_detach_internal+0x1e8
              kernel`if_detach+0x71
              if_epair.ko`epair_clone_destroy+0x62
              kernel`if_clone_destroyif_flags+0x6a
              kernel`if_clone_destroy+0x100
              kernel`ifioctl+0x8a5
              kernel`kern_ioctl+0x286
              kernel`sys_ioctl+0x152
              kernel`amd64_syscall+0x153
              kernel`0xffffffff8102315b

In other words, when we delete the interface if_detach_internal() purges the interface addresses, which ends up rebuilding the fib (rebuild_fd()) via r= ib_del_route().
That ought to ensure that we cannot end up finding this struct ifnet thro= ugh fib6_lo= okup(), as the purging of the addresses (and thus the rebuilding o= f the fib) is done before we if_domdetach() at the end of if_detach_internal(), a= nd the NULL afdata[AF_INET6] demonstrates that we=E2=80=99ve gotten there= =2E

We=E2=80=99ve also gone through if_free(), as the ifindex_tab= le no longer contains the struct ifnet pointer for the relevant interface= =2E
We appear to have not yet called if_free_deferred() (and indeed, ifp->if_r= efcount is 4, so we wouldn=E2=80=99t have called that yet).

I=E2=80=99m confused as to how this can happen, and would= appreciate hints.

Thanks,
Kristof

--=_MailMate_3433B8C0-4555-448B-877E-ABB0265B45A5_=--