From nobody Wed May 22 10:06:19 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vkn5G2QpJz5LNmR for ; Wed, 22 May 2024 10:06:26 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Vkn5G1w8jz4vSS; Wed, 22 May 2024 10:06:26 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1716372386; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7FslKhnPZyjp/5omujOS9WMYXI4fxUMn2INZiwsOTOc=; b=NXTE/a0wMkZGkvfbQI1C/SwgkGZi2/brM9RNPbYtBSrf9MaAlmLI2C+iMCsXqeQWzpjBly NVX+nX3eSxZNJPC2FZSX5tqwB1+KR+WPqi6hKAmU63BDach9cG2mrZbDqQ3IpXv1jJseiX ru6mT9QrOc8N0OT8DY791TzaO7QF0Ird4UGvveOYy3aNcDJvPwf6WaHohIzjY/DW+YMRB+ VMGP03/a91VwsAOP5+KCEYnhKHxTwtNUur0R6JvIKDSGuG2V+nKidwICBhh3deuC0p8aQx UNCAIinjd2cVmSRLe3noeNImkCALJxr3rAMP3+SyaUnyueh4xGYcDrMc3RQK3w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1716372386; a=rsa-sha256; cv=none; b=Kfv4tR2PUX1o3E2+k+xL63Nd+NPHnaguUFZeTshyTj9ykIjeLklK2RckEBvL9mr9S4dgfB CA5hfnT8MLahECMhc55bGm7GnTdh9mic49/8QRLsiG/YuaP8cyhYTMz71X+iO06nQQ/PMc /arv5jymhBlNIQ1v6GvTtAEVd0sKyH/pDKop6mRpB2BwFeZQE3KEphyNOZWVyUisdzehVM LlT0QfeBfxTs6FP5tk4utDtMiXyIg+mafa+837Ya7NGe8uGVm6iKlwwTZO0Ir+lnHjFmFW im18P7THHsRI73e/cjhtvDUxJuNn2v3Yiahc2n7WUTssxX+Y6DD7BByOzOE8Rg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1716372386; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7FslKhnPZyjp/5omujOS9WMYXI4fxUMn2INZiwsOTOc=; b=Yqbq5QZTcMGyhBkvdTs4ZHWOET+ca6W/VXMjtYuIR9iMg3WppT5DnMZM8xfxD+/4UV2H/R x3q6JC+O/5yPjvp7FgW7wHEYh+oHF1HCXDMdbEtAhtGZeDhDpHjpI1NPn85hzUqnMvVaFG 0+LzXC2EQ9kG3+Px5WidWurkEFLKam6OQS4pX32OAKV8iBFS2mkENONMZQrhr3TKzlWnDE CU2deapeGtv1/uJW1utnm8NgJtWx7le1NVIkTGMzdiMthDOh1HFflG9V3Yk9mXaSEcQjjU A1udQvjZvUcwVsKWaFMS4YpXgYfz93dE4WGqvPjSlocOQloTOrBpkmH1vKpAEA== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Vkn5D6fbCzf2m; Wed, 22 May 2024 10:06:24 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_0715C355-6B28-4360-86DE-DF9D867C1F8C" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\)) Subject: Re: ifp gone in ip6_output() -> panic Date: Wed, 22 May 2024 18:06:19 +0800 In-Reply-To: <1p003r05-684o-8542-r153-n850s3sspnp3@yvfgf.mnoonqbm.arg> Cc: freebsd-net@freebsd.org To: "Bjoern A. Zeeb" References: <1p003r05-684o-8542-r153-n850s3sspnp3@yvfgf.mnoonqbm.arg> X-Mailer: Apple Mail (2.3696.120.41.1.8) --Apple-Mail=_0715C355-6B28-4360-86DE-DF9D867C1F8C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On May 22, 2024, at 12:17 PM, Bjoern A. Zeeb = wrote: >=20 > Hi, >=20 > sorry, I cannot dump; this is a diskless and netdump does not do IPv6; > needless to say that would be funny in this case anyway; unfortunately > I have also already re-compiled the kernel so I can only look things = up approx. >=20 > FreeBSD main from May 13 (f3eeeb959c9b00c89a2e1ff009c78162eb398656). >=20 > I assume we lost the ifp from a destroy of a cloned interface in = ip6_output() > between lines 806 and 811? >=20 >=20 > Kernel page fault with the following non-sleepable locks held: > exclusive rw rawinp (rawinp) r =3D 0 (0xfffff80002a6e1a0) locked @ = /usr/src/sys/netinet6/raw_ip6.c:393 > stack backtrace: > #0 0xffffffff80bb679c at witness_debugger+0x6c > #1 0xffffffff80bb7979 at witness_warn+0x3e9 > #2 0xffffffff81061d10 at trap_pfault+0x80 > #3 0xffffffff81033878 at calltrap+0x8 > #4 0xffffffff80d99228 at rip6_send+0x5a8 > #5 0xffffffff80bf570e at sosend_generic+0x5ee > #6 0xffffffff80bf5c49 at sousrsend+0x79 > #7 0xffffffff80bfbd5c at kern_sendit+0x1bc > #8 0xffffffff80bfc073 at sendit+0x1b3 > #9 0xffffffff80bfc1ab at sys_sendmsg+0x5b > #10 0xffffffff81062638 at amd64_syscall+0x158 > #11 0xffffffff8103418b at fast_syscall_common+0xf8 > Created wlan(4) interfaces: wlan Note the creation of wlan, and a following ICMP6 (ping6) packet. > Fatal trap 12: page fault while in kernel mode > cpuid =3D 2; apic id =3D 02 > fault virtual address =3D 0x0 > 30. > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff80d821bd > stack pointer =3D 0x28:0xfffffe00468ba850 > frame pointer =3D 0x28:0xfffffe00468baa00 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 1073 (ping6) > rdi: ffffffff81be2c00 rsi: fffffe00468ba884 rdx: 00000000030002ff > rcx: 0000000000001c1c r8: 0000000000000028 r9: 0000000000000003 > rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe00468baa00 > r10: 000000000000aa01 r11: 00000000000000ff r12: fffff800017d3000 > r13: fffff8000286df20 r14: 0000000000000000 r15: fffff80001f1059c > trap number =3D 12 > panic: page fault > cpuid =3D 2 > time =3D 1716352160 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe00468ba520 > vpanic() at vpanic+0x13f/frame 0xfffffe00468ba650 > panic() at panic+0x43/frame 0xfffffe00468ba6b0 > trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00468ba710 > trap_pfault() at trap_pfault+0xa0/frame 0xfffffe00468ba780 > calltrap() at calltrap+0x8/frame 0xfffffe00468ba780 > --- trap 0xc, rip =3D 0xffffffff80d821bd, rsp =3D 0xfffffe00468ba850, = rbp =3D 0xfffffe00468baa00 --- > ip6_output() at ip6_output+0xb5d/frame 0xfffffe00468baa00 > rip6_send() at rip6_send+0x5a8/frame 0xfffffe00468babb0 > sosend_generic() at sosend_generic+0x5ee/frame 0xfffffe00468bac60 > sousrsend() at sousrsend+0x79/frame 0xfffffe00468bacc0 > kern_sendit() at kern_sendit+0x1bc/frame 0xfffffe00468bad50 > sendit() at sendit+0x1b3/frame 0xfffffe00468bada0 > sys_sendmsg() at sys_sendmsg+0x5b/frame 0xfffffe00468bae00 > amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe00468baf30 > fast_syscall_common() at fast_syscall_common+0xf8/frame = 0xfffffe00468baf30 > --- syscall (28, FreeBSD ELF64, sendmsg), rip =3D 0x2959dfacc85a, rsp = =3D 0x2959dae92668, rbp =3D 0x2959dae926c0 --- > KDB: enter: panic > [ thread pid 1073 tid 100157 ] > Stopped at kdb_enter+0x33: movq $0,0x10544e2(%rip) >=20 >=20 > (lldb) image lookup -v --address rip6_send+0x5a8 > Address: kernel.full[0xffffffff80d99228] = (kernel.full.PT_LOAD[1]..text + 10572328) > Summary: kernel.full`rip6_send + 1448 at raw_ip6.c:533:10 > Module: file =3D = "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.full", arch =3D = "x86_64" > CompileUnit: id =3D {0x000005ba}, file =3D = "/usr/src/sys/netinet6/raw_ip6.c", language =3D "c99" > Function: id =3D {0x023e5ddd}, name =3D "rip6_send", range =3D = [0xffffffff80d98c80-0xffffffff80d9973d) > FuncType: id =3D {0x023e5ddd}, byte-size =3D 0, decl =3D = raw_ip6.c:338, compiler_type =3D "int (struct socket *, int, struct mbuf = *, struct sockaddr *, struct mbuf *, struct thread *)" > Blocks: id =3D {0x023e5ddd}, range =3D = [0xffffffff80d98c80-0xffffffff80d9973d) > LineEntry: [0xffffffff80d99211-0xffffffff80d9922a): = /usr/src/sys/netinet6/raw_ip6.c:533:10 > Symbol: id =3D {0x0000840a}, range =3D = [0xffffffff80d98c80-0xffffffff80d9973d), name=3D"rip6_send" > Variable: id =3D {0x023e5df7}, name =3D "so", type =3D "socket *", = valid ranges =3D , location =3D [0xffffffff80d990c0, = 0xffffffff80d99398) -> DW_OP_breg6 RBP-168, decl =3D raw_ip6.c:338 > Variable: id =3D {0x023e5e07}, name =3D "flags", type =3D "int", = valid ranges =3D , location =3D , decl =3D raw_ip6.c:338 > Variable: id =3D {0x023e5e13}, name =3D "m", type =3D "mbuf *", = valid ranges =3D , location =3D [0xffffffff80d990c0, = 0xffffffff80d99398) -> DW_OP_reg12 R12, decl =3D raw_ip6.c:338 > Variable: id =3D {0x023e5e23}, name =3D "nam", type =3D "sockaddr = *", valid ranges =3D , location =3D [0xffffffff80d98e48, = 0xffffffff80d99398) -> DW_OP_GNU_entry_value(DW_OP_reg2 RCX), = DW_OP_stack_value, decl =3D raw_ip6.c:338 > Variable: id =3D {0x023e5e33}, name =3D "control", type =3D "mbuf = *", valid ranges =3D , location =3D [0xffffffff80d990c0, = 0xffffffff80d99398) -> DW_OP_breg6 RBP-48, decl =3D raw_ip6.c:339 > Variable: id =3D {0x023e5e43}, name =3D "td", type =3D "thread *", = valid ranges =3D , location =3D , decl =3D raw_ip6.c:339 > Variable: id =3D {0x023e5e4f}, name =3D "et", type =3D = "epoch_tracker", valid ranges =3D , location =3D DW_OP_fbreg = -200, decl =3D raw_ip6.c:341 > Variable: id =3D {0x023e5e5f}, name =3D "opt", type =3D = "ip6_pktopts", valid ranges =3D , location =3D DW_OP_fbreg -408, = decl =3D raw_ip6.c:347 > Variable: id =3D {0x023e5e6f}, name =3D "oifp", type =3D "ifnet = *", valid ranges =3D , location =3D DW_OP_fbreg -56, decl =3D = raw_ip6.c:348 > Variable: id =3D {0x023e5e7e}, name =3D "hlim", type =3D "int", = valid ranges =3D , location =3D DW_OP_fbreg -80, decl =3D = raw_ip6.c:353 > Variable: id =3D {0x023e5e8e}, name =3D "in6a", type =3D = "in6_addr", valid ranges =3D , location =3D DW_OP_fbreg -120, = decl =3D raw_ip6.c:354 > Variable: id =3D {0x023e5ebe}, name =3D "type", type =3D "int", = valid ranges =3D , location =3D [0xffffffff80d990c0, = 0xffffffff80d99398) -> DW_OP_breg6 RBP-72, decl =3D raw_ip6.c:350 > Variable: id =3D {0x023e5ece}, name =3D "code", type =3D "int", = valid ranges =3D , location =3D [0xffffffff80d990c0, = 0xffffffff80d99398) -> DW_OP_breg6 RBP-76, decl =3D raw_ip6.c:350 > Variable: id =3D {0x023e5ede}, name =3D "scope_ambiguous", type =3D = "int", valid ranges =3D , location =3D [0xffffffff80d990c0, = 0xffffffff80d99398) -> DW_OP_breg6 RBP-92, decl =3D raw_ip6.c:351 > Variable: id =3D {0x023e5efe}, name =3D "inp", type =3D "inpcb *", = valid ranges =3D , location =3D [0xffffffff80d98cb4, = 0xffffffff80d9973d) -> DW_OP_breg6 RBP-64, decl =3D raw_ip6.c:342 > Variable: id =3D {0x023e5f0e}, name =3D "error", type =3D "int", = valid ranges =3D , location =3D [0xffffffff80d991a9, = 0xffffffff80d9922a) -> DW_OP_consts +0, DW_OP_stack_value, decl =3D = raw_ip6.c:349 > Variable: id =3D {0x023e5f2e}, name =3D "optp", type =3D = "ip6_pktopts *", valid ranges =3D , location =3D = [0xffffffff80d99183, 0xffffffff80d9922a) -> DW_OP_reg3 RBX, decl =3D = raw_ip6.c:347 > Variable: id =3D {0x023e5f3e}, name =3D "ip6", type =3D "ip6_hdr = *", valid ranges =3D , location =3D [0xffffffff80d990c0, = 0xffffffff80d99398) -> DW_OP_breg6 RBP-160, decl =3D raw_ip6.c:345 >=20 > 532 NET_EPOCH_ENTER(et); > 533 error =3D ip6_output(m, optp, NULL, 0, = inp->in6p_moptions, &oifp, inp); > 534 NET_EPOCH_EXIT(et); >=20 >=20 > (lldb) image lookup -v --address ip6_output+0xb5d > Address: kernel.full[0xffffffff80d821bd] = (kernel.full.PT_LOAD[1]..text + 10478013) > Summary: kernel.full`ip6_output + 2909 at ip6_output.c:811:3 > Module: file =3D = "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.full", arch =3D = "x86_64" > CompileUnit: id =3D {0x000005b5}, file =3D = "/usr/src/sys/netinet6/ip6_output.c", language =3D "c99" > Function: id =3D {0x023ab5ce}, name =3D "ip6_output", range =3D = [0xffffffff80d81660-0xffffffff80d838c4) > FuncType: id =3D {0x023ab5ce}, byte-size =3D 0, decl =3D = ip6_output.c:403, compiler_type =3D "int (struct mbuf *, struct = ip6_pktopts *, struct route_in6 *, int, struct ip6_moptions *, struct = ifnet **, struct inpcb *)" > Blocks: id =3D {0x023ab5ce}, range =3D = [0xffffffff80d81660-0xffffffff80d838c4) > LineEntry: [0xffffffff80d821b6-0xffffffff80d821c4): = /usr/src/sys/netinet6/ip6_output.c:811:3 > Symbol: id =3D {0x000101b9}, range =3D = [0xffffffff80d81660-0xffffffff80d838c4), name=3D"ip6_output" > Variable: id =3D {0x023ab5e8}, name =3D "m0", type =3D "mbuf *", = valid ranges =3D , location =3D [0xffffffff80d819c2, = 0xffffffff80d82a26) -> DW_OP_GNU_entry_value(DW_OP_reg5 RDI), = DW_OP_stack_value, decl =3D ip6_output.c:403 > Variable: id =3D {0x023ab5f8}, name =3D "opt", type =3D = "ip6_pktopts *", valid ranges =3D , location =3D = [0xffffffff80d8168e, 0xffffffff80d838c4) -> DW_OP_breg6 RBP-184, decl =3D = ip6_output.c:403 > Variable: id =3D {0x023ab608}, name =3D "ro", type =3D "route_in6 = *", valid ranges =3D , location =3D [0xffffffff80d81a3b, = 0xffffffff80d824c0) -> DW_OP_reg3 RBX, decl =3D ip6_output.c:404 > Variable: id =3D {0x023ab618}, name =3D "flags", type =3D "int", = valid ranges =3D , location =3D [0xffffffff80d81684, = 0xffffffff80d838c4) -> DW_OP_breg6 RBP-152, decl =3D ip6_output.c:404 > Variable: id =3D {0x023ab628}, name =3D "im6o", type =3D = "ip6_moptions *", valid ranges =3D , location =3D = [0xffffffff80d816b7, 0xffffffff80d83032) -> DW_OP_breg6 RBP-352, decl =3D = ip6_output.c:404 > Variable: id =3D {0x023ab638}, name =3D "ifpp", type =3D "ifnet = **", valid ranges =3D , location =3D [0xffffffff80d8167b, = 0xffffffff80d838c4) -> DW_OP_breg6 RBP-304, decl =3D ip6_output.c:405 > Variable: id =3D {0x023ab648}, name =3D "inp", type =3D "inpcb *", = valid ranges =3D , location =3D DW_OP_fbreg +16, decl =3D = ip6_output.c:405 > Variable: id =3D {0x023ab657}, name =3D "m", type =3D "mbuf *", = valid ranges =3D , location =3D DW_OP_fbreg -48, decl =3D = ip6_output.c:409 > Variable: id =3D {0x023ab666}, name =3D "sin6", type =3D = "sockaddr_in6", valid ranges =3D , location =3D DW_OP_fbreg -380, = decl =3D ip6_output.c:413 > Variable: id =3D {0x023ab676}, name =3D "src_sa", type =3D = "sockaddr_in6", valid ranges =3D , location =3D DW_OP_fbreg -336, = decl =3D ip6_output.c:413 > Variable: id =3D {0x023ab686}, name =3D "dst_sa", type =3D = "sockaddr_in6", valid ranges =3D , location =3D DW_OP_fbreg -128, = decl =3D ip6_output.c:413 > Variable: id =3D {0x023ab696}, name =3D "odst", type =3D = "in6_addr", valid ranges =3D , location =3D DW_OP_fbreg -432, = decl =3D ip6_output.c:414 > Variable: id =3D {0x023ab6a6}, name =3D "alwaysfrag", type =3D = "int", valid ranges =3D , location =3D DW_OP_fbreg -192, decl =3D = ip6_output.c:421 > Variable: id =3D {0x023ab6b6}, name =3D "exthdrs", type =3D = "ip6_exthdrs", valid ranges =3D , location =3D DW_OP_fbreg -248, = decl =3D ip6_output.c:423 > Variable: id =3D {0x023ab6c6}, name =3D "src0", type =3D = "in6_addr", valid ranges =3D , location =3D DW_OP_fbreg -416, = decl =3D ip6_output.c:424 > Variable: id =3D {0x023ab6d6}, name =3D "dst0", type =3D = "in6_addr", valid ranges =3D , location =3D DW_OP_fbreg -400, = decl =3D ip6_output.c:424 > Variable: id =3D {0x023ab6e6}, name =3D "zone", type =3D = "u_int32_t", valid ranges =3D , location =3D DW_OP_fbreg -188, = decl =3D ip6_output.c:425 > Variable: id =3D {0x023ab726}, name =3D "error", type =3D "int", = valid ranges =3D , location =3D [0xffffffff80d8214f, = 0xffffffff80d8251a) -> DW_OP_consts +0, DW_OP_stack_value, decl =3D = ip6_output.c:417 > Variable: id =3D {0x023ab736}, name =3D "vlan_pcp", type =3D = "int", valid ranges =3D , location =3D [0xffffffff80d819c2, = 0xffffffff80d82a18) -> DW_OP_breg6 RBP-96, decl =3D ip6_output.c:418 > Variable: id =3D {0x023ab746}, name =3D "ia", type =3D "in6_ifaddr = *", valid ranges =3D , location =3D [0xffffffff80d82187, = 0xffffffff80d82a18) -> DW_OP_breg6 RBP-264, decl =3D ip6_output.c:419 > Variable: id =3D {0x023ab776}, name =3D "ip6", type =3D "ip6_hdr = *", valid ranges =3D , location =3D [0xffffffff80d81e29, = 0xffffffff80d82665) -> DW_OP_breg6 RBP-168, decl =3D ip6_output.c:407 > Variable: id =3D {0x023ab7c6}, name =3D "nexthdrp", type =3D = "u_char *", valid ranges =3D , location =3D [0xffffffff80d81a4f, = 0xffffffff80d82a18) -> DW_OP_breg6 RBP-136, decl =3D ip6_output.c:415 > Variable: id =3D {0x023ab7d6}, name =3D "ro_pmtu", type =3D = "route_in6 *", valid ranges =3D , location =3D = [0xffffffff80d81ab4, 0xffffffff80d824c0) -> DW_OP_reg3 RBX, decl =3D = ip6_output.c:411 > Variable: id =3D {0x023ab7e6}, name =3D "dst", type =3D = "sockaddr_in6 *", valid ranges =3D , location =3D = [0xffffffff80d81b39, 0xffffffff80d82466) -> DW_OP_breg6 RBP-88, decl =3D = ip6_output.c:413 > Variable: id =3D {0x023ab7f6}, name =3D "fibnum", type =3D = "uint32_t", valid ranges =3D , location =3D [0xffffffff80d81b26, = 0xffffffff80d827e3) -> DW_OP_breg6 RBP-72, decl =3D ip6_output.c:429 > Variable: id =3D {0x023ab806}, name =3D "origifp", type =3D "ifnet = *", valid ranges =3D , location =3D [0xffffffff80d8218b, = 0xffffffff80d824ae) -> DW_OP_reg12 R12, decl =3D ip6_output.c:408 > Variable: id =3D {0x023ab876}, name =3D "tlen", type =3D "int", = valid ranges =3D , location =3D , decl =3D = ip6_output.c:416 > Variable: id =3D {0x023ab882}, name =3D "dontfrag", type =3D = "int", valid ranges =3D , location =3D , decl =3D = ip6_output.c:421 >=20 >=20 > 806 KASSERT((ifp !=3D NULL), ("output interface must not be = NULL")); > 807 KASSERT((origifp !=3D NULL), ("output address interface = must not be NULL")); > 808 > 809 if ((flags & IPV6_FORWARDING) =3D=3D 0) { > 810 /* XXX: the FORWARDING flag can be set for = mrouting. */ > 811 in6_ifstat_inc(ifp, ifs6_out_request); I'm not quite sure, but it seems the `ifp` is not fully constructed. See = https://cgit.freebsd.org/src/tree/sys/net/if.c#n950 = =20 If I read the code correctly, the clone created interface is made = visible via `if_link_ifnet(ifp);` , and at that time the `ifp->if_afdata[AF_INET6]` is NULL and is not initialized yet by = `if_attachdomain1()` which will call `in6_domifattach()` to allocate the required data. So I guess there is a race condition. I bet this can be repeated easily. I have not tested this yet, and not sure if it is the right fix, but you = can give it a try. diff --git a/sys/net/if.c b/sys/net/if.c index c3c27fbf678f..16ee5667e7bb 100644 --- a/sys/net/if.c +++ b/sys/net/if.c @@ -947,11 +947,11 @@ if_attach_internal(struct ifnet *ifp, bool vmove) } #endif =20 - if_link_ifnet(ifp); - if (domain_init_status >=3D 2) if_attachdomain1(ifp); =20 + if_link_ifnet(ifp); + EVENTHANDLER_INVOKE(ifnet_arrival_event, ifp); if (IS_DEFAULT_VNET(curvnet)) devctl_notify("IFNET", ifp->if_xname, "ATTACH", NULL); > 812 } > 813 > 814 /* Setup data structures for scope ID checks. */ >=20 > --=20 > Bjoern A. Zeeb = r15:7 >=20 Best regards, Zhenlei --Apple-Mail=_0715C355-6B28-4360-86DE-DF9D867C1F8C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On May 22, 2024, at 12:17 PM, Bjoern A. Zeeb <bzeeb-lists@lists.zabbadoz.net> wrote:

Hi,
sorry, I cannot dump; this is a diskless and = netdump does not do IPv6;
needless to say that would be = funny in this case anyway; unfortunately
I have also = already re-compiled the kernel so I can only look things up approx.

FreeBSD main from May 13 = (f3eeeb959c9b00c89a2e1ff009c78162eb398656).

I= assume we lost the ifp from a destroy of a cloned interface in = ip6_output()
between lines 806 and 811?


Kernel page fault with the following = non-sleepable locks held:
exclusive rw rawinp (rawinp) r =3D= 0 (0xfffff80002a6e1a0) locked @ /usr/src/sys/netinet6/raw_ip6.c:393
stack backtrace:
#0 0xffffffff80bb679c at = witness_debugger+0x6c
#1 0xffffffff80bb7979 at = witness_warn+0x3e9
#2 0xffffffff81061d10 at = trap_pfault+0x80
#3 0xffffffff81033878 at calltrap+0x8
#4 0xffffffff80d99228 at rip6_send+0x5a8
#5 = 0xffffffff80bf570e at sosend_generic+0x5ee
#6 = 0xffffffff80bf5c49 at sousrsend+0x79
#7 0xffffffff80bfbd5c = at kern_sendit+0x1bc
#8 0xffffffff80bfc073 at = sendit+0x1b3
#9 0xffffffff80bfc1ab at sys_sendmsg+0x5b
#10 0xffffffff81062638 at amd64_syscall+0x158
#11= 0xffffffff8103418b at fast_syscall_common+0xf8
Created = wlan(4) interfaces: wlan

Note the creation of wlan, and a following ICMP6 = (ping6) packet.

Fatal trap 12: page fault = while in kernel mode
cpuid =3D 2; apic id =3D 02
fault virtual address   =3D 0x0
30.
fault code =             &n= bsp;=3D supervisor read data, page not present
instruction = pointer     =3D 0x20:0xffffffff80d821bd
stack pointer =           =3D = 0x28:0xfffffe00468ba850
frame pointer =           =3D = 0x28:0xfffffe00468baa00
code segment =            =3D = base 0x0, limit 0xfffff, type 0x1b
=             &n= bsp;          =3D DPL = 0, pres 1, long 1, def32 0, gran 1
processor eflags =        =3D interrupt enabled, resume, = IOPL =3D 0
current process =         =3D 1073 (ping6)
rdi: ffffffff81be2c00 rsi: fffffe00468ba884 rdx: = 00000000030002ff
rcx: 0000000000001c1c  r8: = 0000000000000028  r9: 0000000000000003
rax: = 0000000000000000 rbx: 0000000000000000 rbp: fffffe00468baa00
r10: 000000000000aa01 r11: 00000000000000ff r12: = fffff800017d3000
r13: fffff8000286df20 r14: = 0000000000000000 r15: fffff80001f1059c
trap number =             =3D= 12
panic: page fault
cpuid =3D 2
time =3D 1716352160
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe00468ba520
vpanic() at vpanic+0x13f/frame = 0xfffffe00468ba650
panic() at panic+0x43/frame = 0xfffffe00468ba6b0
trap_fatal() at trap_fatal+0x40b/frame = 0xfffffe00468ba710
trap_pfault() at trap_pfault+0xa0/frame = 0xfffffe00468ba780
calltrap() at calltrap+0x8/frame = 0xfffffe00468ba780
--- trap 0xc, rip =3D = 0xffffffff80d821bd, rsp =3D 0xfffffe00468ba850, rbp =3D = 0xfffffe00468baa00 ---
ip6_output() at = ip6_output+0xb5d/frame 0xfffffe00468baa00
rip6_send() at = rip6_send+0x5a8/frame 0xfffffe00468babb0
sosend_generic() = at sosend_generic+0x5ee/frame 0xfffffe00468bac60
sousrsend()= at sousrsend+0x79/frame 0xfffffe00468bacc0
kern_sendit() = at kern_sendit+0x1bc/frame 0xfffffe00468bad50
sendit() at = sendit+0x1b3/frame 0xfffffe00468bada0
sys_sendmsg() at = sys_sendmsg+0x5b/frame 0xfffffe00468bae00
amd64_syscall() = at amd64_syscall+0x158/frame 0xfffffe00468baf30
fast_syscall_common() at fast_syscall_common+0xf8/frame = 0xfffffe00468baf30
--- syscall (28, FreeBSD ELF64, = sendmsg), rip =3D 0x2959dfacc85a, rsp =3D 0x2959dae92668, rbp =3D = 0x2959dae926c0 ---
KDB: enter: panic
[ = thread pid 1073 tid 100157 ]
Stopped at =      kdb_enter+0x33: movq =    $0,0x10544e2(%rip)


(lldb) image lookup -v --address rip6_send+0x5a8
=      Address: kernel.full[0xffffffff80d99228] = (kernel.full.PT_LOAD[1]..text + 10572328)
=      Summary: kernel.full`rip6_send + 1448 at = raw_ip6.c:533:10
=       Module: file =3D = "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.full", arch =3D = "x86_64"
 CompileUnit: id =3D {0x000005ba}, file =3D = "/usr/src/sys/netinet6/raw_ip6.c", language =3D "c99"
=     Function: id =3D {0x023e5ddd}, name =3D = "rip6_send", range =3D [0xffffffff80d98c80-0xffffffff80d9973d)
    FuncType: id =3D {0x023e5ddd}, = byte-size =3D 0, decl =3D raw_ip6.c:338, compiler_type =3D "int (struct = socket *, int, struct mbuf *, struct sockaddr *, struct mbuf *, struct = thread *)"
      Blocks: id = =3D {0x023e5ddd}, range =3D [0xffffffff80d98c80-0xffffffff80d9973d)
   LineEntry: = [0xffffffff80d99211-0xffffffff80d9922a): = /usr/src/sys/netinet6/raw_ip6.c:533:10
=       Symbol: id =3D {0x0000840a}, range =3D= [0xffffffff80d98c80-0xffffffff80d9973d), name=3D"rip6_send"
    Variable: id =3D {0x023e5df7}, name = =3D "so", type =3D "socket *", valid ranges =3D <block>, location = =3D [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_breg6 RBP-168, = decl =3D raw_ip6.c:338
    Variable: = id =3D {0x023e5e07}, name =3D "flags", type =3D "int", valid ranges =3D = <block>, location =3D <empty>, decl =3D raw_ip6.c:338
    Variable: id =3D {0x023e5e13}, name = =3D "m", type =3D "mbuf *", valid ranges =3D <block>, location =3D = [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_reg12 R12, decl =3D = raw_ip6.c:338
    Variable: id =3D = {0x023e5e23}, name =3D "nam", type =3D "sockaddr *", valid ranges =3D = <block>, location =3D [0xffffffff80d98e48, 0xffffffff80d99398) = -> DW_OP_GNU_entry_value(DW_OP_reg2 RCX), DW_OP_stack_value, decl =3D = raw_ip6.c:338
    Variable: id =3D = {0x023e5e33}, name =3D "control", type =3D "mbuf *", valid ranges =3D = <block>, location =3D [0xffffffff80d990c0, 0xffffffff80d99398) = -> DW_OP_breg6 RBP-48, decl =3D raw_ip6.c:339
=     Variable: id =3D {0x023e5e43}, name =3D "td", = type =3D "thread *", valid ranges =3D <block>, location =3D = <empty>, decl =3D raw_ip6.c:339
=     Variable: id =3D {0x023e5e4f}, name =3D "et", = type =3D "epoch_tracker", valid ranges =3D <block>, location =3D = DW_OP_fbreg -200, decl =3D raw_ip6.c:341
=     Variable: id =3D {0x023e5e5f}, name =3D "opt", = type =3D "ip6_pktopts", valid ranges =3D <block>, location =3D = DW_OP_fbreg -408, decl =3D raw_ip6.c:347
=     Variable: id =3D {0x023e5e6f}, name =3D "oifp", = type =3D "ifnet *", valid ranges =3D <block>, location =3D = DW_OP_fbreg -56, decl =3D raw_ip6.c:348
=     Variable: id =3D {0x023e5e7e}, name =3D "hlim", = type =3D "int", valid ranges =3D <block>, location =3D DW_OP_fbreg = -80, decl =3D raw_ip6.c:353
=     Variable: id =3D {0x023e5e8e}, name =3D "in6a", = type =3D "in6_addr", valid ranges =3D <block>, location =3D = DW_OP_fbreg -120, decl =3D raw_ip6.c:354
=     Variable: id =3D {0x023e5ebe}, name =3D "type", = type =3D "int", valid ranges =3D <block>, location =3D = [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_breg6 RBP-72, decl = =3D raw_ip6.c:350
    Variable: id =3D = {0x023e5ece}, name =3D "code", type =3D "int", valid ranges =3D = <block>, location =3D [0xffffffff80d990c0, 0xffffffff80d99398) = -> DW_OP_breg6 RBP-76, decl =3D raw_ip6.c:350
=     Variable: id =3D {0x023e5ede}, name =3D = "scope_ambiguous", type =3D "int", valid ranges =3D <block>, = location =3D [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_breg6 = RBP-92, decl =3D raw_ip6.c:351
=     Variable: id =3D {0x023e5efe}, name =3D "inp", = type =3D "inpcb *", valid ranges =3D <block>, location =3D = [0xffffffff80d98cb4, 0xffffffff80d9973d) -> DW_OP_breg6 RBP-64, decl = =3D raw_ip6.c:342
    Variable: id =3D = {0x023e5f0e}, name =3D "error", type =3D "int", valid ranges =3D = <block>, location =3D [0xffffffff80d991a9, 0xffffffff80d9922a) = -> DW_OP_consts +0, DW_OP_stack_value, decl =3D raw_ip6.c:349
    Variable: id =3D {0x023e5f2e}, name = =3D "optp", type =3D "ip6_pktopts *", valid ranges =3D <block>, = location =3D [0xffffffff80d99183, 0xffffffff80d9922a) -> DW_OP_reg3 = RBX, decl =3D raw_ip6.c:347
=     Variable: id =3D {0x023e5f3e}, name =3D "ip6", = type =3D "ip6_hdr *", valid ranges =3D <block>, location =3D = [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_breg6 RBP-160, decl = =3D raw_ip6.c:345

   532 =         NET_EPOCH_ENTER(et);
   533 =         error =3D ip6_output(m, = optp, NULL, 0, inp->in6p_moptions, &oifp, inp);
=    534 =         NET_EPOCH_EXIT(et);


(lldb) image lookup -v = --address ip6_output+0xb5d
=      Address: kernel.full[0xffffffff80d821bd] = (kernel.full.PT_LOAD[1]..text + 10478013)
=      Summary: kernel.full`ip6_output + 2909 at = ip6_output.c:811:3
=       Module: file =3D = "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.full", arch =3D = "x86_64"
 CompileUnit: id =3D {0x000005b5}, file =3D = "/usr/src/sys/netinet6/ip6_output.c", language =3D "c99"
=     Function: id =3D {0x023ab5ce}, name =3D = "ip6_output", range =3D [0xffffffff80d81660-0xffffffff80d838c4)
    FuncType: id =3D {0x023ab5ce}, = byte-size =3D 0, decl =3D ip6_output.c:403, compiler_type =3D "int = (struct mbuf *, struct ip6_pktopts *, struct route_in6 *, int, struct = ip6_moptions *, struct ifnet **, struct inpcb *)"
=       Blocks: id =3D {0x023ab5ce}, range =3D= [0xffffffff80d81660-0xffffffff80d838c4)
=    LineEntry: [0xffffffff80d821b6-0xffffffff80d821c4): = /usr/src/sys/netinet6/ip6_output.c:811:3
=       Symbol: id =3D {0x000101b9}, range =3D= [0xffffffff80d81660-0xffffffff80d838c4), name=3D"ip6_output"
    Variable: id =3D {0x023ab5e8}, name = =3D "m0", type =3D "mbuf *", valid ranges =3D <block>, location =3D = [0xffffffff80d819c2, 0xffffffff80d82a26) -> = DW_OP_GNU_entry_value(DW_OP_reg5 RDI), DW_OP_stack_value, decl =3D = ip6_output.c:403
    Variable: id =3D = {0x023ab5f8}, name =3D "opt", type =3D "ip6_pktopts *", valid ranges =3D = <block>, location =3D [0xffffffff80d8168e, 0xffffffff80d838c4) = -> DW_OP_breg6 RBP-184, decl =3D ip6_output.c:403
=     Variable: id =3D {0x023ab608}, name =3D "ro", = type =3D "route_in6 *", valid ranges =3D <block>, location =3D = [0xffffffff80d81a3b, 0xffffffff80d824c0) -> DW_OP_reg3 RBX, decl =3D = ip6_output.c:404
    Variable: id =3D = {0x023ab618}, name =3D "flags", type =3D "int", valid ranges =3D = <block>, location =3D [0xffffffff80d81684, 0xffffffff80d838c4) = -> DW_OP_breg6 RBP-152, decl =3D ip6_output.c:404
=     Variable: id =3D {0x023ab628}, name =3D "im6o", = type =3D "ip6_moptions *", valid ranges =3D <block>, location =3D = [0xffffffff80d816b7, 0xffffffff80d83032) -> DW_OP_breg6 RBP-352, decl = =3D ip6_output.c:404
    Variable: id = =3D {0x023ab638}, name =3D "ifpp", type =3D "ifnet **", valid ranges =3D = <block>, location =3D [0xffffffff80d8167b, 0xffffffff80d838c4) = -> DW_OP_breg6 RBP-304, decl =3D ip6_output.c:405
=     Variable: id =3D {0x023ab648}, name =3D "inp", = type =3D "inpcb *", valid ranges =3D <block>, location =3D = DW_OP_fbreg +16, decl =3D ip6_output.c:405
=     Variable: id =3D {0x023ab657}, name =3D "m", = type =3D "mbuf *", valid ranges =3D <block>, location =3D = DW_OP_fbreg -48, decl =3D ip6_output.c:409
=     Variable: id =3D {0x023ab666}, name =3D "sin6", = type =3D "sockaddr_in6", valid ranges =3D <block>, location =3D = DW_OP_fbreg -380, decl =3D ip6_output.c:413
=     Variable: id =3D {0x023ab676}, name =3D = "src_sa", type =3D "sockaddr_in6", valid ranges =3D <block>, = location =3D DW_OP_fbreg -336, decl =3D ip6_output.c:413
=     Variable: id =3D {0x023ab686}, name =3D = "dst_sa", type =3D "sockaddr_in6", valid ranges =3D <block>, = location =3D DW_OP_fbreg -128, decl =3D ip6_output.c:413
=     Variable: id =3D {0x023ab696}, name =3D "odst", = type =3D "in6_addr", valid ranges =3D <block>, location =3D = DW_OP_fbreg -432, decl =3D ip6_output.c:414
=     Variable: id =3D {0x023ab6a6}, name =3D = "alwaysfrag", type =3D "int", valid ranges =3D <block>, location =3D= DW_OP_fbreg -192, decl =3D ip6_output.c:421
=     Variable: id =3D {0x023ab6b6}, name =3D = "exthdrs", type =3D "ip6_exthdrs", valid ranges =3D <block>, = location =3D DW_OP_fbreg -248, decl =3D ip6_output.c:423
=     Variable: id =3D {0x023ab6c6}, name =3D "src0", = type =3D "in6_addr", valid ranges =3D <block>, location =3D = DW_OP_fbreg -416, decl =3D ip6_output.c:424
=     Variable: id =3D {0x023ab6d6}, name =3D "dst0", = type =3D "in6_addr", valid ranges =3D <block>, location =3D = DW_OP_fbreg -400, decl =3D ip6_output.c:424
=     Variable: id =3D {0x023ab6e6}, name =3D "zone", = type =3D "u_int32_t", valid ranges =3D <block>, location =3D = DW_OP_fbreg -188, decl =3D ip6_output.c:425
=     Variable: id =3D {0x023ab726}, name =3D "error", = type =3D "int", valid ranges =3D <block>, location =3D = [0xffffffff80d8214f, 0xffffffff80d8251a) -> DW_OP_consts +0, = DW_OP_stack_value, decl =3D ip6_output.c:417
=     Variable: id =3D {0x023ab736}, name =3D = "vlan_pcp", type =3D "int", valid ranges =3D <block>, location =3D = [0xffffffff80d819c2, 0xffffffff80d82a18) -> DW_OP_breg6 RBP-96, decl = =3D ip6_output.c:418
    Variable: id = =3D {0x023ab746}, name =3D "ia", type =3D "in6_ifaddr *", valid ranges =3D= <block>, location =3D [0xffffffff80d82187, 0xffffffff80d82a18) = -> DW_OP_breg6 RBP-264, decl =3D ip6_output.c:419
=     Variable: id =3D {0x023ab776}, name =3D "ip6", = type =3D "ip6_hdr *", valid ranges =3D <block>, location =3D = [0xffffffff80d81e29, 0xffffffff80d82665) -> DW_OP_breg6 RBP-168, decl = =3D ip6_output.c:407
    Variable: id = =3D {0x023ab7c6}, name =3D "nexthdrp", type =3D "u_char *", valid ranges = =3D <block>, location =3D [0xffffffff80d81a4f, 0xffffffff80d82a18) = -> DW_OP_breg6 RBP-136, decl =3D ip6_output.c:415
=     Variable: id =3D {0x023ab7d6}, name =3D = "ro_pmtu", type =3D "route_in6 *", valid ranges =3D <block>, = location =3D [0xffffffff80d81ab4, 0xffffffff80d824c0) -> DW_OP_reg3 = RBX, decl =3D ip6_output.c:411
=     Variable: id =3D {0x023ab7e6}, name =3D "dst", = type =3D "sockaddr_in6 *", valid ranges =3D <block>, location =3D = [0xffffffff80d81b39, 0xffffffff80d82466) -> DW_OP_breg6 RBP-88, decl = =3D ip6_output.c:413
    Variable: id = =3D {0x023ab7f6}, name =3D "fibnum", type =3D "uint32_t", valid ranges =3D= <block>, location =3D [0xffffffff80d81b26, 0xffffffff80d827e3) = -> DW_OP_breg6 RBP-72, decl =3D ip6_output.c:429
=     Variable: id =3D {0x023ab806}, name =3D = "origifp", type =3D "ifnet *", valid ranges =3D <block>, location = =3D [0xffffffff80d8218b, 0xffffffff80d824ae) -> DW_OP_reg12 R12, decl = =3D ip6_output.c:408
    Variable: id = =3D {0x023ab876}, name =3D "tlen", type =3D "int", valid ranges =3D = <block>, location =3D <empty>, decl =3D ip6_output.c:416
    Variable: id =3D {0x023ab882}, name = =3D "dontfrag", type =3D "int", valid ranges =3D <block>, location = =3D <empty>, decl =3D ip6_output.c:421


   806 =         KASSERT((ifp !=3D NULL), = ("output interface must not be NULL"));
=    807 =         KASSERT((origifp !=3D = NULL), ("output address interface must not be NULL"));
=    808
   809 =         if ((flags & = IPV6_FORWARDING) =3D=3D 0) {
   810 =             &n= bsp;   /* XXX: the FORWARDING flag can be set for = mrouting. */
   811 =             &n= bsp;   in6_ifstat_inc(ifp, ifs6_out_request);

I'm = not quite sure, but it seems the `ifp` is not fully constructed. = See https://cgit.freebsd.org/src/tree/sys/net/if.c#n950 

If I read the code correctly, the = clone created interface is made visible via `if_link_ifnet(ifp);` , and = at that time the
`ifp->if_afdata[AF_INET6]` is NULL and is = not initialized yet by `if_attachdomain1()` which will call = `in6_domifattach()`
to allocate the required = data.

So I guess there is a race = condition. I bet this can be repeated easily.

I have not tested this yet, and not sure if it is = the right fix, but you can give it a try.

diff --git a/sys/net/if.c = b/sys/net/if.c
index c3c27fbf678f..16ee5667e7bb = 100644
--- a/sys/net/if.c
+++ = b/sys/net/if.c
@@ -947,11 +947,11 @@ if_attach_internal(struct = ifnet *ifp, bool vmove)
        = }
 #endif
 
-     =   if_link_ifnet(ifp);
-
      =   if (domain_init_status >=3D 2)
      =           = if_attachdomain1(ifp);
 
+       = if_link_ifnet(ifp);
+
        = EVENTHANDLER_INVOKE(ifnet_arrival_event, ifp);
    =     if (IS_DEFAULT_VNET(curvnet))
    =             devctl_notify("IFNET", = ifp->if_xname, "ATTACH", NULL);

=    812 =         }
=    813
   814 =         /* Setup data structures = for scope ID checks. */

--
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7


Best regards,
Zhenlei

= --Apple-Mail=_0715C355-6B28-4360-86DE-DF9D867C1F8C--