From nobody Thu May 23 22:12:20 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vlj8d5XmQz5LRTs for ; Thu, 23 May 2024 22:12:33 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE Root Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Vlj8c02zPz4KC1; Thu, 23 May 2024 22:12:31 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of bzeeb-lists@lists.zabbadoz.net designates 195.201.62.131 as permitted sender) smtp.mailfrom=bzeeb-lists@lists.zabbadoz.net Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 3A6DA8D4A126; Thu, 23 May 2024 22:12:24 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 7E5332D029D8; Thu, 23 May 2024 22:12:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id e8VdqL1ni7Ux; Thu, 23 May 2024 22:12:22 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 5768B2D029D2; Thu, 23 May 2024 22:12:21 +0000 (UTC) Date: Thu, 23 May 2024 22:12:20 +0000 (UTC) From: "Bjoern A. Zeeb" To: Zhenlei Huang cc: freebsd-net@freebsd.org Subject: Re: ifp gone in ip6_output() -> panic In-Reply-To: Message-ID: References: <1p003r05-684o-8542-r153-n850s3sspnp3@yvfgf.mnoonqbm.arg> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip4:195.201.62.131]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:24940, ipnet:195.201.0.0/16, country:DE]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zabbadoz.net]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4Vlj8c02zPz4KC1 On Wed, 22 May 2024, Zhenlei Huang wrote: > > >> On May 22, 2024, at 12:17 PM, Bjoern A. Zeeb wrote: >> >> Hi, >> >> sorry, I cannot dump; this is a diskless and netdump does not do IPv6; >> needless to say that would be funny in this case anyway; unfortunately >> I have also already re-compiled the kernel so I can only look things up approx. >> >> FreeBSD main from May 13 (f3eeeb959c9b00c89a2e1ff009c78162eb398656). >> >> I assume we lost the ifp from a destroy of a cloned interface in ip6_output() >> between lines 806 and 811? >> >> >> Kernel page fault with the following non-sleepable locks held: >> exclusive rw rawinp (rawinp) r = 0 (0xfffff80002a6e1a0) locked @ /usr/src/sys/netinet6/raw_ip6.c:393 >> stack backtrace: >> #0 0xffffffff80bb679c at witness_debugger+0x6c >> #1 0xffffffff80bb7979 at witness_warn+0x3e9 >> #2 0xffffffff81061d10 at trap_pfault+0x80 >> #3 0xffffffff81033878 at calltrap+0x8 >> #4 0xffffffff80d99228 at rip6_send+0x5a8 >> #5 0xffffffff80bf570e at sosend_generic+0x5ee >> #6 0xffffffff80bf5c49 at sousrsend+0x79 >> #7 0xffffffff80bfbd5c at kern_sendit+0x1bc >> #8 0xffffffff80bfc073 at sendit+0x1b3 >> #9 0xffffffff80bfc1ab at sys_sendmsg+0x5b >> #10 0xffffffff81062638 at amd64_syscall+0x158 >> #11 0xffffffff8103418b at fast_syscall_common+0xf8 >> Created wlan(4) interfaces: wlan > > Note the creation of wlan, and a following ICMP6 (ping6) packet. Yes I think it was running netif restart wlan0 in loops. [...] > > I'm not quite sure, but it seems the `ifp` is not fully constructed. See https://cgit.freebsd.org/src/tree/sys/net/if.c#n950 > > If I read the code correctly, the clone created interface is made visible via `if_link_ifnet(ifp);` , and at that time the > `ifp->if_afdata[AF_INET6]` is NULL and is not initialized yet by `if_attachdomain1()` which will call `in6_domifattach()` > to allocate the required data. > > So I guess there is a race condition. I bet this can be repeated easily. > > I have not tested this yet, and not sure if it is the right fix, but you can give it a try. I'll do; I haven't seen the error happening since on other test machines, so not sure about repeatability. I am also not entirely sure this is not a ping6 ff02::1%wlan0 while the ifp was destroyed by netif restart at the same time the packet was still on the way out? If it was during create, the wlan(4) interface would not be associated and UP at that point of if_attach_internal() and `ifconfig inet6 -ifdisabled` would not have been run to be able to send that packet in first place? Othwerwise the packet would have had to "survive" the clone destroy and clone create cycle somewhere ...? > diff --git a/sys/net/if.c b/sys/net/if.c > index c3c27fbf678f..16ee5667e7bb 100644 > --- a/sys/net/if.c > +++ b/sys/net/if.c > @@ -947,11 +947,11 @@ if_attach_internal(struct ifnet *ifp, bool vmove) > } > #endif > > - if_link_ifnet(ifp); > - > if (domain_init_status >= 2) > if_attachdomain1(ifp); > > + if_link_ifnet(ifp); > + > EVENTHANDLER_INVOKE(ifnet_arrival_event, ifp); > if (IS_DEFAULT_VNET(curvnet)) > devctl_notify("IFNET", ifp->if_xname, "ATTACH", NULL); -- Bjoern A. Zeeb r15:7