Date: Thu, 23 Jul 2020 09:19:45 +0200 From: "Kristof Provost" <kp@FreeBSD.org> To: "John-Mark Gurney" <jmg@funkthat.com> Cc: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net>, freebsd-net@freebsd.org, freebsd-current@freebsd.org Subject: Re: somewhat reproducable vimage panic Message-ID: <2FFC49F9-83DE-4FA1-A47F-1D8A7AF4B241@FreeBSD.org> In-Reply-To: <20200722221509.GI4213@funkthat.com> References: <20200721091654.GC4213@funkthat.com> <20200721113153.42d83119@x23> <20200721202323.GE4213@funkthat.com> <38F5A3A6-B578-4BA4-8F69-C248163CB6E0@libassi.se> <20200722060514.GF4213@funkthat.com> <20200722193443.GG4213@funkthat.com> <6C149617-55BB-4A87-B993-195E5E133790@lists.zabbadoz.net> <20200722221509.GI4213@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 23 Jul 2020, at 0:15, John-Mark Gurney wrote: > So, it's pretty easy to trigger, just attach a couple USB ethernet > adapters, in my case, they were ure, but likely any two spare ethernet > interfaces will work, and wire them back to back.. > I’ve been able to trigger it using epair as well: `sudo sh testinterfaces.txt epair0a epair0b` I did have to comment out the waitcarrier() check. Best regards, Kristof From owner-freebsd-current@freebsd.org Thu Jul 23 08:09:31 2020 Return-Path: <owner-freebsd-current@freebsd.org> Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5C1123759D9; Thu, 23 Jul 2020 08:09:31 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BC4k31rjvz4DQR; Thu, 23 Jul 2020 08:09:31 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id E7E0E17F2D; Thu, 23 Jul 2020 08:09:30 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 6D7823D5E4; Thu, 23 Jul 2020 10:09:29 +0200 (CEST) From: "Kristof Provost" <kp@FreeBSD.org> To: "John-Mark Gurney" <jmg@funkthat.com> Cc: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net>, freebsd-net@freebsd.org, freebsd-current@freebsd.org Subject: Re: somewhat reproducable vimage panic Date: Thu, 23 Jul 2020 10:09:25 +0200 X-Mailer: MailMate (1.13.1r5671) Message-ID: <6847FB6B-0B1A-43C7-B567-15BF21AB5D56@FreeBSD.org> In-Reply-To: <2FFC49F9-83DE-4FA1-A47F-1D8A7AF4B241@FreeBSD.org> References: <20200721091654.GC4213@funkthat.com> <20200721113153.42d83119@x23> <20200721202323.GE4213@funkthat.com> <38F5A3A6-B578-4BA4-8F69-C248163CB6E0@libassi.se> <20200722060514.GF4213@funkthat.com> <20200722193443.GG4213@funkthat.com> <6C149617-55BB-4A87-B993-195E5E133790@lists.zabbadoz.net> <20200722221509.GI4213@funkthat.com> <2FFC49F9-83DE-4FA1-A47F-1D8A7AF4B241@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current <freebsd-current.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-current>, <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>; List-Post: <mailto:freebsd-current@freebsd.org> List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-current>, <mailto:freebsd-current-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 23 Jul 2020 08:09:31 -0000 On 23 Jul 2020, at 9:19, Kristof Provost wrote: > On 23 Jul 2020, at 0:15, John-Mark Gurney wrote: >> So, it's pretty easy to trigger, just attach a couple USB ethernet >> adapters, in my case, they were ure, but likely any two spare >> ethernet >> interfaces will work, and wire them back to back.. >> > I’ve been able to trigger it using epair as well: > > `sudo sh testinterfaces.txt epair0a epair0b` > > I did have to comment out the waitcarrier() check. > I’ve done a little bit of digging, and I think I’m starting to see how this breaks. This always affects the jailed vlan interfaces. They’re getting deleted, but the ifp doesn’t go away just yet because it’s still in use by the multicast code. The multicast code does its cleanup in task queues, so by the time it gets around to doing that the ifp is already marked as dying and the vnet is gone. There are still references to the ifp though, and when the multicast code tries to do its cleanup we get the panic. This hack stops the panic for me, but I don’t know if this is the best solution: diff --git a/sys/net/if.c b/sys/net/if.c index 59dd38267cf..bd0c87eddf1 100644 --- a/sys/net/if.c +++ b/sys/net/if.c @@ -3681,6 +3685,10 @@ if_delmulti_ifma_flags(struct ifmultiaddr *ifma, int flags) ifp = NULL; } #endif + + if (ifp && ifp->if_flags & IFF_DYING) + return; + /* * If and only if the ifnet instance exists: Acquire the address lock. */ diff --git a/sys/netinet/in_mcast.c b/sys/netinet/in_mcast.c index 39fc82c5372..6493e2a5bfb 100644 --- a/sys/netinet/in_mcast.c +++ b/sys/netinet/in_mcast.c @@ -623,7 +623,7 @@ inm_release(struct in_multi *inm) /* XXX this access is not covered by IF_ADDR_LOCK */ CTR2(KTR_IGMPV3, "%s: purging ifma %p", __func__, ifma); - if (ifp != NULL) { + if (ifp != NULL && (ifp->if_flags & IFF_DYING) == 0) { CURVNET_SET(ifp->if_vnet); inm_purge(inm); free(inm, M_IPMADDR); Best regards, Kristof From owner-freebsd-current@freebsd.org Thu Jul 23 09:00:19 2020 Return-Path: <owner-freebsd-current@freebsd.org> Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 17184377514; Thu, 23 Jul 2020 09:00:19 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:13b:39f::9f:25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BC5rd66t0z4HMv; Thu, 23 Jul 2020 09:00:17 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 9A6F08D4A156; Thu, 23 Jul 2020 09:00:13 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 17B2AE708CD; Thu, 23 Jul 2020 09:00:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id WBYfcBpt6JUQ; Thu, 23 Jul 2020 09:00:11 +0000 (UTC) Received: from [127.0.0.1] (unknown [IPv6:fde9:577b:c1a9:4902:5ca3:9f20:8345:9910]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 40A2EE707BC; Thu, 23 Jul 2020 09:00:10 +0000 (UTC) From: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> To: "Kristof Provost" <kp@FreeBSD.org> Cc: "John-Mark Gurney" <jmg@funkthat.com>, freebsd-net@freebsd.org, freebsd-current@freebsd.org Subject: Re: somewhat reproducable vimage panic Date: Thu, 23 Jul 2020 09:00:09 +0000 X-Mailer: MailMate (2.0BETAr6146) Message-ID: <8B72C0B9-9CF0-4557-81D7-77190775805C@lists.zabbadoz.net> In-Reply-To: <6847FB6B-0B1A-43C7-B567-15BF21AB5D56@FreeBSD.org> References: <20200721091654.GC4213@funkthat.com> <20200721113153.42d83119@x23> <20200721202323.GE4213@funkthat.com> <38F5A3A6-B578-4BA4-8F69-C248163CB6E0@libassi.se> <20200722060514.GF4213@funkthat.com> <20200722193443.GG4213@funkthat.com> <6C149617-55BB-4A87-B993-195E5E133790@lists.zabbadoz.net> <20200722221509.GI4213@funkthat.com> <2FFC49F9-83DE-4FA1-A47F-1D8A7AF4B241@FreeBSD.org> <6847FB6B-0B1A-43C7-B567-15BF21AB5D56@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4BC5rd66t0z4HMv X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of bzeeb-lists@lists.zabbadoz.net designates 2a01:4f8:13b:39f::9f:25 as permitted sender) smtp.mailfrom=bzeeb-lists@lists.zabbadoz.net X-Spamd-Result: default: False [-2.87 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; NEURAL_HAM_MEDIUM(-0.96)[-0.956]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:4f8:13b:39f::9f:25]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zabbadoz.net]; NEURAL_HAM_LONG(-1.02)[-1.020]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.59)[-0.591]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current <freebsd-current.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-current>, <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>; List-Post: <mailto:freebsd-current@freebsd.org> List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-current>, <mailto:freebsd-current-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 23 Jul 2020 09:00:19 -0000 On 23 Jul 2020, at 8:09, Kristof Provost wrote: > On 23 Jul 2020, at 9:19, Kristof Provost wrote: >> On 23 Jul 2020, at 0:15, John-Mark Gurney wrote: >>> So, it's pretty easy to trigger, just attach a couple USB ethernet >>> adapters, in my case, they were ure, but likely any two spare >>> ethernet >>> interfaces will work, and wire them back to back.. >>> >> I’ve been able to trigger it using epair as well: >> >> `sudo sh testinterfaces.txt epair0a epair0b` >> >> I did have to comment out the waitcarrier() check. >> > I’ve done a little bit of digging, and I think I’m starting to see > how this breaks. > > This always affects the jailed vlan interfaces. They’re getting > deleted, but the ifp doesn’t go away just yet because it’s still > in use by the multicast code. > The multicast code does its cleanup in task queues, Wow, did I miss that back then? Did I review a change and not notice? Sorry if that was the case. Vnet teardown is blocking and forceful. Doing deferred cleanup work isn’t a good idea at all. I think that is the real problem here. I’d rather have us fix this than putting more bandaids into the code. /bz PS: I love that you can repro this with epairs, means we can write a generic test code piece for this and commit it. > so by the time it gets around to doing that the ifp is already marked > as dying and the vnet is gone. > There are still references to the ifp though, and when the multicast > code tries to do its cleanup we get the panic. > > This hack stops the panic for me, but I don’t know if this is the > best solution: > > diff --git a/sys/net/if.c b/sys/net/if.c > index 59dd38267cf..bd0c87eddf1 100644 > --- a/sys/net/if.c > +++ b/sys/net/if.c > @@ -3681,6 +3685,10 @@ if_delmulti_ifma_flags(struct ifmultiaddr > *ifma, int flags) > ifp = NULL; > } > #endif > + > + if (ifp && ifp->if_flags & IFF_DYING) > + return; > + > /* > * If and only if the ifnet instance exists: Acquire the address > lock. > */ > diff --git a/sys/netinet/in_mcast.c b/sys/netinet/in_mcast.c > index 39fc82c5372..6493e2a5bfb 100644 > --- a/sys/netinet/in_mcast.c > +++ b/sys/netinet/in_mcast.c > @@ -623,7 +623,7 @@ inm_release(struct in_multi *inm) > > /* XXX this access is not covered by IF_ADDR_LOCK */ > CTR2(KTR_IGMPV3, "%s: purging ifma %p", __func__, ifma); > - if (ifp != NULL) { > + if (ifp != NULL && (ifp->if_flags & IFF_DYING) == 0) { > CURVNET_SET(ifp->if_vnet); > inm_purge(inm); > free(inm, M_IPMADDR); > > Best regards, > Kristof
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2FFC49F9-83DE-4FA1-A47F-1D8A7AF4B241>