From owner-freebsd-net@freebsd.org Wed Jul 22 04:54:23 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 89A463708F4 for ; Wed, 22 Jul 2020 04:54:23 +0000 (UTC) (envelope-from peter@libassi.se) Received: from smtp.outgoing.loopia.se (smtp.outgoing.loopia.se [93.188.3.37]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBNRL1Gj2z4fWH for ; Wed, 22 Jul 2020 04:54:21 +0000 (UTC) (envelope-from peter@libassi.se) Received: from s807.loopia.se (localhost [127.0.0.1]) by s807.loopia.se (Postfix) with ESMTP id F34A32F651B3 for ; Wed, 22 Jul 2020 06:54:18 +0200 (CEST) Received: from s499.loopia.se (unknown [172.22.191.6]) by s807.loopia.se (Postfix) with ESMTP id D49AE2E27E7E; Wed, 22 Jul 2020 06:54:18 +0200 (CEST) Received: from s475.loopia.se (unknown [172.22.191.5]) by s499.loopia.se (Postfix) with ESMTP id D07391CDAEBC; Wed, 22 Jul 2020 06:54:18 +0200 (CEST) X-Virus-Scanned: amavisd-new at amavis.loopia.se X-Spam-Flag: NO X-Spam-Score: -0.999 X-Spam-Level: X-Spam-Status: No, score=-0.999 tagged_above=-999 required=6.2 tests=[ALL_TRUSTED=-1, HTML_MESSAGE=0.001] autolearn=disabled Received: from s499.loopia.se ([172.22.191.5]) by s475.loopia.se (s475.loopia.se [172.22.190.15]) (amavisd-new, port 10024) with LMTP id UOlLaKgzaQ42; Wed, 22 Jul 2020 06:54:17 +0200 (CEST) X-Loopia-Auth: user X-Loopia-User: peter@libassi.se X-Loopia-Originating-IP: 217.31.161.79 Received: from [10.0.2.62] (h-161-79.A444.priv.bahnhof.se [217.31.161.79]) (Authenticated sender: peter@libassi.se) by s499.loopia.se (Postfix) with ESMTPSA id BF4001CDADDC; Wed, 22 Jul 2020 06:54:17 +0200 (CEST) From: Peter Libassi Message-Id: <38F5A3A6-B578-4BA4-8F69-C248163CB6E0@libassi.se> Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: somewhat reproducable vimage panic Date: Wed, 22 Jul 2020 06:54:17 +0200 In-Reply-To: <20200721202323.GE4213@funkthat.com> Cc: Marko Zec , freebsd-net@freebsd.org, freebsd-current@freebsd.org To: John-Mark Gurney References: <20200721091654.GC4213@funkthat.com> <20200721113153.42d83119@x23> <20200721202323.GE4213@funkthat.com> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 4BBNRL1Gj2z4fWH X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of peter@libassi.se has no SPF policy when checking 93.188.3.37) smtp.mailfrom=peter@libassi.se X-Spamd-Result: default: False [-0.76 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; NEURAL_HAM_MEDIUM(-0.77)[-0.771]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[libassi.se]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-0.75)[-0.751]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.64)[-0.635]; RCVD_IN_DNSWL_NONE(0.00)[93.188.3.37:from]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:39570, ipnet:93.188.2.0/23, country:SE]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[93.188.3.37:from] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 04:54:23 -0000 Is this related to=20 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234985 = and = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D238326 = /Peter > 21 juli 2020 kl. 22:23 skrev John-Mark Gurney : >=20 > Marko Zec wrote this message on Tue, Jul 21, 2020 at 11:31 +0200: >> On Tue, 21 Jul 2020 02:16:55 -0700 >> John-Mark Gurney wrote: >>=20 >>> I'm running: >>> FreeBSD test 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r362596: Thu Jun = 25 >>> 05:02:51 UTC 2020 >>> = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC >>> amd64 >>>=20 >>> and I'm working on improve the if_ure driver. I've put together a >>> little script that I've attached that I'm using to test the driver.. >>> It puts a couple ue interfaces each into their own jail, configures >>> them, and tries to pass traffic. This assumes that the two = interfaces >>> are connected together. >>>=20 >>> Pretty regularly when destroying the jails, I get the following >>> panic: CURVNET_SET at /usr/src/sys/netinet/in_mcast.c:626 >>> inm_release() curvnet=3D0 vnet=3D0xfffff80154c82a80 >>=20 >> Perhaps the attached patch could help? (disclaimer: not even >> compile-tested) >=20 > The patch compiled, but it just moved the panic earlier than before. >=20 > #4 0xffffffff80bc2123 in panic (fmt=3D) > at ../../../kern/kern_shutdown.c:839 > #5 0xffffffff80d61726 in inm_release_task (arg=3D,=20 > pending=3D) at ../../../netinet/in_mcast.c:633 > #6 0xffffffff80c2166a in taskqueue_run_locked = (queue=3D0xfffff800033cfd00) > at ../../../kern/subr_taskqueue.c:476 > #7 0xffffffff80c226e4 in taskqueue_thread_loop (arg=3D) > at ../../../kern/subr_taskqueue.c:793 >=20 > Now it panics at the location of the new CURVNET_SET and not the > old one.. >=20 > Ok, decided to dump the contents of the vnet, and it looks like > it's a use after free: > (kgdb) print/x *(struct vnet *)0xfffff8012a283140 > $2 =3D {vnet_le =3D {le_next =3D 0xdeadc0dedeadc0de, le_prev =3D = 0xdeadc0dedeadc0de}, vnet_magic_n =3D 0xdeadc0de,=20 > vnet_ifcnt =3D 0xdeadc0de, vnet_sockcnt =3D 0xdeadc0de, vnet_state =3D = 0xdeadc0de, vnet_data_mem =3D 0xdeadc0dedeadc0de,=20 > vnet_data_base =3D 0xdeadc0dedeadc0de, vnet_shutdown =3D 0xde} >=20 > The patch did seem to make it happen quicker, or maybe I was just more > lucky this morning... >=20 >>> (kgdb) #0 __curthread () at = /usr/src/sys/amd64/include/pcpu_aux.h:55 >>> #1 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:394 >>> #2 0xffffffff80bc6250 in kern_reboot (howto=3D260) >>> at /usr/src/sys/kern/kern_shutdown.c:481 >>> #3 0xffffffff80bc66aa in vpanic (fmt=3D, = ap=3D>> out>) at /usr/src/sys/kern/kern_shutdown.c:913 >>> #4 0xffffffff80bc6403 in panic (fmt=3D) >>> at /usr/src/sys/kern/kern_shutdown.c:839 >>> #5 0xffffffff80d6553b in inm_release (inm=3D0xfffff80029043700) >>> at /usr/src/sys/netinet/in_mcast.c:630 >>> #6 inm_release_task (arg=3D, pending=3D) >>> at /usr/src/sys/netinet/in_mcast.c:312 >>> #7 0xffffffff80c2521a in taskqueue_run_locked >>> (queue=3D0xfffff80003116b00) at = /usr/src/sys/kern/subr_taskqueue.c:476 >>> #8 0xffffffff80c26294 in taskqueue_thread_loop (arg=3D) >>> at /usr/src/sys/kern/subr_taskqueue.c:793 >>> #9 0xffffffff80b830f0 in fork_exit ( >>> callout=3D0xffffffff80c26200 ,=20 >>> arg=3D0xffffffff81cf4f70 , >>> frame=3D0xfffffe0049e99b80) at /usr/src/sys/kern/kern_fork.c:1052 >>> #10 >>> (kgdb)=20 >>>=20 >>> I have the core files so I can get additional information. >>>=20 >>> Let me know if you need any additional information. >>>=20 >>=20 >=20 >> Index: sys/netinet/in_mcast.c >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> --- sys/netinet/in_mcast.c (revision 363386) >> +++ sys/netinet/in_mcast.c (working copy) >> @@ -309,8 +309,10 @@ >> IN_MULTI_LOCK(); >> SLIST_FOREACH_SAFE(inm, &inm_free_tmp, inm_nrele, tinm) { >> SLIST_REMOVE_HEAD(&inm_free_tmp, inm_nrele); >> + CURVNET_SET(inm->inm_ifp->if_vnet); >> MPASS(inm); >> inm_release(inm); >> + CURVNET_RESTORE(); >> } >> IN_MULTI_UNLOCK(); >> } >=20 >=20 > --=20 > John-Mark Gurney Voice: +1 415 225 5579 >=20 > "All that I will do, has been done, All that I have, has not." > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"