From owner-freebsd-virtualization@FreeBSD.ORG Sun Jul 6 18:12:33 2014 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AEA8523B; Sun, 6 Jul 2014 18:12:33 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4EE3E278E; Sun, 6 Jul 2014 18:12:33 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s66ICQ2p044478 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 6 Jul 2014 21:12:26 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua s66ICQ2p044478 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s66ICQXm044477; Sun, 6 Jul 2014 21:12:26 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 6 Jul 2014 21:12:26 +0300 From: Konstantin Belousov To: Steve Wills Subject: Re: tmpfs panic Message-ID: <20140706181226.GE93733@kib.kiev.ua> References: <20140706135333.GA80856@mouf.net> <20140706154621.GA81830@mouf.net> <20140706172511.GA84461@mouf.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="l118U0+vX1D/6gtA" Content-Disposition: inline In-Reply-To: <20140706172511.GA84461@mouf.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: virtualization@freebsd.org, Ryan Stone , FreeBSD Current X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jul 2014 18:12:33 -0000 --l118U0+vX1D/6gtA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jul 06, 2014 at 05:25:12PM +0000, Steve Wills wrote: > On Sun, Jul 06, 2014 at 12:28:07PM -0400, Ryan Stone wrote: > > On Sun, Jul 6, 2014 at 11:46 AM, Steve Wills wrote: > > > I should have noted this system is running in bhyve. Also I'm told th= is panic > > > may be related to the fact that the system is running in bhyve. > > > > > > Looking at it a little more closely: > > > > > > (kgdb) list *__mtx_lock_sleep+0xb1 > > > 0xffffffff809638d1 is in __mtx_lock_sleep (/usr/src/sys/kern/kern_mut= ex.c:431). > > > 426 * owner stops running or the state of the lo= ck changes. > > > 427 */ > > > 428 v =3D m->mtx_lock; > > > 429 if (v !=3D MTX_UNOWNED) { > > > 430 owner =3D (struct thread *)(v & ~MTX_= FLAGMASK); > > > 431 if (TD_IS_RUNNING(owner)) { > > > 432 if (LOCK_LOG_TEST(&m->lock_ob= ject, 0)) > > > 433 CTR3(KTR_LOCK, > > > 434 "%s: spinning on = %p held by %p", > > > 435 __func__, m, owne= r); > > > (kgdb) > > > > > > I'm told that MTX_CONTESTED was set on the unlocked mtx and that MTX_= CONTENDED > > > is spuriously left behind, and to ask how lock prefix is handled in b= hyve. Any > > > of that make sense to anyone? > >=20 > > The mutex has both MTX_CONTESTED and MTX_UNOWNED set on it? That is a > > special sentinel value that is set on a mutex when it is destroyed > > (see MTX_DESTROYED in sys/mutex.h). If that is the case it looks like > > you've stumbled upon some kind of use-after-free in tmpfs. I doubt > > that bhyve is responsible (other than perhaps changing the timing > > around making the panic more likely to happen). >=20 > Given the first thing seen was: >=20 > Freed UMA keg (TMPFS node) was not empty (16 items). Lost 1 pages of mem= ory. >=20 > this sounds reasonable to me. >=20 > What can I do to help find and elliminate the source of the error? The most worrying fact there is that the mutex which is creating trouble cannot be anything other but allnode_lock, from the backtrace. For this mutex to be destroyed, the unmount of the corresponding mount point must run to completion. In particular, it must get past the vflush(9) call in tmpfs_unmount(). This call reclaims all vnodes belonging to the unmounted mount point. New vnodes cannot be instantiated meantime, since insmntque(9) is blocked by the MNTK_UNMOUNT flag. That said, the backtrace indicates that we have live vnode, which is reclaimed, and also we have the mutex which is in the destroyed (?) state. My basic claim is that the two events cannot co-exist, at least, this code path was heavily exercised and most issues were fixed during several years. I cannot exclude the possibility of tmpfs/VFS screwing things up, but given the above reasoning, and the fact that this is the first appearance of the MTX_DESTROED problem for the tmpfs unmounting code, which was not changed for long time, I would at least ask some things about bhyve. I.e., I would rather first look at the locked prefix emulation then at the tmpfs. --l118U0+vX1D/6gtA Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTuZGJAAoJEJDCuSvBvK1B1QUP/AlpvZasnrIqIIq2SK8EuVFs 9G4jIYeaYI8bDSFrzQe/RbZETI89YLRVTSOBvO2W4lazu8em4oLBFYw7U8Kdw3bn ITX14XRvCvRAb27Dg2PKo2tr2vN7sdlIZUdl4prBcs8ik9zQi1+kgiT1BF/snTAD CqRsLoVmKlocfEfZSINEA7Emm+9OM5rsF7ktCiPIMVl/NeoznvfVz7j+ilz5RAer Qeb3TWfBorbzZXO5nXFYxBaw6lgP/ZHqfdj0bGyaUQhhrNz/V02cmi1gITtZEKmR QBdbdATcV/Y6hdAZkEUwRrHAw7/Zl/JF+0Hk4mjru7Ri8cV1Ki4K4bZ4iEWgCIqn aDLIFjX1iTixmA03wDbX30Vo9gsX8m5qq4wYHHDLbttsO2LtclWiwpj1rqS7bjKS Un6I2KUS+kT5vJ/YPNgg1a0P340CbLQy4rz6akl+T+VS390VrDwOH+IiTMD5hoWd Dy+BRZn0gVxToHls89xu+Wc6u1KPK+0Iwngr+JktiCYlJggsEYjs+RsC0fRX3eC1 9mVYvVBo4Mjg8hFWP/U1xE+stJvYfzplj/JlUACtoV7gK90gHBllTL7Gqu1DoZjY g2ozS3yvosdeiVB8q9hXCtY03yZt+ddNFjd67oQ2k8/3O41zhoHPeqFWTMepL/id GlPcsrk94NTHNiVAxNqt =9jD2 -----END PGP SIGNATURE----- --l118U0+vX1D/6gtA--