From owner-freebsd-geom@FreeBSD.ORG Wed Jun 20 07:56:44 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19795106564A for ; Wed, 20 Jun 2012 07:56:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id A99A48FC0C for ; Wed, 20 Jun 2012 07:56:43 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q5K7uaua034416; Wed, 20 Jun 2012 10:56:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q5K7uavb071440; Wed, 20 Jun 2012 10:56:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q5K7uaXg071439; Wed, 20 Jun 2012 10:56:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 20 Jun 2012 10:56:36 +0300 From: Konstantin Belousov To: Steven Haber Message-ID: <20120620075635.GT2337@deviant.kiev.zoral.com.ua> References: <56CE86D6660FF84498426FA7A33170B4033672EF@seaxch01.desktop.isilon.com> <20120611204334.GH2337@deviant.kiev.zoral.com.ua> <56CE86D6660FF84498426FA7A33170B403367535@seaxch01.desktop.isilon.com> <20120611215610.GJ2337@deviant.kiev.zoral.com.ua> <56CE86D6660FF84498426FA7A33170B403429115@seaxch01.desktop.isilon.com> <20120611225304.GK2337@deviant.kiev.zoral.com.ua> <56CE86D6660FF84498426FA7A33170B403589417@seaxch01.desktop.isilon.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="l7itP/1EBO9PCc/O" Content-Disposition: inline In-Reply-To: <56CE86D6660FF84498426FA7A33170B403589417@seaxch01.desktop.isilon.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-geom@freebsd.org Subject: Re: Geom / destroy_dev() deadlock X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jun 2012 07:56:44 -0000 --l7itP/1EBO9PCc/O Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 19, 2012 at 11:24:03AM -0700, Steven Haber wrote: > > On Mon, Jun 11, 2012 at 03:27:39PM -0700, Steven Haber wrote: > > > I do not understand what you are proposing. Could you, please, show > > > > the patch ? > > >=20 > > > --- > > > src/sys/geom/geom_dev.c | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > >=20 > > > diff --git a/src/sys/geom/geom_dev.c b/src/sys/geom/geom_dev.c > > > index 38251e1..787235a 100644 > > > --- a/src/sys/geom/geom_dev.c > > > +++ b/src/sys/geom/geom_dev.c > > > @@ -497,7 +497,7 @@ g_dev_orphan(struct g_consumer *cp) > > > =20 > > > /* Destroy the struct cdev *so we get no more requests */ > > > unit =3D dev2unit(dev); > > > - destroy_dev(dev); > > > + destroy_dev_sched(dev); > > > free_unr(unithdr, unit); > > > =20 > > > /* Wait for the cows to come home */ > > > > From: Konstantin Belousov > > Sent: Monday, June 11, 2012 3:53 PM > > To: Steven Haber > > Cc: freebsd-geom@freebsd.org > > Subject: Re: Geom / destroy_dev() deadlock > > > > Did you noted the comment above the block you changing ? > > The patch would cause races allowing arbitrary kernel memory > corruption. > > > > The moment when the cdev is destroyed is somewhere in future, while > > structures that the cdev reference are freed synchronously. > > > > I tried to put some safety measures into destroy_dev_sched(9), namely > > CDP_SCHED_DTR flag that somewhat reduces the possibility of usermode > > accessing cdev after destroy_dev_sched(), but this cannot be > eliminated > > entirely. >=20 > So destroy_dev_sched() is inherently racey. That explains why there > aren't many examples of usage in the kernel. It is not. I admit (and did this quite often) that correct use of destroy_dev_sched{,_cb}() is not trivial, but the KPI was designed explicitely to handle otherwise unmanageable deadlocks and races. >=20 > Without doing a scheduled destroy, can you think of any way to prevent > the devdrn/geom deadlock? From the original discussion: >=20 > GEOM calls destroy_dev() while holding the topology lock. >=20 > Destroy_dev() wants to destroy device, but can't because there > are > threads that still have it open. >=20 > The threads can't close it, because to close it they need the > topology > lock. No, destroy_dev_sched (most likely destroy_dev_sched_cb) is the right facility to defer destroy_dev. Usual technique is to set some flag that causes devsw methods to not proceed if CDP_SCHED_DTR protection is not enough for some reason, and do actual cleanup of resources in callback. >=20 > There may be additional locking that can be done in the dev layer, like > a third sleepable lock. We could also set a devdrn flag on the cdev to > cause g_dev calls to return with an error prior to taking the topology > lock. Well, if you can end up with working correct patch, feel free. --l7itP/1EBO9PCc/O Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk/hgjMACgkQC3+MBN1Mb4i5HQCg5ZDmxRmPFX7bb4oxfhLAX1ua UZ0AniEKVfg9JYe57IED+3w36koMoRJT =VaEY -----END PGP SIGNATURE----- --l7itP/1EBO9PCc/O--