Date: Wed, 20 Jun 2012 10:56:36 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Steven Haber <steven.haber@isilon.com> Cc: freebsd-geom@freebsd.org Subject: Re: Geom / destroy_dev() deadlock Message-ID: <20120620075635.GT2337@deviant.kiev.zoral.com.ua> In-Reply-To: <56CE86D6660FF84498426FA7A33170B403589417@seaxch01.desktop.isilon.com> References: <56CE86D6660FF84498426FA7A33170B4033672EF@seaxch01.desktop.isilon.com> <20120611204334.GH2337@deviant.kiev.zoral.com.ua> <56CE86D6660FF84498426FA7A33170B403367535@seaxch01.desktop.isilon.com> <20120611215610.GJ2337@deviant.kiev.zoral.com.ua> <56CE86D6660FF84498426FA7A33170B403429115@seaxch01.desktop.isilon.com> <20120611225304.GK2337@deviant.kiev.zoral.com.ua> <56CE86D6660FF84498426FA7A33170B403589417@seaxch01.desktop.isilon.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--l7itP/1EBO9PCc/O
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Tue, Jun 19, 2012 at 11:24:03AM -0700, Steven Haber wrote:
> > On Mon, Jun 11, 2012 at 03:27:39PM -0700, Steven Haber wrote:
> > > I do not understand what you are proposing. Could you, please, show
> > > > the patch ?
> > >=20
> > > ---
> > > src/sys/geom/geom_dev.c | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >=20
> > > diff --git a/src/sys/geom/geom_dev.c b/src/sys/geom/geom_dev.c
> > > index 38251e1..787235a 100644
> > > --- a/src/sys/geom/geom_dev.c
> > > +++ b/src/sys/geom/geom_dev.c
> > > @@ -497,7 +497,7 @@ g_dev_orphan(struct g_consumer *cp)
> > > =20
> > > /* Destroy the struct cdev *so we get no more requests */
> > > unit =3D dev2unit(dev);
> > > - destroy_dev(dev);
> > > + destroy_dev_sched(dev);
> > > free_unr(unithdr, unit);
> > > =20
> > > /* Wait for the cows to come home */
> >
> > From: Konstantin Belousov
> > Sent: Monday, June 11, 2012 3:53 PM
> > To: Steven Haber
> > Cc: freebsd-geom@freebsd.org
> > Subject: Re: Geom / destroy_dev() deadlock
> >
> > Did you noted the comment above the block you changing ?
> > The patch would cause races allowing arbitrary kernel memory
> corruption.
> >
> > The moment when the cdev is destroyed is somewhere in future, while
> > structures that the cdev reference are freed synchronously.
> >
> > I tried to put some safety measures into destroy_dev_sched(9), namely
> > CDP_SCHED_DTR flag that somewhat reduces the possibility of usermode
> > accessing cdev after destroy_dev_sched(), but this cannot be
> eliminated
> > entirely.
>=20
> So destroy_dev_sched() is inherently racey. That explains why there
> aren't many examples of usage in the kernel.
It is not. I admit (and did this quite often) that correct use
of destroy_dev_sched{,_cb}() is not trivial, but the KPI was designed
explicitely to handle otherwise unmanageable deadlocks and races.
>=20
> Without doing a scheduled destroy, can you think of any way to prevent
> the devdrn/geom deadlock? From the original discussion:
>=20
> GEOM calls destroy_dev() while holding the topology lock.
>=20
> Destroy_dev() wants to destroy device, but can't because there
> are
> threads that still have it open.
>=20
> The threads can't close it, because to close it they need the
> topology
> lock.
No, destroy_dev_sched (most likely destroy_dev_sched_cb) is the right
facility to defer destroy_dev. Usual technique is to set some flag
that causes devsw methods to not proceed if CDP_SCHED_DTR protection
is not enough for some reason, and do actual cleanup of resources in
callback.
>=20
> There may be additional locking that can be done in the dev layer, like
> a third sleepable lock. We could also set a devdrn flag on the cdev to
> cause g_dev calls to return with an error prior to taking the topology
> lock.
Well, if you can end up with working correct patch, feel free.
--l7itP/1EBO9PCc/O
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)
iEYEARECAAYFAk/hgjMACgkQC3+MBN1Mb4i5HQCg5ZDmxRmPFX7bb4oxfhLAX1ua
UZ0AniEKVfg9JYe57IED+3w36koMoRJT
=VaEY
-----END PGP SIGNATURE-----
--l7itP/1EBO9PCc/O--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120620075635.GT2337>