From owner-freebsd-hackers@FreeBSD.ORG Sat Jan 30 19:34:08 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C7A64106568D; Sat, 30 Jan 2010 19:34:08 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id 976E88FC12; Sat, 30 Jan 2010 19:34:07 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o0UJY32S077289 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 30 Jan 2010 21:34:03 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3) with ESMTP id o0UJY2js006692; Sat, 30 Jan 2010 21:34:02 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3/Submit) id o0UJY2pt006691; Sat, 30 Jan 2010 21:34:02 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 30 Jan 2010 21:34:02 +0200 From: Kostik Belousov To: Alexander Motin Message-ID: <20100130193402.GB3877@deviant.kiev.zoral.com.ua> References: <4B636812.8060403@FreeBSD.org> <20100130112749.GA1660@garage.freebsd.pl> <20100130114451.GB1660@garage.freebsd.pl> <4B647FAF.4090409@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Vu7hzOi38yxTgbOc" Content-Disposition: inline In-Reply-To: <4B647FAF.4090409@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-hackers@freebsd.org, FreeBSD-Current , Pawel Jakub Dawidek , freebsd-geom@freebsd.org Subject: Re: Deadlock between GEOM and devfs device destroy and process exit. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jan 2010 19:34:08 -0000 --Vu7hzOi38yxTgbOc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 30, 2010 at 08:51:27PM +0200, Alexander Motin wrote: > Pawel Jakub Dawidek wrote: > > On Sat, Jan 30, 2010 at 12:27:49PM +0100, Pawel Jakub Dawidek wrote: > >> On Sat, Jan 30, 2010 at 12:58:26AM +0200, Alexander Motin wrote: > >>> Experimenting with SATA hot-plug I've found quite repeatable deadlock > >>> case. Problem observed when several SATA devices, opened via devfs, > >>> disappear at exactly same time. In my case, at time of unplugging SATA > >>> Port Multiplier with several disks beyond it. All I have to do is to = run > >>> several `dd if=3D/dev/adaX of=3D/dev/null bs=3D1m &` commands and unp= lug > >>> multiplier. That causes predictable I/O errors and devices destructio= n. > >>> But with high probability several dd processes getting stuck in kerne= l. > >> [...] > >> > >> I observed the same thing yesterday while stress-testing HAST: > >> > >> 3659 2504 3659 0 DE+ GEOM top 0x8079a348 dd > >> 3658 2102 2102 0 DE+ GEOM top 0x8079a348 hastd > >> 2 0 0 0 DL devdrn 0x85b1bc68 [g_event] > >> > >> Both dd(1) and hastd(8) wait for the GEOM topology lock in the exit pa= th, > >> which is already held by the g_event thread. > >=20 > > Maybe I'll add how I understand what's going on: > >=20 > > GEOM calls destroy_dev() while holding the topology lock. > >=20 > > Destroy_dev() wants to destroy device, but can't because there are > > threads that still have it open. > >=20 > > The threads can't close it, because to close it they need the topology > > lock. > >=20 > > The deadlock is quite obvious, IMHO. >=20 > You are right, but as it happens not every time I was interested why. > After closer look I found two different scenarios. >=20 > In first case application receives I/O error and closes device. On > device close CAM calls disk_destroy(), which schedules device > destruction. When destroy_dev() called, device already free and there is > no problem, as these events are always asynchronous. >=20 > In second case, application also receives I/O error, but before it is > able to react, GEOM starts handling of disk_gone(), called by CAM. As > result, destroy_dev() called with device still opened, and it can't ever > be closed due to topology lock held. >=20 > I've played a bit with destroy_dev_sched(), but locking indeed looks not > to be easy. Is there some known good practice? destroy_dev_sched_cb() > looks a bit more promising. What do you mean by not easy locking ? destroy_dev_sched(dev) =3D=3D destroy_dev_sched_cb(dev, NULL, NULL). There is even a man page describing the interface. Main issue with destroy_dev_sched is the window between a moment when device is scheduled for destruction and thus kept in half-demolished state, and actual removal of devfs node. My exemplary case has been snp(4) before tty got rewritten, see r. 1.107 of sys/dev/snp/snp.c. No calls to destroy_dev_sched() that I placed in the src/ a kept around, that is good because corresponding subsystems got serious rewrite. --Vu7hzOi38yxTgbOc Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAktkiaoACgkQC3+MBN1Mb4igcACeI1FTL2MKQZW5g92KEk1V6PJD CsEAoKaG2t3br7mDNjSSVcfGA9zA0Khp =rl8T -----END PGP SIGNATURE----- --Vu7hzOi38yxTgbOc--