From owner-freebsd-hackers@FreeBSD.ORG Sat Jan 30 11:45:01 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4FC5106566B; Sat, 30 Jan 2010 11:45:01 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id DE63A8FC16; Sat, 30 Jan 2010 11:45:00 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 0B20145E9C; Sat, 30 Jan 2010 12:44:59 +0100 (CET) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id D5DA145CAC; Sat, 30 Jan 2010 12:44:53 +0100 (CET) Date: Sat, 30 Jan 2010 12:44:51 +0100 From: Pawel Jakub Dawidek To: Alexander Motin Message-ID: <20100130114451.GB1660@garage.freebsd.pl> References: <4B636812.8060403@FreeBSD.org> <20100130112749.GA1660@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="EuxKj2iCbKjpUGkD" Content-Disposition: inline In-Reply-To: <20100130112749.GA1660@garage.freebsd.pl> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-hackers@freebsd.org, FreeBSD-Current , kib@FreeBSD.org, freebsd-geom@freebsd.org Subject: Re: Deadlock between GEOM and devfs device destroy and process exit. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jan 2010 11:45:01 -0000 --EuxKj2iCbKjpUGkD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 30, 2010 at 12:27:49PM +0100, Pawel Jakub Dawidek wrote: > On Sat, Jan 30, 2010 at 12:58:26AM +0200, Alexander Motin wrote: > > Hi. > >=20 > > Experimenting with SATA hot-plug I've found quite repeatable deadlock > > case. Problem observed when several SATA devices, opened via devfs, > > disappear at exactly same time. In my case, at time of unplugging SATA > > Port Multiplier with several disks beyond it. All I have to do is to run > > several `dd if=3D/dev/adaX of=3D/dev/null bs=3D1m &` commands and unplug > > multiplier. That causes predictable I/O errors and devices destruction. > > But with high probability several dd processes getting stuck in kernel. > [...] >=20 > I observed the same thing yesterday while stress-testing HAST: >=20 > 3659 2504 3659 0 DE+ GEOM top 0x8079a348 dd > 3658 2102 2102 0 DE+ GEOM top 0x8079a348 hastd > 2 0 0 0 DL devdrn 0x85b1bc68 [g_event] >=20 > Both dd(1) and hastd(8) wait for the GEOM topology lock in the exit path, > which is already held by the g_event thread. Maybe I'll add how I understand what's going on: GEOM calls destroy_dev() while holding the topology lock. Destroy_dev() wants to destroy device, but can't because there are threads that still have it open. The threads can't close it, because to close it they need the topology lock. The deadlock is quite obvious, IMHO. I believe the problem could be solved by dropping the topology lock in g_dev_orphan() when calling destroy_dev(dev), but it is hard to say if it is safe to drop the topology lock there. Maybe Poul-Henning could take a look. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --EuxKj2iCbKjpUGkD Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFLZBuyForvXbEpPzQRAoaLAJ9X1IIhEfBcTNHc2CYBkh4RAzc/twCgj6x0 y1PsqIMgcFnE/ILC2kevD28= =hEg0 -----END PGP SIGNATURE----- --EuxKj2iCbKjpUGkD--