From owner-freebsd-geom@FreeBSD.ORG Mon Jun 11 17:22:17 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 52BA3106566B for ; Mon, 11 Jun 2012 17:22:17 +0000 (UTC) (envelope-from steven.haber@isilon.com) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by mx1.freebsd.org (Postfix) with ESMTP id 0897B8FC15 for ; Mon, 11 Jun 2012 17:22:16 +0000 (UTC) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5BHJX7W003081 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 11 Jun 2012 13:19:33 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor) for ; Mon, 11 Jun 2012 13:19:24 -0400 Received: from seacasht01.desktop.isilon.com (seacasht01.isilon.com [137.69.159.79]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5BHJLLq004871 for ; Mon, 11 Jun 2012 13:19:22 -0400 Received: from seaxch01.isilon.com (137.69.158.60) by SEACASHT01.desktop.isilon.com (137.69.159.79) with Microsoft SMTP Server id 14.2.298.4; Mon, 11 Jun 2012 10:19:20 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Mon, 11 Jun 2012 10:19:07 -0700 Message-ID: <56CE86D6660FF84498426FA7A33170B4033672EF@seaxch01.desktop.isilon.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Geom / destroy_dev() deadlock Thread-Index: Ac1H9k6wBIuqsGqISD6SyAxkH7DByQ== From: Steven Haber To: X-EMM-MHVC: 1 Subject: Geom / destroy_dev() deadlock X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2012 17:22:17 -0000 Hey FreeBSD devs, I hope this is the right forum for this. I haven't used the FreeBSD mailing lists before. I'm a relatively new hire at EMC Isilon. We are using FreeBSD 7.0 as the basis for our scale-out NAS product line. We are frequently hitting the deadlock described here: http://lists.freebsd.org/pipermail/freebsd-geom/2010-February/003888.htm l The race is that destroy_dev() sleeps indefinitely waiting for thread references to drop from a dev. In the case of geom, we hold the topology lock the whole time we're in the dev layer. In devfs_open() and devfs_close(), we take a ref on the dev before calling into to geom. The proposed solution of destroy_dev_sched() makes sense to me. I am trying to understand the necessity of Alexander Motin's additional changes. destroy_dev_tq() holds the devmtx during the destroy and devfs uses this lock to take refs before calling into geom. To me it seems we should be able to protect the cdev with just the devmtx, provided consumers of d_close(), d_open(), etc. ref and rel the dev appropriately. Steven Haber Software Engineer, EMC Isilon (206) 753-1526