From owner-freebsd-current@FreeBSD.ORG Sat Jan 30 18:51:33 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B4BF106568B; Sat, 30 Jan 2010 18:51:33 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.153]) by mx1.freebsd.org (Postfix) with ESMTP id DA62F8FC15; Sat, 30 Jan 2010 18:51:31 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id e21so473520fga.13 for ; Sat, 30 Jan 2010 10:51:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=dEXZ1Vj0P9wqQ2ZosrktV+LFjbF33lwlRYQktMttHto=; b=B562ljXk9MSAlrEc3CcjbqRtts1ajmsJoWmCi7LTVICuHDBZBmZ+cZJMK/s0loPN2p 3bb3eLGKfa3AI0Y5rNaD0uLyeADXwG2033Hs3CCqpXrM1n0P3AUCEfljKaI2Y8YbHERt hlY3g2RXvFxGO+3EokysMnYY2Ia9nGugm5ywk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=TOyD6AFwyTsGKverlDiI6053vcBcTXxzXBoZfm7CHWvtasDeCGJUFe9AisXO097Z0o oIieYmLjqgJc+79tNGmf7dmPfi5O3sNoX+/7O1GgM5mFPXfAUc07q56AgWbnoxbffUaa uNlDUTmS5uI7Al5T1naqFsozGZ9ugxO+hKWz4= Received: by 10.103.35.5 with SMTP id n5mr1155025muj.132.1264877490919; Sat, 30 Jan 2010 10:51:30 -0800 (PST) Received: from mavbook.mavhome.dp.ua (pc.mavhome.dp.ua [212.86.226.226]) by mx.google.com with ESMTPS id 23sm3341149mun.41.2010.01.30.10.51.29 (version=SSLv3 cipher=RC4-MD5); Sat, 30 Jan 2010 10:51:30 -0800 (PST) Sender: Alexander Motin Message-ID: <4B647FAF.4090409@FreeBSD.org> Date: Sat, 30 Jan 2010 20:51:27 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.23 (X11/20091212) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4B636812.8060403@FreeBSD.org> <20100130112749.GA1660@garage.freebsd.pl> <20100130114451.GB1660@garage.freebsd.pl> In-Reply-To: <20100130114451.GB1660@garage.freebsd.pl> X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, FreeBSD-Current , kib@FreeBSD.org, freebsd-geom@freebsd.org Subject: Re: Deadlock between GEOM and devfs device destroy and process exit. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jan 2010 18:51:33 -0000 Pawel Jakub Dawidek wrote: > On Sat, Jan 30, 2010 at 12:27:49PM +0100, Pawel Jakub Dawidek wrote: >> On Sat, Jan 30, 2010 at 12:58:26AM +0200, Alexander Motin wrote: >>> Experimenting with SATA hot-plug I've found quite repeatable deadlock >>> case. Problem observed when several SATA devices, opened via devfs, >>> disappear at exactly same time. In my case, at time of unplugging SATA >>> Port Multiplier with several disks beyond it. All I have to do is to run >>> several `dd if=/dev/adaX of=/dev/null bs=1m &` commands and unplug >>> multiplier. That causes predictable I/O errors and devices destruction. >>> But with high probability several dd processes getting stuck in kernel. >> [...] >> >> I observed the same thing yesterday while stress-testing HAST: >> >> 3659 2504 3659 0 DE+ GEOM top 0x8079a348 dd >> 3658 2102 2102 0 DE+ GEOM top 0x8079a348 hastd >> 2 0 0 0 DL devdrn 0x85b1bc68 [g_event] >> >> Both dd(1) and hastd(8) wait for the GEOM topology lock in the exit path, >> which is already held by the g_event thread. > > Maybe I'll add how I understand what's going on: > > GEOM calls destroy_dev() while holding the topology lock. > > Destroy_dev() wants to destroy device, but can't because there are > threads that still have it open. > > The threads can't close it, because to close it they need the topology > lock. > > The deadlock is quite obvious, IMHO. You are right, but as it happens not every time I was interested why. After closer look I found two different scenarios. In first case application receives I/O error and closes device. On device close CAM calls disk_destroy(), which schedules device destruction. When destroy_dev() called, device already free and there is no problem, as these events are always asynchronous. In second case, application also receives I/O error, but before it is able to react, GEOM starts handling of disk_gone(), called by CAM. As result, destroy_dev() called with device still opened, and it can't ever be closed due to topology lock held. I've played a bit with destroy_dev_sched(), but locking indeed looks not to be easy. Is there some known good practice? destroy_dev_sched_cb() looks a bit more promising. -- Alexander Motin