From owner-freebsd-geom@FreeBSD.ORG Fri Jan 29 23:23:32 2010 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32C39106568D; Fri, 29 Jan 2010 23:23:32 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-fx0-f227.google.com (mail-fx0-f227.google.com [209.85.220.227]) by mx1.freebsd.org (Postfix) with ESMTP id 63F688FC13; Fri, 29 Jan 2010 23:23:31 +0000 (UTC) Received: by fxm27 with SMTP id 27so832917fxm.3 for ; Fri, 29 Jan 2010 15:23:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=LtEFmPGthK3fbozT5A2Sd4pzqLPAYGeXNOpggIFXl2I=; b=MUY3bZha8/l9HHbORTWYd2BY5netrT3cp8mlnJ9DzXa1zrCLS/WZbtvNxmnzgM/amI 1in7h8GkAMobFPtHDPB2LkuhcKcuM8ilgbQKofB3C00kSkt22oAbJMCPmSyG3D6BcbE/ G1Mc2QvKtz8Q9ESfQQipBlGwkWnV5aTKfnrw4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=JtZBYRduj2DfrG22ICVJ82hPr/6ebOuzGIS8PkRtaxQP5ZHAvlTD4UOcpH0KDajazt SeCL+3+IEzbxXtyH2Ew2riLWSE2PLmKpNtKEoJBJyDJUne65Bu5ldIRsRFBoFPoSjl19 gtTVO1gQtzy4W4ODEO0FI7FtGH442efU23ip0= Received: by 10.102.235.36 with SMTP id i36mr747418muh.56.1264807410244; Fri, 29 Jan 2010 15:23:30 -0800 (PST) Received: from mavbook.mavhome.dp.ua (pc.mavhome.dp.ua [212.86.226.226]) by mx.google.com with ESMTPS id j9sm11120931mue.6.2010.01.29.15.23.29 (version=SSLv3 cipher=RC4-MD5); Fri, 29 Jan 2010 15:23:29 -0800 (PST) Sender: Alexander Motin Message-ID: <4B636DEA.2060901@FreeBSD.org> Date: Sat, 30 Jan 2010 01:23:22 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.23 (X11/20091212) MIME-Version: 1.0 To: Kostik Belousov References: <4B636812.8060403@FreeBSD.org> <20100129231110.GS3877@deviant.kiev.zoral.com.ua> In-Reply-To: <20100129231110.GS3877@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, FreeBSD-Current , freebsd-geom@freebsd.org Subject: Re: Deadlock between GEOM and devfs device destroy and process exit. X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2010 23:23:32 -0000 Kostik Belousov wrote: > On Sat, Jan 30, 2010 at 12:58:26AM +0200, Alexander Motin wrote: >> Hi. >> >> Experimenting with SATA hot-plug I've found quite repeatable deadlock >> case. Problem observed when several SATA devices, opened via devfs, >> disappear at exactly same time. In my case, at time of unplugging SATA >> Port Multiplier with several disks beyond it. All I have to do is to run >> several `dd if=/dev/adaX of=/dev/null bs=1m &` commands and unplug >> multiplier. That causes predictable I/O errors and devices destruction. >> But with high probability several dd processes getting stuck in kernel. >> >> I've discovered such pieces of problem: >> - CAM receives disconnect event and starts device destruction. But as >> device is still opened, it can't do it immediately. >> - dd receives I/O error and exits. >> - exit1() call closes all descriptors, including adaX device. It >> triggers final device destruction, by sending event to geom_dev. >> >> adaclose(4571fa00,4,40c16576,76,0,...) at 0x4049c521 >> g_disk_access(457e2200,ffffffff,0,0,0,...) at 0x4080b9a4 >> g_access(45643d80,ffffffff,0,0,2000,...) at 0x40810ccb >> g_dev_close(45766500,1,2000,4569fd80,4569fd80,...) at 0x4080a425 >> devfs_close(7b604aa8,80000,457f8000,80000,7b604acc,...) at 0x407f2762 >> VOP_CLOSE_APV(40d03180,7b604aa8,40c2e681,128,0,...) at 0x40b6da55 >> vn_close(457f8000,1,45624300,4569fd80,451271e0,...) at 0x40912750 >> vn_closefile(4566da48,4569fd80,4566da48,0,7b604b58,...) at 0x40912854 >> devfs_close_f(4566da48,4569fd80,3,0,4566da48,...) at 0x407f235b >> _fdrop(4566da48,4569fd80,7b604b8c,408b5cec,0,4569fe24,40eb23a8,40d10460,40c1a8bb,4560672c,721,40c1a8b2,7b604bb4,40878220,4560672c,8,40c1a8b2,721) >> at 0x40836da3 >> closef(4566da48,4569fd80,721,71e,4569fe24,...) at 0x40838ad0 >> fdfree(4569fd80,0,40c1b1a9,107,7b604c80,...) at 0x408394da >> exit1(4569fd80,100,7b604d2c,40b565c0,4569fd80,...) at 0x40844423 >> sys_exit(4569fd80,7b604cf8,40c59d34,40c26be4,4569d2a8,...) at 0x408450fd >> syscall(7b604d38) at 0x40b565c0 >> >> - GEOM event thread tries to destroy /dev/adaX device (which should be >> already free at this moment), but for some reason freezes, waiting for >> device to be freed: >> >> 0 2 0 0 -8 0 0 8 devdrn DL ?? 0:02.89 >> [g_event] >> >> - as GEOM event is still not handled, exit1() waits for it: >> >> kdb_backtrace(40c16bc4,0,40c16ab1,56,4540e640,...) at 0x408a2909 >> g_waitidle(4569fd80,0,40c1b1a9,107,7b604c80,...) at 0x4080cd1f >> exit1(4569fd80,100,7b604d2c,40b565c0,4569fd80,...) at 0x40844431 >> sys_exit(4569fd80,7b604cf8,40c59d34,40c26be4,4569d2a8,...) at 0x408450fd >> syscall(7b604d38) at 0x40b565c0 >> >> - system stationary. GEOM frozen. No way to get out of this, except >> pushing reset. >> >> 0 1065 1055 0 44 0 5344 3040 g_wait DE 0 0:00.43 dd >> if=/dev/ada1 of=/dev/null bs=1m >> 0 1066 1055 0 44 0 5344 3040 GEOM t DE 0 0:00.07 dd >> if=/dev/ada2 of=/dev/null bs=1m >> >> >> So, does anybody have good idea why destroy_dev() can't complete? > > The devdrn state means that thread performing the device destruction, > i.e. the thread called destroy_dev(), is waiting for threads to leave > the cdevsw d_* methods. The thread that notified the destruction thread > did that from d_close() method. This resulted in the deadlock. d_close() doesn't call destroy_dev() directly. It schedules different thread to do it. destroy_dev() should run after d_close() already complete. Though I haven't checked how it is locked. > I introduced destroy_dev_sched(9) KPI to handle this and similar issues. > Note that race-free use of destroy_dev_sched(9) is quite hard. I think it should work without it here. Shouldn't it? -- Alexander Motin