From owner-freebsd-hardware@FreeBSD.ORG Mon Jul 24 06:58:05 2006 Return-Path: X-Original-To: freebsd-hardware@FreeBSD.org Delivered-To: freebsd-hardware@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1277116A4DD for ; Mon, 24 Jul 2006 06:58:05 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailout2.pacific.net.au (mailout2.pacific.net.au [61.8.0.115]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6A15643D4C for ; Mon, 24 Jul 2006 06:58:04 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy1.pacific.net.au (mailproxy1.pacific.net.au [61.8.2.162]) by mailout2.pacific.net.au (Postfix) with ESMTP id 71200109A2B; Mon, 24 Jul 2006 16:58:02 +1000 (EST) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailproxy1.pacific.net.au (8.13.4/8.13.4/Debian-3sarge1) with ESMTP id k6O6vvNm018897; Mon, 24 Jul 2006 16:57:59 +1000 Date: Mon, 24 Jul 2006 16:57:57 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Jo Rhett In-Reply-To: <20060721004731.GC8868@svcolo.com> Message-ID: <20060724154856.I58894@delplex.bde.org> References: <20060721000018.GA99237@svcolo.com> <20060721001607.GA64376@megan.kiwi-computer.com> <20060721004731.GC8868@svcolo.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "Rick C. Petty" , freebsd-hardware@FreeBSD.org Subject: Re: device busy -- no locks? X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jul 2006 06:58:05 -0000 On Thu, 20 Jul 2006, Jo Rhett wrote: > Thanks for the super-quick reply! Responses are inline... > > On Thu, Jul 20, 2006 at 07:16:07PM -0500, Rick C. Petty wrote: >>> root@scapa 47# fstat /dev/ttyd0 >>> USER CMD PID FD MOUNT INUM MODE SZ|DV R/W NAME >> >> What about "fstat /dev/cuad0" ? Anyway, I've found that fstat is useless, >> try using sysutils/lsof instead. This is related to a longstanding (design) bug in vfs (first-open/last-close semantics). vfs counts devices as being open when it calls the device open routine, despite devices not actually being open until the device open routine returns successfully, which may happen much later (or not at all in the case of failure, but this doesn't cause any additional problems). This causes the device close routine to not be called in some cases where it should be. For bidirectional serial devices, not calling the device close routine results in "callin" devices that are sleeping in open being treated the same as "callin" devices that have successfully completed the open. The former shouldn't give EBUSY for opens of the corresponding "callout" device, but the latter should and do. FreeBSD-1 has a hack to vfs to work around the bug, but the hack was lost in FreeBSD-2. I still use the hack locally. Startting with about FreeBSD-4, there is a D_TRACKCLOSE device flag that can be used to fix the problem less hackishly (but still not in the right way, since it requires individual drivers to do generic things). I haven't got around to using it to fix sio even locally. D_TRACKCLOSE is mostly unused and mostly used bogusly when it is used. The bug rarely causes problems since it is only activated by doing something like the following: thread1: "open" /dev/ttyd0. Actually, block in open waiting for carrier. thread2: open /dev/ttyd0 using O_NONBLOCK to prevent blocking. thread2: perhaps actually use /dev/ttyd0 thread2: "close" /dev/ttyd0. Actually, don't complete the close due to the bogus vfs close. thread1: remain blocked in open through all the above. thread3: try to open /dev/cuaa^Hd0. Get EBUSY because the non-open by thread1 is seen as an open. Starting with about FreeBSD-5, there may be additional problems from races. First-open/last-close semantics basically require opens to be synchronous. Sleeping in open for serial device drivers gives large race windows in which to open may race open/close of the same device in other threads. Prempting the kernel gives small race windows. In practice, Giant locking limits problems. For serial drivers, open/close should still be Giant- locked since the whole tty subsystem is still Giant-locked. (Note that all vfs locks are dropped before calling device open/close. The bogus vfs count provides some psuedo-locking.) Rearrangement of serial drivers in -current may have enlarged the bug, but I can't see any enlargement except that from more serial drivers now supporting bidirectional devices. > Sorry, yes. Same results. And if lsof shows things that fstat doesn't, > then this is a bug in FreeBSD. > > But anyway, > root@scapa 63# lsof /dev/cuad0 > root@scapa 64# lsof /dev/ttyd0 > > Nada. I think fstat and lsof can't see threads sleeping in open since the open hasn't really completed -- the open has completed enough to confuse vfs but not for vfs to report its confusion to userland. It should be possible to see threads sleeping in open using "ps -lax | grep ttydcd" ("ttydcd" is the string for -current; the string for sio used to be "siodcd". Grep for "tty" and "dcd" too). This won't distinguish between threads sleeping normally in open (ttyopen) and ones that are in a bogus state due to a missing close. > Also note that this system is pretty bone stock. Standard install, plus > mysql and apache. Nothing else would be using the port. It's something > that left it locked, and really only "login" could be the culprit. Quite likely, but login doesn't use O_NONBLOCK so I don't know how it could trigger the bug. Maybe nopise on DCD cound do it. The easiest way to trigger the bug is "stty -f /dev/ttyd0" while there is a login blocked in open on ttyd0. >>> No locks? No processes using it. Okay, this is uncool. >>> And yet "ktrace tip com1" and "kdump -f ktrace.out" clearly show: >>> >>> 50461 tip CALL open(0x8059030,0x6,0) >>> 50461 tip NAMI "/dev/cuad0" >>> 50461 tip RET open -1 errno 16 Device busy >> >> This isn't very useful. A ktrace on the process that's locking the file >> would be. :-P > > See above. I can't find it. :-( You might need to start ktracing very early to locate the original problematic open/open/close sequence. >>> NOTE: at this time I am suspecting that CD is being misread (it's not >>> present - I have a break out box on the line) and that this problem is >>> somehow tied to that. This problem appears at random after login has >>> exerted itself on the system. I've disabled the getty on ttyd0 and login >>> has timed out, but it continues to show "device busy". >> >> How did you disable the getty? Was this prior to or after a restart? It >> sounds like /etc/ttys is maybe running a process on it. You need to >> "killall -HUP init" after changing /etc/ttys. But you probably already >> know that. > > Yes, I change "on" to "off" in /etc/ttys and "kill -1 1" :-) Killing all processes sleeping in serial device open unwedges the port for the bug that I know about (provided the close doesn't hang). This and making the open succeed by raising DCD in hardware are the only ways that I know of to unwedge the port once the open gets stuck. Bruce