From owner-freebsd-bugs@FreeBSD.ORG Tue May 23 07:05:16 2006 Return-Path: X-Original-To: freebsd-bugs@freebsd.org Delivered-To: freebsd-bugs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0B50D16A421; Tue, 23 May 2006 07:05:16 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailout1.pacific.net.au (mailout1.pacific.net.au [61.8.0.84]) by mx1.FreeBSD.org (Postfix) with ESMTP id 63B0143D45; Tue, 23 May 2006 07:05:15 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy2.pacific.net.au (mailproxy2.pacific.net.au [61.8.0.87]) by mailout1.pacific.net.au (Postfix) with ESMTP id EC02A329FC0; Tue, 23 May 2006 17:05:13 +1000 (EST) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailproxy2.pacific.net.au (8.13.4/8.13.4/Debian-3sarge1) with ESMTP id k4N75Atk013868; Tue, 23 May 2006 17:05:11 +1000 Date: Tue, 23 May 2006 17:05:10 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Miles Nordin In-Reply-To: Message-ID: <20060523142938.E2445@delplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-bugs@freebsd.org, FreeBSD-gnats-submit@freebsd.org Subject: Re: kern/97665: hang in sio driver X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 May 2006 07:05:16 -0000 On Mon, 22 May 2006, Miles Nordin wrote: >> Description: >> How-To-Repeat: > freebsd's serial driver seems to hang up a lot. processes get stuck > in uninterruptible sleep (don't respond to 'kill -9'), and i can > release them by, say, power-cycling a modem. > > try this: > > first, get a serial device that holds CTS low. ^ external (?) > > # stty crtscts < /dev/ttyd0.lock Don't normally do this. It doesn't seem to be necessary to demonstrate the bug here, and tends to break programs that actually understand crtscts. E.g., locking crtscts on for a serious mouse breaks the X11 mouse driver turning off crtscts, and then the port hangs in a way related to this PR. > # stty crtscts < /dev/ttyd0.init This is necessary to demonstrate the bug here, since support for setting crtscts in the program was lost when cu was broken by replacing Taylor cu (part of uucp) by native cu (part of tip) many years ago. I forget if Taylor cu clobbered the system (initial) default crtscts setting. > # cu -l ttyd0 -s 9600 Another thing lost in cu is defaulting to using the system default line speed. cu now defaults to its hard-coded default speed of 9600. This can be fixed by locking the system default speed. > Connected. > asdfasd;ljk;bouns;douahf > ~. > [EOT] > ^T > load: 0.00 cmd: cu 42104 [ttywai] 0.00u 0.03s 0% 752k > > now, open another window, and try 'kill -9 42104'. doesn't work. Did you connect it to an external device that holds CTS low (or just to nothing) so that it blocks waiting for CTS, and then on "~." it blocks in exit waiting for the output to drain? In this setup, the write() happens to complete since it is small (it goes to driver buffers, and write() somewhat bogusly returns success although the output hasn't actually all gone out (and with CTS blocking it, _none_ has gone out)). This makes it possible for cu to read the "~." and clean up and exit. Like most programs, cu has low-quality cleanup before exit. If it actually cared about output going out, it would use one of the following methods: A: for serial devices, use tcdrain(). Maybe use a timeout and tcflush() to avoid endless waits. B: for general devices, close() the device and actually check the return status. This provides less control. For tty devices, close() essentially does tcdrain() in the kernel, but interrupting this doesn't work quite right (it causes the output to be flushed and no error to be returned by close()). Programs that don't care about their output going out use the following method: C: just exit(). This pushes the close() to the kernel. It has all the disadvantages of method (B), plus the following: C1: there is no way to act on the result of close(). C2: there is no way to send a signal to an exiting process, or at least no way for one to affect close(). (C) and (C2) cause the symptoms reported in the above part of this PR. (C2) is the only bug here. It has nothing to do with sio or even tty drivers generally. All tty drivers are required to wait for output to drain in close(), and this is handled in the tty layer (function ttywait()). To work around bug (C2) and also endless waits in tcdrain(), use the drainwait ioctl (e.g., comcontrol /dev/ttyd0 drainwait ). This defaults to 180 seconds, so most hangs on [ttywai] aren't actually endless (they just seem to be). I normally use the default, but change to 1 second when something hangs (which happens fairly often since I have lots of unconnected ports with crtscts initially on and sometimes forget which ports are connected). Note that setting a too-small drainwait affects tcdrain() and thus may break normal output. tcdrain() seems to do the right thing (it returns an error after the timeout, and presumably doesn't flush the output), but programs wouldn't expect tcdrain() to fail due to a timeout. > now for real fun try this: > > # ls -l /dev/cuad0 > > provided you type that command for the first time while the > serial port is hung, you will hang devfs which will pretty soon hang > the whole goddamned machine. once cuad0 node is instantiated, that > vulnerability no longer exists. Devfs has lots of bugs, but I don't work on versions of FreeBSD that have it and can only guess its bug here. Apparently it does something like an open() on /dev/cuad0. open() has side effects for all devices so ls shouldn't go anywhere near it. The side effects are particularly large for serial ports. open()s of cuad0 must block waiting for ttyd0 to go away. Even reopens of ttyd0 may take arbitrarily long -- they wait for DTR to be held on long enough, and you can make the wait arbitrarily long using the dtrwait ioctl (comcontrol /dev/ttyd0 dtrwait ). The default for dtrwait is 3 seconds so waiting for it is normally not as noticeable as waiting for drainwait. > after some very long timeout on the order of minutes, the system may > recover itself. Always after 3 minutes? > Fix: > IMHO, a process should always respond to 'kill -9' no matter _what_ SIO > is doing, waiting for carrier, with data in the output buffer waiting > for CTS to assert itself, whatever, period. I shouldn't have the process > table cluttered with anything that can be removed only by changing the logic > state of some serial port pin. serial is not a SCSI port---it's highly > public. It's no more public. Both are controlled by device permissions and accesses to control ports are not normally granted to everyone. Users of serial ports can only set crtscts if crtscts is not locked off, and the sysadmin can lock it off for hardwware that doesn't support it. The sysadmin can also set drainwait to limit the effects of this bug when it is permitted to occur. > definitely sio activity should not be able to hang devfs. A system-wide hang is much more serious. Other bugs near this area: - the vfs layer doesn't count devices sleeping in open() properly, so the count of activity on tty devices vs the cua devices can get messed up and it isn't possible to open devices in one of these classes until all the others reach close(). This was fixed in FreeBSD-1, but the fix was lost in FreeBSD-2. - it is possible for open()/close() to (double)cross close(). This is necessary for reducing drainwait to be possible (since an open() is required to do an ioctl()), but it causes races that aren't handled properly (ones which seem to be harmful in theory but harmless in practice). sioopen() is or was very careful about concurrent opens, but sioclose() isn't so careful. It basically assumes that concurrent closes aren't possible (last-close semantics are supposed to prevent concurrent closes). However, when close() sleeps, as it often does for draining serial devices, the following can occur: 1. close() sleeps in thread 1 2. open() completes in thread 2. The state changes for this may mess up the state for thread 1 (but I think there are only problems with the hardware state). 3. ioctl() and even i/o in thread 2. Input might even work, but output would block for the same reason as thread 1, unless thread 1 races thread 2 and completes while thread 2 is active (then the completion may change the state of the hardware and break i/o in thread 1). 4. close() completes in thread 2. It doesn't normally block, but I'm not sure how it manages this since the tty struct is common. State changes in this should prevent the output in thread 1 from completing in the normal way. Note that the device close() is only reachable due to the poor open/close counting in the vfs layer. Last-close semantics is supposed to prevent multiple devices in close(), but the counts are decremented before calling close(); thus there can be any number of threads sleeping in close() and 1 thread at a time clobbering the state for the sleeping threads. The fix would involve backing out of close() if the device was reopened while we were sleeping in close() and is still open, and not completing close() if another thread is already sleeping in close() (i.e., change some last-closes into non-last ones and vice versa, and count things better so that this is possible). Bruce