From owner-freebsd-fs@FreeBSD.ORG Thu Dec 27 13:15:28 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B23FD16A417 for ; Thu, 27 Dec 2007 13:15:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id 4695A13C467 for ; Thu, 27 Dec 2007 13:15:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=deviant.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1J7sa6-000Jlz-J2 for freebsd-fs@freebsd.org; Thu, 27 Dec 2007 15:15:27 +0200 Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id lBRDFNj2026151; Thu, 27 Dec 2007 15:15:23 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id lBRDFM9A026150; Thu, 27 Dec 2007 15:15:22 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 27 Dec 2007 15:15:21 +0200 From: Kostik Belousov To: Raja Sivaramakrishnan Message-ID: <20071227131521.GO57756@deviant.kiev.zoral.com.ua> References: <701759.46468.qm@web90511.mail.mud.yahoo.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="5tiY7shzwSGI9p6W" Content-Disposition: inline In-Reply-To: <701759.46468.qm@web90511.mail.mud.yahoo.com> User-Agent: Mutt/1.4.2.3i X-Scanner-Signature: 6bd56f30920205a35a58b455a86ead09 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1966 [Dec 27 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {SMTP from is not routable} X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 19 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org Subject: Re: namei lookup vnode locking X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2007 13:15:28 -0000 --5tiY7shzwSGI9p6W Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Dec 26, 2007 at 04:46:48PM -0800, Raja Sivaramakrishnan wrote: > Hello, > I encountered an issue with FreeBSD 6.1 and would > appreciate some feedback on this. The problem happens First, there were the significant locking fixes for the devfs after 6.1, in particular, check that 6.2, or, even better 6.3-latest RC shows the errant behaviour. It may be all fixed already. For proper reporting of the deadlock, see kernel debug chapter of the developer handbook, in particular, deadlock section. I have a doubt regarding you analysis as far as I was able to understand it. ttydrain() ioctl does not hold the dev vnode lock while calling the driver. Anyway, do what I recommended above. > when a perl script running on some client system does=20 > a telnet into the FreeBSD box, exits from the login=20 > shell and immediately exits the perl script too. After > the script exits, there is a deadlock on the FreeBSD > box that prevents new processes (such as ps, top etc.) > from starting. Upon investigation, this seems to be > caused due to the following sequence of events on the > FreeBSD system. >=20 > 1) login process exits. exit call in the kernel closes > all file descriptors. One of these is the fd for > /dev/ttyp0, used for the telnet session. login locks > the vnode for /dev/ttyp0 and waits for 5 minutes in > order for the tty to drain (ttywait() call). >=20 > 2) The tty is supposed to be drained by telnetd.=20 > However, telnetd sees the network connection go=20 > down when the perl script exits. As a result, it > jumps to cleanup code, where it tries to do chmod > on /dev/ttyp0. chmod syscall attempts to lock > /dev/ttyp0, but fails as the lock is held by login, > which puts telnetd process to sleep. However,=20 > telnetd holds the lock on the vnode for /dev. > It appears that the lock was acquired when doing the > namei lookup for /dev/ttyp0. The current state is > that there is output in the tty that has to be > read by telnetd, but it can't because it is sleeping > for the /dev/ttyp0 lock. telnetd is holding the > /dev vnode lock while sleeping. >=20 > 3) As a result, any process that needs the /dev > vnode lock is put to sleep for 5 minutes (ttywait > waits for a default of 5 minutes). Even if a > process wants to open an unrelated device file, > /dev/foo, it is not able to do so because the /dev > lock is held by telnetd. >=20 > Few questions: >=20 > 1) Does namei lookup need to acquire an exclusive > lock on intermediate vnodes when looking up a pathname > i.e. if telnetd is trying to lookup /dev/ttyp0, does > it need to get an exclusive lock on /dev? Can it > be a shared lock that will allow at least other > readers > to make progress? >=20 > 2) Besides relaxing the locking above, any other > thoughts on how to fix this? Reducing the tty timeout > from the close routine is another option, but that > only limits the duration of the deadlock. >=20 > Thanks, >=20 > Raja >=20 >=20 > ___________________________________________________________________= _________________ > Never miss a thing. Make Yahoo your home page.=20 > http://www.yahoo.com/r/hs > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --5tiY7shzwSGI9p6W Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHc6VpC3+MBN1Mb4gRAhrBAJsFhdoxgeovIz5GlGNnUq+FrEXJUwCgpzpX 5pLJcQy/LtDqoVIk1Ce/9Jo= =NQA7 -----END PGP SIGNATURE----- --5tiY7shzwSGI9p6W--