From owner-freebsd-stable@FreeBSD.ORG Sat Aug 24 17:41:13 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 807C37A0 for ; Sat, 24 Aug 2013 17:41:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1B3BA2A77 for ; Sat, 24 Aug 2013 17:41:12 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r7OHf3g5095969; Sat, 24 Aug 2013 20:41:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r7OHf3g5095969 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r7OHf31r095967; Sat, 24 Aug 2013 20:41:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 24 Aug 2013 20:41:03 +0300 From: Konstantin Belousov To: J David Subject: Re: NFS deadlock on 9.2-Beta1 Message-ID: <20130824174103.GQ4972@kib.kiev.ua> References: <158996730.12825374.1377265437471.JavaMail.root@uoguelph.ca> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="g1mh5Fh3W6jKnNId" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Rick Macklem , freebsd-stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Aug 2013 17:41:13 -0000 --g1mh5Fh3W6jKnNId Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Aug 24, 2013 at 01:08:05PM -0400, J David wrote: > The requested information about the deadlock was finally obtained and > provided off-list to the requested parties due to size. Thank you, the problem is clear now. The problematic process backtrace is Tracing command httpd pid 86383 tid 100138 td 0xfffffe000b7b2900 sched_switch() at sched_switch+0x234/frame 0xffffff834c442360 mi_switch() at mi_switch+0x15c/frame 0xffffff834c4423a0 sleepq_switch() at sleepq_switch+0x17d/frame 0xffffff834c4423e0 sleepq_wait() at sleepq_wait+0x43/frame 0xffffff834c442410 sleeplk() at sleeplk+0x11a/frame 0xffffff834c442460 __lockmgr_args() at __lockmgr_args+0x9a9/frame 0xffffff834c442580 nfs_lock1() at nfs_lock1+0x87/frame 0xffffff834c4425b0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff834c4425e0 _vn_lock() at _vn_lock+0x63/frame 0xffffff834c442640 ncl_upgrade_vnlock() at ncl_upgrade_vnlock+0x5e/frame 0xffffff834c442670 ncl_bioread() at ncl_bioread+0x195/frame 0xffffff834c4427e0 VOP_READ_APV() at VOP_READ_APV+0xd1/frame 0xffffff834c442810 vn_rdwr() at vn_rdwr+0x2bc/frame 0xffffff834c4428d0 kern_sendfile() at kern_sendfile+0xa90/frame 0xffffff834c442ac0 do_sendfile() at do_sendfile+0x92/frame 0xffffff834c442b20 amd64_syscall() at amd64_syscall+0x259/frame 0xffffff834c442c30 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff834c442c30 --- syscall (393, FreeBSD ELF64, sys_sendfile), rip =3D 0x801b24f4c, rsp = =3D 0x7fffffffce98, rbp =3D 0x7fffffffd1d0 --- It tries to do the upgrade of the nfs vnode lock, and for this, the lock is dropped and re-acquired. Since this happens with the vnode vm object' page busied, we get a reversal between vnode lock and page busy state. So effectively, my suspicion that NFS read path drops vnode lock was true, and in fact I knew about the upgrade. I think the easiest route is to a partial merge of the r253927 from HEAD. Index: fs =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- fs (revision 254800) +++ fs (working copy) Property changes on: fs ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/fs:r253927 Index: kern/uipc_syscalls.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- kern/uipc_syscalls.c (revision 254800) +++ kern/uipc_syscalls.c (working copy) @@ -2124,11 +2124,6 @@ else { ssize_t resid; =20 - /* - * Ensure that our page is still around - * when the I/O completes. - */ - vm_page_io_start(pg); VM_OBJECT_UNLOCK(obj); =20 /* @@ -2144,10 +2139,8 @@ IO_VMIO | ((MAXBSIZE / bsize) << IO_SEQSHIFT), td->td_ucred, NOCRED, &resid, td); VFS_UNLOCK_GIANT(vfslocked); - VM_OBJECT_LOCK(obj); - vm_page_io_finish(pg); - if (!error) - VM_OBJECT_UNLOCK(obj); + if (error) + VM_OBJECT_LOCK(obj); mbstat.sf_iocnt++; } if (error) { Index: . =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- . (revision 254800) +++ . (working copy) Property changes on: . ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys:r253927 --g1mh5Fh3W6jKnNId Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.21 (FreeBSD) iQIcBAEBAgAGBQJSGPAuAAoJEJDCuSvBvK1BnbgP/RDnR0CDCza8/lpHlqKrnIwD cpVk+J+aJdUznwAGt+wuhQld2BaTXwnaQjy1fCQzjJ9kVjrTqqPeqD0DU9eD6i+V 6aCizKb3EB8BLjK6v5qzkdfficr/ojFX7a7x6pDEP7srOoNmoy+m/ZPy45HVzDbJ YYXuMPW93bC927Tmbds8qdifhe3BzNXKIHH9NVpU1m4JsuhwR3reFjd4kW2B4FJk +oMeKdASP2CYUzmniNdVqq55Q6RXc/SXZefcxVLTYwgC0TU456TlXr5dOLZCL+gO wIBfBHi8/e0L0J/2xKe+5LT8F11jWupWoHKHWuyAFf77N7fMoeUeB9quiu6BaUaT fZ/Pnvu1xzH2bQoeO0ncscF1xK13f9lz0dFKPnynGxKRdcwcQDACHKI8SJLJIN94 leuCV1EZRJdT8ZFqBUlyejagIKu2CnAYd96jLuloqytqksnBQxym2VPAuz2oNVZJ m5b67brz7iUcHS1FH81htvAic4tSHcnboPo3uWjbZ1kaHyH7KLyuihPsva8pS0hx GYBc+UrKVgW70r1/JacX0tNhlN6LKPgMNWIy268z8CBGjpU4Z5oY8RB5aArzwETv ku+klJqbRj3IOByJZi5Vi3zM0galWrnInLlfOnebNqeiRUft2I7sbdvAsVALKZaS kgOf13FX7LxmW5FV3nKB =KIxM -----END PGP SIGNATURE----- --g1mh5Fh3W6jKnNId--