Date: Sat, 24 Aug 2013 20:41:03 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: J David <j.david.lists@gmail.com> Cc: Rick Macklem <rmacklem@uoguelph.ca>, freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: NFS deadlock on 9.2-Beta1 Message-ID: <20130824174103.GQ4972@kib.kiev.ua> In-Reply-To: <CABXB=RTYZRs2okKu%2B3eZA1twNx%2B2J849Efo-8t=8ZSO3K9sRHw@mail.gmail.com> References: <CABXB=RSGrpUr34EdDu_6kab4nDm4NmYAoJZR8MPJOitpVzSNrA@mail.gmail.com> <158996730.12825374.1377265437471.JavaMail.root@uoguelph.ca> <CABXB=RTYZRs2okKu%2B3eZA1twNx%2B2J849Efo-8t=8ZSO3K9sRHw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
On Sat, Aug 24, 2013 at 01:08:05PM -0400, J David wrote:
> The requested information about the deadlock was finally obtained and
> provided off-list to the requested parties due to size.
Thank you, the problem is clear now.
The problematic process backtrace is
Tracing command httpd pid 86383 tid 100138 td 0xfffffe000b7b2900
sched_switch() at sched_switch+0x234/frame 0xffffff834c442360
mi_switch() at mi_switch+0x15c/frame 0xffffff834c4423a0
sleepq_switch() at sleepq_switch+0x17d/frame 0xffffff834c4423e0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff834c442410
sleeplk() at sleeplk+0x11a/frame 0xffffff834c442460
__lockmgr_args() at __lockmgr_args+0x9a9/frame 0xffffff834c442580
nfs_lock1() at nfs_lock1+0x87/frame 0xffffff834c4425b0
VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff834c4425e0
_vn_lock() at _vn_lock+0x63/frame 0xffffff834c442640
ncl_upgrade_vnlock() at ncl_upgrade_vnlock+0x5e/frame 0xffffff834c442670
ncl_bioread() at ncl_bioread+0x195/frame 0xffffff834c4427e0
VOP_READ_APV() at VOP_READ_APV+0xd1/frame 0xffffff834c442810
vn_rdwr() at vn_rdwr+0x2bc/frame 0xffffff834c4428d0
kern_sendfile() at kern_sendfile+0xa90/frame 0xffffff834c442ac0
do_sendfile() at do_sendfile+0x92/frame 0xffffff834c442b20
amd64_syscall() at amd64_syscall+0x259/frame 0xffffff834c442c30
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff834c442c30
--- syscall (393, FreeBSD ELF64, sys_sendfile), rip = 0x801b24f4c, rsp = 0x7fffffffce98, rbp = 0x7fffffffd1d0 ---
It tries to do the upgrade of the nfs vnode lock, and for this, the lock
is dropped and re-acquired. Since this happens with the vnode vm object'
page busied, we get a reversal between vnode lock and page busy state.
So effectively, my suspicion that NFS read path drops vnode lock was
true, and in fact I knew about the upgrade.
I think the easiest route is to a partial merge of the r253927 from HEAD.
Index: fs
===================================================================
--- fs (revision 254800)
+++ fs (working copy)
Property changes on: fs
___________________________________________________________________
Modified: svn:mergeinfo
Merged /head/sys/fs:r253927
Index: kern/uipc_syscalls.c
===================================================================
--- kern/uipc_syscalls.c (revision 254800)
+++ kern/uipc_syscalls.c (working copy)
@@ -2124,11 +2124,6 @@
else {
ssize_t resid;
- /*
- * Ensure that our page is still around
- * when the I/O completes.
- */
- vm_page_io_start(pg);
VM_OBJECT_UNLOCK(obj);
/*
@@ -2144,10 +2139,8 @@
IO_VMIO | ((MAXBSIZE / bsize) << IO_SEQSHIFT),
td->td_ucred, NOCRED, &resid, td);
VFS_UNLOCK_GIANT(vfslocked);
- VM_OBJECT_LOCK(obj);
- vm_page_io_finish(pg);
- if (!error)
- VM_OBJECT_UNLOCK(obj);
+ if (error)
+ VM_OBJECT_LOCK(obj);
mbstat.sf_iocnt++;
}
if (error) {
Index: .
===================================================================
--- . (revision 254800)
+++ . (working copy)
Property changes on: .
___________________________________________________________________
Modified: svn:mergeinfo
Merged /head/sys:r253927
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.21 (FreeBSD)
iQIcBAEBAgAGBQJSGPAuAAoJEJDCuSvBvK1BnbgP/RDnR0CDCza8/lpHlqKrnIwD
cpVk+J+aJdUznwAGt+wuhQld2BaTXwnaQjy1fCQzjJ9kVjrTqqPeqD0DU9eD6i+V
6aCizKb3EB8BLjK6v5qzkdfficr/ojFX7a7x6pDEP7srOoNmoy+m/ZPy45HVzDbJ
YYXuMPW93bC927Tmbds8qdifhe3BzNXKIHH9NVpU1m4JsuhwR3reFjd4kW2B4FJk
+oMeKdASP2CYUzmniNdVqq55Q6RXc/SXZefcxVLTYwgC0TU456TlXr5dOLZCL+gO
wIBfBHi8/e0L0J/2xKe+5LT8F11jWupWoHKHWuyAFf77N7fMoeUeB9quiu6BaUaT
fZ/Pnvu1xzH2bQoeO0ncscF1xK13f9lz0dFKPnynGxKRdcwcQDACHKI8SJLJIN94
leuCV1EZRJdT8ZFqBUlyejagIKu2CnAYd96jLuloqytqksnBQxym2VPAuz2oNVZJ
m5b67brz7iUcHS1FH81htvAic4tSHcnboPo3uWjbZ1kaHyH7KLyuihPsva8pS0hx
GYBc+UrKVgW70r1/JacX0tNhlN6LKPgMNWIy268z8CBGjpU4Z5oY8RB5aArzwETv
ku+klJqbRj3IOByJZi5Vi3zM0galWrnInLlfOnebNqeiRUft2I7sbdvAsVALKZaS
kgOf13FX7LxmW5FV3nKB
=KIxM
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130824174103.GQ4972>