Date: Thu, 24 Jul 2014 20:37:19 +0200 From: Harald Schmalzbauer <h.schmalzbauer@omnilan.de> To: Konstantin Belousov <kostikbel@gmail.com> Cc: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: panic/lock on 9.3-RELEASE with nullfs/nfs/zfs combination Message-ID: <53D1525F.8090902@omnilan.de> In-Reply-To: <20140724165917.GT93733@kib.kiev.ua> References: <53D12973.3010805@omnilan.de> <20140724165917.GT93733@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Bezüglich Konstantin Belousov's Nachricht vom 24.07.2014 18:59 (localtime): > … >> panic: LK_RETRY set with incompatible flags (0x200400) or an error >> occured (11) >> cpuid = 3 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame >> 0xffffff82e54bcc70 >> kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e54bcd30 >> panic() at panic+0x1cd/frame 0xffffff82e54bce30 >> _vn_lock() at _vn_lock+0x67/frame 0xffffff82e54bce90 >> zfs_lookup() at zfs_lookup+0x420/frame 0xffffff82e54bcf20 >> zfs_freebsd_lookup() at zfs_freebsd_lookup+0xa6/frame 0xffffff82e54bd070 >> VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xd8/frame 0xffffff82e54bd0a0 >> vfs_cache_lookup() at vfs_cache_lookup+0xff/frame 0xffffff82e54bd110 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xd8/frame 0xffffff82e54bd140 >> null_lookup() at null_lookup+0x92/frame 0xffffff82e54bd1c0 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xd8/frame 0xffffff82e54bd1f0 >> lookup() at lookup+0x389/frame 0xffffff82e54bd290 >> namei() at namei+0x3df/frame 0xffffff82e54bd340 >> vn_open_cred() at vn_open_cred+0x1e2/frame 0xffffff82e54bd4b0 >> vop_stdvptocnp() at vop_stdvptocnp+0x1af/frame 0xffffff82e54bd7e0 >> null_vptocnp() at null_vptocnp+0xf5/frame 0xffffff82e54bd850 >> VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0xdb/frame 0xffffff82e54bd880 >> vn_vptocnp_locked() at vn_vptocnp_locked+0x15b/frame 0xffffff82e54bd910 >> vn_fullpath1() at vn_fullpath1+0x100/frame 0xffffff82e54bd970 >> kern___getcwd() at kern___getcwd+0xd4/frame 0xffffff82e54bd9d0 >> amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e54bdaf0 >> Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e54bdaf0 >> --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x8011a191c, rsp = >> 0x7fffffffe658, rbp = 0x801873400 --- >> KDB: enter: panic >> [ thread pid 1905 tid 100856 ] >> Stopped at kdb_enter+0x3b: movq $0,0x642172(%rip) >> >> Like mentioned, this panic happens only if a nfs(v4) client visits fs15 >> (the exported and nullfs_mounted fs) and I try to rw-open any file on >> the nullfs afterwards!!! >> >> How can I provide useful info with KDB? I don't have a dumpdev available >> in that machine??? >> http://www.es.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html >> seems not applicaple, no /var/crash/?*??? >> > The lockmgr flags are LK_SHARE | LK_RETRY, and error 11 == EDEADLK > indicates that the lock is already taken by the curthread in the > exclusive mode. I am interested in what line of code did the locking. > > Add ddb, INVARIANTS, WITNESS and DEBUG_VFS_LOCKS options to the kernel > config, reproduce the issue and, after the panic occured and you > get at the ddb prompt, issue command 'show alllocks'. > > Also, do 'show mount', after which do 'show mount <addr>', where <addr> > is the address of your nullfs mount point, printed by 'show mount'. > > I need all console output starting from the panic message. Maybe it's of any use for you: (kgdb) backtrace #0 doadump (textdump=1) at /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/kern_shutdown.c:271 #1 0xffffffff804aa198 in kern_reboot (howto=260) at /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/kern_shutdown.c:454 #2 0xffffffff804a9bd5 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/kern_shutdown.c:642 #3 0xffffffff8055dcd7 in _vn_lock (vp=0xfffffe002dc119d8, flags=2098176, file=<value optimized out>, line=<value optimized out>) at /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/vfs_vnops.c:1402 #4 0xffffffff80fb8c90 in u8_decomp_b4_tbl () from /boot/kernel/zfs.ko #5 0xfffffe0007ac0b00 in ?? () #6 0x0000000081051480 in ?? () #7 0xffffff82e5660598 in ?? () #8 0xffffff82e5660180 in ?? () #9 0x0000000081050c88 in ?? () #10 0xfffffe002d37c0e8 in ?? () #11 0xffffffff80a8f940 in sdt_vfs_vop_vop_unlock_return () #12 0xfffffe002dc119d8 in ?? () #13 0x0000000900000000 in ?? () #14 0x000000002dcaa0c0 in ?? () #15 0x000001e8e565ff20 in ?? () #16 0xffffff82e565ff40 in ?? () #17 0xffffff82e5660598 in ?? () #18 0xffffff82e56600b0 in ?? () #19 0xffffff82e5660180 in ?? () #20 0xffffff82e56600b0 in ?? () #21 0xffffff82e5660070 in ?? () #22 0xffffffff80fb8e36 in u8_decomp_b4_tbl () from /boot/kernel/zfs.ko #23 0xfffffe00ba802920 in ?? () #24 0xfffffe0000000000 in ?? () #25 0x0000000000002e2e in ?? () #26 0x0000000000000000 in ?? () (kgdb) up 10 #10 0xfffffe002d37c0e8 in ?? () (kgdb) list 1402 KASSERT((flags & LK_RETRY) == 0 || error == 0, 1403 ("LK_RETRY set with incompatible flags (0x%x) or an error occured (%d)", 1404 flags, error)); 1405 /* 1406 * Callers specify LK_RETRY if they wish to get dead vnodes. 1407 * If RETRY is not set, we return ENOENT instead. 1408 */ 1409 if (error == 0 && vp->v_iflag & VI_DOOMED && 1410 (flags & LK_RETRY) == 0) { 1411 VOP_UNLOCK(vp, 0); Just tellme if you need anything else, hav vmcore in kgdb available! Thanks, -Harry [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlPRUl8ACgkQLDqVQ9VXb8h8xwCgipONw2/yv8Oc6+3lTTCv+m7O 16EAn2oFyUuqjbpzJKYQC5gkRk5AsR8v =LY9M -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53D1525F.8090902>