Date: Fri, 21 Oct 2016 20:57:08 +0200 From: "O. Hartmann" <ohartman@zedat.fu-berlin.de> To: FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: r3077XX: LOR in vfs Message-ID: <20161021205708.678fe3cc.ohartman@zedat.fu-berlin.de>
next in thread | raw e-mail | index | archive | help
--Sig_/sA7qdw.qFyQLeiDiZRmr+Yk Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Since r307157 I'm bugged with sporadic reboots/crashes of all boxes running= CURRENT. That is different CPU types (XEON C2D, XEON Haswell, XEON IvyBridge, mobile Hasw= ell and desktop IvyBridge). Below, you'll finde some messages I gathered from the console. I hope this = is of any use, if not: I'm very sorry. I'm also sorry being incapable of providing you with more informations of t= he crashes. I do not have any clue how the crashes are related - I try to configure some = boxes I have direct access to with debugging options, but unfortunately they all have nv= idia GPUs and configured vt() for the UEFI and non UEFI systems - that means, the console= is wreckage since there is an unresolved buf as long as the nvidia-mode.ko is loaded. S= o far, the crashing boxes didn't capture a core. =46rom the reports I got of the other crashing servers running recent CURRENT= , last process in braces is in most cases "(swapper)" or "(pager)" - I got this report via= phone, sorry. Anothe observation is that boxes with larger amounts of RAM last longer - a= workstation with 32 GB and r3076XX lasted for ~ 32 hours befor it crashed, a server wit= h 16 GB takes several hours, sometimes only minutes and another machine with 8 GB g= oes down mor frequently. Anyway, try to give some mor infos in case I'm able to capture those. I'm r= eally surprised that it seems that I'm the only one facing those problems at the = moment. =20 [...] Oct 21 05:09:59 <0.2> thor kernel: lock order reversal: Oct 21 05:09:59 <0.2> thor kernel: 1st 0xfffffe01e98449f8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3505 Oct 21 05:09:59 <0.2> thor kernel: 2nd 0xfffff800098c2800 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:2= 81 Oct 21 05:09:59 <0.2> thor kernel: stack backtrace: Oct 21 05:09:59 <0.2> thor ker= nel: #0 0xffffffff80818410 at witness_debugger+0x70 Oct 21 05:09:59 <0.2> thor kern= el: #1 0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 05:09:59 <0.2> thor k= ernel: #2 0xffffffff807bbdc2 at _sx_xlock+0x72 Oct 21 05:09:59 <0.2> thor kernel: #3 0xffffffff80a8b507 at ufsdirhash_remove+0x37 Oct 21 05:09:59 <0.2> thor ker= nel: #4 0xffffffff80a8e760 at ufs_dirremove+0x120 Oct 21 05:09:59 <0.2> thor kernel= : #5 0xffffffff80a961a8 at ufs_rename+0x14a8 Oct 21 05:09:59 <0.2> thor kernel: = #6 0xffffffff80b6ce88 at VOP_RENAME_APV+0xe8 Oct 21 05:09:59 <0.2> thor kernel= : #7 0xffffffff80891fa8 at kern_renameat+0x4a8 Oct 21 05:09:59 <0.2> thor kernel= : #8 0xffffffff80af6934 at amd64_syscall+0x314 Oct 21 05:09:59 <0.2> thor kernel= : #9 0xffffffff80ad6f1b at Xfast_syscall+0xfb Oct 21 05:10:00 <3.3> thor nslcd[5= 43]: [d0722e] <group/member=3D"root"> no available LDAP server found: Server is unavailab= le: Resource temporarily unavailable Oct 21 05:10:00 <3.3> thor nslcd[543]: [d0722e] <group/member=3D"root"> no available LDAP server found: Server is unavailab= le: Resource temporarily unavailable Oct 21 05:10:00 <0.2> thor kernel: acquiring duplic= ate lock of same type: "os.lock_sx" Oct 21 05:10:00 <0.2> thor kernel: 1st os.lock_sx @ nvidia_os.c:599 Oct 21 05:10:00 <0.2> thor kernel: 2nd os.lock_sx @ nvidia_= os.c:599 Oct 21 05:10:00 <0.2> thor kernel: stack backtrace: Oct 21 05:10:00 <0.2> thor = kernel: #0 0xffffffff80818410 at witness_debugger+0x70 Oct 21 05:10:00 <0.2> thor kern= el: #1 0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 05:10:00 <0.2> thor k= ernel: #2 0xffffffff807bbdc2 at _sx_xlock+0x72 Oct 21 05:10:00 <0.2> thor kernel: #3 0xffffffff81e792b2 at os_acquire_mutex+0x32 Oct 21 05:10:00 <0.2> thor kern= el: #4 0xffffffff81e48bc8 at _nv017102rm+0x18 Oct 21 05:10:00 <0.2> thor kernel: a= cquiring duplicate lock of same type: "os.lock_mtx" Oct 21 05:10:00 <0.2> thor kerne= l: 1st os.lock_mtx @ nvidia_os.c:778 Oct 21 05:10:00 <0.2> thor kernel: 2nd os.loc= k_mtx @ nvidia_os.c:778 Oct 21 05:10:00 <0.2> thor kernel: stack backtrace: Oct 21 = 05:10:00 <0.2> thor kernel: #0 0xffffffff80818410 at witness_debugger+0x70 Oct 21 05:10:00= <0.2> thor kernel: #1 0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 05:10:00 <= 0.2> thor kernel: #2 0xffffffff80792604 at __mtx_lock_flags+0xa4 Oct 21 05:10:00 <0.2= > thor kernel: #3 0xffffffff81e7962b at os_acquire_spinlock+0x1b Oct 21 05:10:00 <0.2> tho= r kernel: #4 0xffffffff81b97705 at _nv011755rm+0x185 Oct 21 05:10:00 <0.6> thor kernel: nvidia-modeset: Allocated GPU:0 (GPU-85fde95a-7974-9962-f1a4-d7c164413929) @ PCI:0000:01:00.0 [...] Oct 21 05:23:19 <0.2> thor kernel: lock order reversal: Oct 21 05:23:19 <0.2> thor kernel: 1st 0xfffff8017ee32d50 ufs (ufs) @ /usr/src/sys/kern/vfs_lookup.c:522 Oct 21 05:23:19 <0.2> thor kernel: 2nd 0xfffffe01ea131fa8 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:278= Oct 21 05:23:19 <0.2> thor kernel: 3rd 0xfffff8017eebe5f0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2592 Oct 21 05:23:19 <0.2> thor kernel: stac= k backtrace: Oct 21 05:23:19 <0.2> thor kernel: #0 0xffffffff80818410 at witness_debugge= r+0x70 Oct 21 05:23:19 <0.2> thor kernel: #1 0xffffffff8081830d at witness_checkorder+0xf= 0d Oct 21 05:23:19 <0.2> thor kernel: #2 0xffffffff80788e5f at __lockmgr_args+0x4df O= ct 21 05:23:19 <0.2> thor kernel: #3 0xffffffff80a855f6 at ffs_lock+0xa6 Oct 21 05:23:19 <= 0.2> thor kernel: #4 0xffffffff80b6d9f0 at VOP_LOCK1_APV+0xe0 Oct 21 05:23:19 <0.2> t= hor kernel: #5 0xffffffff808955ca at _vn_lock+0x9a Oct 21 05:23:19 <0.2> thor kernel: #6 0xffffffff80884eb4 at vget+0x64 Oct 21 05:23:19 <0.2> thor kernel: #7 0xfff= fffff80876d7e at vfs_hash_get+0xfe Oct 21 05:23:19 <0.2> thor kernel: #8 0xffffffff80a812= 20 at ffs_vgetf+0x40 Oct 21 05:23:19 <0.2> thor kernel: #9 0xffffffff80a76dee at softdep_sync_buf+0xd4e Oct 21 05:23:19 <0.2> thor kernel: #10 0xffffffff80a= 86973 at ffs_syncvnode+0x2a3 Oct 21 05:23:19 <0.2> thor kernel: #11 0xffffffff80a59f= d2 at ffs_truncate+0x7d2 Oct 21 05:23:19 <0.2> thor kernel: #12 0xffffffff80a8e24= b at ufs_direnter+0x68b Oct 21 05:23:19 <0.2> thor kernel: #13 0xffffffff80a978b= d at ufs_makeinode+0x59d Oct 21 05:23:19 <0.2> thor kernel: #14 0xffffffff80a935= c3 at ufs_create+0x33 Oct 21 05:23:19 <0.2> thor kernel: #15 0xffffffff80b6b2da at VOP_CREATE_APV+0xda Oct 21 05:23:19 <0.2> thor kernel: #16 0xffffffff80894d= e3 at vn_open_cred+0x573 Oct 21 05:23:19 <0.2> thor kernel: #17 0xffffffff8088d52= c at kern_openat+0x25c [...] Oct 21 20:16:57 <0.2> thor kernel: All buffers synced. Oct 21 20:16:57 <0.2> thor kernel: lock order reversal: Oct 21 20:16:57 <0.2> thor kernel: 1st 0xfffff800080419a0 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1253 Oct 21 20:16:57 <0.2> thor kernel: 2nd 0xfffff80004f2a240 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:2732 Oct = 21 20:16:57 <0.2> thor kernel: stack backtrace: Oct 21 20:16:57 <0.2> thor kernel: #0 0xffffffff80818410 at witness_debugger+0x70 Oct 21 20:16:57 <0.2> thor kern= el: #1 0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 20:16:57 <0.2> thor k= ernel: #2 0xffffffff80788e5f at __lockmgr_args+0x4df Oct 21 20:16:57 <0.2> thor kerne= l: #3 0xffffffff8087318c at vop_stdlock+0x3c Oct 21 20:16:57 <0.2> thor kernel: #4 0xffffffff80b6d9f0 at VOP_LOCK1_APV+0xe0 Oct 21 20:16:57 <0.2> thor kernel:= #5 0xffffffff808955ca at _vn_lock+0x9a Oct 21 20:16:57 <0.2> thor kernel: #6 0xffffffff80885539 at vputx+0x169 Oct 21 20:16:57 <0.2> thor kernel: #7 0xffffffff8087cfeb at dounmount+0x5eb Oct 21 20:16:57 <0.2> thor kernel: #8 0xffffffff80886ccb at vfs_unmountall+0x6b Oct 21 20:16:57 <0.2> thor kernel= : #9 0xffffffff80863441 at bufshutdown+0x421 Oct 21 20:16:57 <0.2> thor kernel: = #10 0xffffffff807b2fc9 at kern_reboot+0x189 Oct 21 20:16:57 <0.2> thor kernel: = #11 0xffffffff807b2de3 at sys_reboot+0x3e3 Oct 21 20:16:57 <0.2> thor kernel: #= 12 0xffffffff80af6934 at amd64_syscall+0x314 Oct 21 20:16:57 <0.2> thor kernel= : #13 0xffffffff80ad6f1b at Xfast_syscall+0xfb Oct 21 20:16:57 <0.2> thor kernel:= lock order reversal: Oct 21 20:16:57 <0.2> thor kernel: 1st 0xfffff80008166240 devfs (= devfs) @ /usr/src/sys/kern/vfs_mount.c:1253 Oct 21 20:16:57 <0.2> thor kernel: 2nd 0xfffff80008165d50 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:2732 Oct = 21 20:16:57 <0.2> thor kernel: stack backtrace: Oct 21 20:16:57 <0.2> thor kernel: #0 0xffffffff80818410 at witness_debugger+0x70 Oct 21 20:16:57 <0.2> thor kern= el: #1 0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 20:16:57 <0.2> thor k= ernel: #2 0xffffffff80788e5f at __lockmgr_args+0x4df Oct 21 20:16:57 <0.2> thor kerne= l: #3 0xffffffff8087318c at vop_stdlock+0x3c Oct 21 20:16:57 <0.2> thor kernel: #4 0xffffffff80b6d9f0 at VOP_LOCK1_APV+0xe0 Oct 21 20:16:57 <0.2> thor kernel:= #5 0xffffffff808955ca at _vn_lock+0x9a Oct 21 20:16:57 <0.2> thor kernel: #6 0xffffffff80885539 at vputx+0x169 Oct 21 20:16:57 <0.2> thor kernel: #7 0xffffffff8087cfeb at dounmount+0x5eb Oct 21 20:16:57 <0.2> thor kernel: #8 0xffffffff80886ccb at vfs_unmountall+0x6b Oct 21 20:16:57 <0.2> thor kernel= : #9 0xffffffff80863441 at bufshutdown+0x421 Oct 21 20:16:57 <0.2> thor kernel: = #10 0xffffffff807b2fc9 at kern_reboot+0x189 Oct 21 20:16:57 <0.2> thor kernel: = #11 0xffffffff807b2de3 at sys_reboot+0x3e3 Oct 21 20:16:57 <0.2> thor kernel: #= 12 0xffffffff80af6934 at amd64_syscall+0x314 Oct 21 20:16:57 <0.2> thor kernel= : #13 0xffffffff80ad6f1b at Xfast_syscall+0xfb Oct 21 20:16:57 <0.2> thor kernel:= lock order reversal: Oct 21 20:16:57 <0.2> thor kernel: 1st 0xfffff800081669a0 ufs (uf= s) @ /usr/src/sys/kern/vfs_mount.c:1253 Oct 21 20:16:57 <0.2> thor kernel: 2nd 0xfffff80004f2b068 devfs (devfs) @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1386 O= ct 21 20:16:57 <0.2> thor kernel: stack backtrace: Oct 21 20:16:57 <0.2> thor kernel: #0 0xffffffff80818410 at witness_debugger+0x70 Oct 21 20:16:57 <0.2> thor kern= el: #1 0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 20:16:57 <0.2> thor k= ernel: #2 0xffffffff80788e5f at __lockmgr_args+0x4df Oct 21 20:16:57 <0.2> thor kerne= l: #3 0xffffffff8087318c at vop_stdlock+0x3c Oct 21 20:16:57 <0.2> thor kernel: #4 0xffffffff80b6d9f0 at VOP_LOCK1_APV+0xe0 Oct 21 20:16:57 <0.2> thor kernel:= #5 0xffffffff808955ca at _vn_lock+0x9a Oct 21 20:16:57 <0.2> thor kernel: #6 0xffffffff80a81168 at ffs_flushfiles+0x118 Oct 21 20:16:57 <0.2> thor kerne= l: #7 0xffffffff80a62aa9 at softdep_flushfiles+0x569 Oct 21 20:16:57 <0.2> thor k= ernel: #8 0xffffffff80a837a2 at ffs_unmount+0x82 Oct 21 20:16:57 <0.2> thor kernel: #9 0xffffffff8087d10d at dounmount+0x70d Oct 21 20:16:57 <0.2> thor kernel: #10 0xffffffff80886ccb at vfs_unmountall+0x6b Oct 21 20:16:57 <0.2> thor kernel= : #11 0xffffffff80863441 at bufshutdown+0x421 Oct 21 20:16:57 <0.2> thor kernel: = #12 0xffffffff807b2fc9 at kern_reboot+0x189 Oct 21 20:16:57 <0.2> thor kernel: = #13 0xffffffff807b2de3 at sys_reboot+0x3e3 Oct 21 20:16:57 <0.2> thor kernel: #= 14 0xffffffff80af6934 at amd64_syscall+0x314 Oct 21 20:16:57 <0.2> thor kernel= : #15 0xffffffff80ad6f1b at Xfast_syscall+0xfb --Sig_/sA7qdw.qFyQLeiDiZRmr+Yk Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQEcBAEBCAAGBQJYCmUEAAoJEOgBcD7A/5N85HIIAOH0Odzs3lGKoiSBaqW7eD0q IgzcmQu2j5nqvm3h/J7oCUc+hplF82tOHZQnZAr3qI4RVOHkH+3WFhIknxO66/vb rovvT2WA+qwrGTFHvwPCB9u004IKpqZA0Hz7zv/Z+YLVjWE9/VcrW3QmLxPzYQvV fGvOeV1YsKM/5QkQo96+4KiqJnP+qdzjR0i4u5XSlD8dOzyHqfR6r93eUNnDraE4 iOvdGHDCvouuTXnQgsZhSfNtFv/eB69A5Kg+DoI9QVv7C5oH3XM+0UCS1bnbTyOc 179CCTjQlcg6/AHlsynSA5KUAsUgu8wftwMeppgpO3wnWQSK5DU2jfBc0KBUWpo= =Cooa -----END PGP SIGNATURE----- --Sig_/sA7qdw.qFyQLeiDiZRmr+Yk--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20161021205708.678fe3cc.ohartman>