Date: Tue, 17 Jan 2012 15:02:42 +0400 From: Gleb Smirnoff <glebius@FreeBSD.org> To: current@FreeBSD.org Subject: new panic in cpu_reset() with WITNESS Message-ID: <20120117110242.GD12760@glebius.int.ru>
next in thread | raw e-mail | index | archive | help
New panic has been introduced somewhere between r229851 and r229932, that happens on shutdown if kernel has WITNESS and doesn't have WITNESS_SKIPSPIN. Uptime: 1h0m17s Rebooting... panic: mtx_lock_spin: recursed on non-recursive mutex cnputs_mtx @ /usr/src/head/sys/kern/kern_cons.c:500 cpuid = 0 KDB: enter: panic [ thread pid 1 tid 100001 ] Stopped at kdb_enter+0x3b: movq $0,0x514d32(%rip) db> db> bt Tracing pid 1 tid 100001 td 0xfffffe0001d5e000 kdb_enter() at kdb_enter+0x3b panic() at panic+0x1c7 _mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x10f cnputs() at cnputs+0x7a putchar() at putchar+0x11f kvprintf() at kvprintf+0x83 vprintf() at vprintf+0x85 printf() at printf+0x67 witness_checkorder() at witness_checkorder+0x773 _mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x99 uart_cnputc() at uart_cnputc+0x3e cnputc() at cnputc+0x4c cnputs() at cnputs+0x26 putchar() at putchar+0x11f kvprintf() at kvprintf+0x83 vprintf() at vprintf+0x85 printf() at printf+0x67 cpu_reset() at cpu_reset+0x81 kern_reboot() at kern_reboot+0x3a5 --More--^M ^Msys_reboot() at sys_reboot+0x42 amd64_syscall() at amd64_syscall+0x39e Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (55, FreeBSD ELF64, sys_reboot), rip = 0x40ea3c, rsp = 0x7fffffffd6d8, rbp = 0x49 --- db> db> show locks exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff809bc560) locked @ /usr/src/head/sys/kern/kern_module.c:101 exclusive spin mutex smp rendezvous (smp rendezvous) r = 0 (0xffffffff80a08840) locked @ /usr/src/head/sys/kern/kern_shutdown.c:542 db> So the problem is that we are holding smp rendezvous mutex during the cpu_reset(). No mutexes should be obtained after it. However, since cpu_reset() does priting we obtain cnputs_mtx, and later obtain uart_hwmtx. The latter is hardcoded in the subr_witness.c as mutex to obtain before smp rendezvous, this triggers yet another printf from witness, that finally panics due to recursing on cnputs_mtx. -- Totus tuus, Glebius.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120117110242.GD12760>