Date: Tue, 20 May 2014 11:37:53 -0400 From: Nathaniel W Filardo <nwf@cs.jhu.edu> To: Chris Ross <cross+freebsd@distal.com> Cc: freebsd-sparc64@freebsd.org Subject: Re: FreeBSD 10-STABLE/sparc64 panic Message-ID: <20140520153753.GV24043@gradx.cs.jhu.edu> In-Reply-To: <AB5649B5-BBFB-4284-9CFF-4784D28A18F3@distal.com> References: <20140518083413.GK24043@gradx.cs.jhu.edu> <751F7778-95CE-40FC-857F-222FB37737C0@distal.com> <20140518235853.GM24043@gradx.cs.jhu.edu> <20140519145222.GN24043@gradx.cs.jhu.edu> <A092DFEB-D5CF-473E-88BD-81B005C26C57@distal.com> <20140519193529.GO24043@gradx.cs.jhu.edu> <20140519205047.GP24043@gradx.cs.jhu.edu> <CA75738D-066D-4EDC-9018-89936EE861C6@distal.com> <AB5649B5-BBFB-4284-9CFF-4784D28A18F3@distal.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--l5oECiFRo5dp+2y7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Well, I am having a bad day; I tried to roll back to 9 but ended up building a 10 kernel again (too many source checkouts, sigh). In any case, I kicked WITNESS and INVARIANTS on this time and, while I cannot say for sure that it is related, early in boot I am told lock order reversal: 1st 0xc0756998 entropy harvest mutex (entropy harvest mutex) @ /systank/src-git/sys/dev/random/random_harvestq.c:198 2nd 0xfffff800055c7c38 uart_hwmtx (uart_hwmtx) @ /systank/src-git/sys/dev/uart/uart_cpu.h:94 KDB: stack backtrace: _witness_debugger() at _witness_debugger+0x38 witness_checkorder() at witness_checkorder+0xea0 __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x134 uart_cnputc() at uart_cnputc+0x60 cnputc() at cnputc+0xac putchar() at putchar+0xe8 kvprintf() at kvprintf+0x88 _vprintf() at _vprintf+0x38 vprintf() at vprintf+0x10 printf() at printf+0x20 witness_checkorder() at witness_checkorder+0xbf8 __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x134 sleepq_lock() at sleepq_lock+0x58 msleep_spin_sbt() at msleep_spin_sbt+0xb8 random_kthread() at random_kthread+0x294 fork_exit() at fork_exit+0xa4 fork_trampoline() at fork_trampoline+0x8 lock order reversal: 1st 0xc0756998 entropy harvest mutex (entropy harvest mutex) @ /systank/src-git/sys/dev/random/random_harvestq.c:198 2nd 0xc07b1510 sleepq chain (sleepq chain) @ /systank/src-git/sys/kern/subr_sleepqueue.c:237 The middle bit there looks like our favorite lockup: spin lock 0xc07b3cb0 (smp rendezvous) held by 0xfffff80009116920 (tid 100335) too long exclusive spin mutex smp rendezvous (smp rendezvous) r = 0 (0xc07b3cb0) locked @ /systank/src-git/sys/kern/subr_smp.c:497 timeout stopping cpus panic: spin lock held too long cpuid = 1 KDB: stack backtrace: vpanic() at vpanic+0x1b4 panic() at panic+0x20 _mtx_lock_spin_failed() at _mtx_lock_spin_failed+0x74 _mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0xb8 __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x190 tick_get_timecount_mp() at tick_get_timecount_mp+0x94 binuptime() at binuptime+0x3c timercb() at timercb+0x6c tick_intr() at tick_intr+0x220 -- interrupt level=0xe pil=0 %o7=0xc061d208 -- spinlock_exit() at spinlock_exit+0x2c __mtx_unlock_spin_flags() at __mtx_unlock_spin_flags+0x138 uart_cnputc() at uart_cnputc+0xd0 cnputc() at cnputc+0xac putchar() at putchar+0xe8 kvprintf() at kvprintf+0x890 _vprintf() at _vprintf+0x38 log() at log+0x48 do_link_state_change() at do_link_state_change+0x21c taskqueue_run_locked() at taskqueue_run_locked+0x100 taskqueue_run() at taskqueue_run+0x64 taskqueue_swi_run() at taskqueue_swi_run+0x18 intr_event_execute_handlers() at intr_event_execute_handlers+0x154 ithread_loop() at ithread_loop+0x120 fork_exit() at fork_exit+0xa4 fork_trampoline() at fork_trampoline+0x8 So we might be seeing a deadlock between the console mutex and entropy harvesting? Cheers, --nwf; --l5oECiFRo5dp+2y7 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlN7dtEACgkQTeQabvr9Tc/9IgCfQfmqUGKgJde7Dq8hNgoRXp7q cq0AniZcXnhXzt/ouf0bY0GWh4d8LT64 =vPGv -----END PGP SIGNATURE----- --l5oECiFRo5dp+2y7--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140520153753.GV24043>