Date: Tue, 16 Oct 2012 19:29:29 +0300 From: Andriy Gapon <avg@FreeBSD.org> To: dg17@penx.com Cc: freebsd-fs@FreeBSD.org Subject: Re: I have a DDB session open to a crashed ZFS server Message-ID: <507D8B69.3090903@FreeBSD.org> In-Reply-To: <201210161215.33369.jhb@freebsd.org> References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
on 16/10/2012 19:15 John Baldwin said the following: > On Tuesday, October 16, 2012 11:16:37 am Dennis Glatting wrote: >> On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote: >>> On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: >>>> FreeBSD/amd64 (mc) (ttyu0) >>>> >>>> login: NMI ... going to debugger >>>> [ thread pid 11 tid 100003 ] >>> >>> You got an NMI, not a crash. What happens if you just continue ('c' command) >>> from DDB? >>> >> >> I hit the NMI button because of the "crash," which is a misword, to get >> into DDB. > > Ah, I would suggest "hung" or "deadlocked" next time. It certainly seems like > a deadlock since all CPUs are idle. Some helpful commands here might be > 'show sleepchain' and 'show lockchain'. > > Pick a "stuck" process (like find) and run: > > 'show sleepchain <pid>' > > In your case though it seems both 'find' and the various 'pbzip2' threads > are stuck on a condition variable, so there isn't an easy way to identify > an "owner" that is supposed to awaken these threads. It could be a case > of a missed wakeup perhaps, but you'll need to get someone more familiar > with ZFS to identify where these codes should be awakened normally. > I would also re-iterate a suggestion that I made to Nikolay ealrier: http://article.gmane.org/gmane.os.freebsd.devel.file-systems/15981 BTW, in that case it turned out to be a genuine deadlock in ZFS ARC handling of lowmem. procstat -kk -a is a great help for analyzing such situations. -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?507D8B69.3090903>