Date: Wed, 17 Oct 2012 10:04:38 +0300 From: Nikolay Denev <ndenev@gmail.com> To: Dennis Glatting <dg@pki2.com> Cc: freebsd-fs@freebsd.org, dg17@penx.com, Andriy Gapon <avg@freebsd.org> Subject: Re: I have a DDB session open to a crashed ZFS server Message-ID: <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> In-Reply-To: <alpine.BSF.2.00.1210161139060.22959@btw.pki2.com> References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> <alpine.BSF.2.00.1210161139060.22959@btw.pki2.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Oct 16, 2012, at 9:48 PM, Dennis Glatting <dg@pki2.com> wrote:
>=20
>=20
> On Tue, 16 Oct 2012, Andriy Gapon wrote:
>=20
>> on 16/10/2012 19:15 John Baldwin said the following:
>>> On Tuesday, October 16, 2012 11:16:37 am Dennis Glatting wrote:
>>>> On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote:
>>>>> On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote:
>>>>>> FreeBSD/amd64 (mc) (ttyu0)
>>>>>>=20
>>>>>> login: NMI ... going to debugger
>>>>>> [ thread pid 11 tid 100003 ]
>>>>>=20
>>>>> You got an NMI, not a crash. What happens if you just continue =
('c' command)
>>>>> from DDB?
>>>>>=20
>>>>=20
>>>> I hit the NMI button because of the "crash," which is a misword, to =
get
>>>> into DDB.
>>>=20
>>> Ah, I would suggest "hung" or "deadlocked" next time. It certainly =
seems like
>>> a deadlock since all CPUs are idle. Some helpful commands here =
might be
>>> 'show sleepchain' and 'show lockchain'.
>>>=20
>>> Pick a "stuck" process (like find) and run:
>>>=20
>>> 'show sleepchain <pid>'
>>>=20
>>> In your case though it seems both 'find' and the various 'pbzip2' =
threads
>>> are stuck on a condition variable, so there isn't an easy way to =
identify
>>> an "owner" that is supposed to awaken these threads. It could be a =
case
>>> of a missed wakeup perhaps, but you'll need to get someone more =
familiar
>>> with ZFS to identify where these codes should be awakened normally.
>>>=20
>>=20
>> I would also re-iterate a suggestion that I made to Nikolay ealrier:
>> http://article.gmane.org/gmane.os.freebsd.devel.file-systems/15981
>>=20
>> BTW, in that case it turned out to be a genuine deadlock in ZFS ARC =
handling of
>> lowmem.
>> procstat -kk -a is a great help for analyzing such situations.
>>=20
>=20
> Without restarting the server and from memory, I believe the ARC on =
this server is 32GB. The L2ARC is a 50-60GB SSD. The ZIL is a 16GB =
partitioned SSD but my non-ZIL systems have the same problem. Main =
memory is 128GB.
>=20
> I can run procstat to a serial console and scarf the output. What =
interval would be helpful? Five seconds? Remember when the system hangs, =
no commands will run so the data will be pre-hang.
>=20
> BTW, it takes 4-24 hours to hang under load.
>=20
> Also, are you suggesting I apply the patch in the URL and run again? I =
have been following your other posts but the patches you posted did not =
cleanly apply, so I removed them from my rev.
>=20
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Hi,
I'm running with the patch from here : =
http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/16000/focus=3D=
16017
And there were no deadlocks since it's applied.
If you're hitting the same issue as I was, this should help.
Regards,
Nikolay=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0B0CA833-79FA-4C8E-86AC-828E7947FF67>