Date: Mon, 13 Apr 2009 16:36:22 -0700 From: Artem Belevich <fbsdlist@src.cx> To: Ben Kelly <ben@wanderview.com> Cc: freebsd-current@freebsd.org Subject: Re: [patch] zfs livelock and thread priorities Message-ID: <ed91d4a80904131636u18c90474w7cdaa57bc7000e02@mail.gmail.com> In-Reply-To: <BDABA909-C2AE-4A55-869B-CA01BE778A82@wanderview.com> References: <DC9F2088-A0AF-467D-8574-F24A045ABD81@wanderview.com> <49C2CFF6.8070608@egr.msu.edu> <BDABA909-C2AE-4A55-869B-CA01BE778A82@wanderview.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Tried your patch that used PRIBIO+{1,2} for priorities with -current
r191008 and the kernel died with "spinlock held too long" panic.
Actually, there apparently were two instances of panic on different
cores..
Here's output of "alltrace" and "ps" after the crash:
http://pastebin.com/f140f4596
I've reverted the change and kernel booted just fine.
The box is quad-core with two ZFS pools -- one single-disk and another
one is a two-disk mirror. Freebsd is installed on UFS partitions, ZFS
is used for user stuff only.
--Artem
On Thu, Mar 19, 2009 at 5:19 PM, Ben Kelly <ben@wanderview.com> wrote:
> On Mar 19, 2009, at 7:06 PM, Adam McDougall wrote:
>>
>> I was really impressed with your diagnosis but didn't try your patch unt=
il
>> this afternoon. =A0I had not seen processes spin, but I have had zfs get=
stuck
>> roughly every 2 days on a somewhat busy ftp/rsync server until I turned =
off
>> zil again, then it was up for over 13 days when I decided to try this pa=
tch.
>> =A0This system boots from a ufs / and turns around to try mounting a zfs=
root
>> over top, but the first time it stalled for a few minutes at the root mo=
unt
>> and "gave up" with a spinlock held too long, second time same thing but =
I
>> didn't wait long enough for the spinlock error. Then I tried a power cyc=
le
>> just because, and the next two tries I got a page fault kernel panic. =
=A0I'd
>> try to give more details but right now im trying to get the server back =
up
>> with a livecd because I goofed and don't have an old kernel to fall back=
on.
>> =A0Just wanted to let you know, and thanks for getting as far as you did=
!
>
> Ouch! =A0Sorry you ran into that.
>
> I haven't seen these problems, but I keep my root partition on UFS and on=
ly
> use zfs for /usr, /var, etc. =A0Perhaps that explains the difference in
> behavior.
>
> You could try changing the patch to use lower priorities. =A0To do this c=
hange
> compat/opensolaris/sys/proc.h so that it reads:
>
> =A0#define =A0 =A0 =A0 =A0minclsyspri =A0 =A0 PRI_MAX_REALTIME
> =A0#define =A0 =A0 =A0 =A0maxclsyspri =A0 =A0(PRI_MAX_REALTIME - 4)
>
> This compiles and runs on my machine. =A0The theory here is that other ke=
rnel
> threads will be able to run as they used to, but the zfs threads will sti=
ll
> be fixed relative to one another. =A0Its really just a stab in the dark,
> though. =A0I don't have any experience with the "zfs mounted on top of uf=
s
> root" configuration. =A0If this works we should try to see if we can repl=
ace
> PRI_MAX_REALTIME with PRI_MAX_KERN so that the zfs kernel threads run in =
the
> kernel priority range.
>
> If you could get a stack trace of the kernel panic that would be helpful.
> =A0Also, if you have console access, can you break to debugger during the=
boot
> spinlock hang and get a backtrace of the blocked process?
>
> If you want to compare other aspects of your environment to mine I upload=
ed
> a bunch of info here:
>
> =A0http://www.wanderview.com/svn/public/misc/zfs_livelock
>
> Finally, I'm CC'ing the list and some other people so they are aware that
> the patch runs the risk of a panic.
>
> I hope that helps.
>
> - Ben
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org=
"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ed91d4a80904131636u18c90474w7cdaa57bc7000e02>