Date: Mon, 5 Oct 2009 08:51:34 -0700 From: Artem Belevich <fbsdlist@src.cx> To: Attila Nagy <bra@fsn.hu> Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek <pjd@freebsd.org> Subject: Re: ARC size constantly shrinks, then ZFS slows down extremely Message-ID: <ed91d4a80910050851m3d599f7ai67a57ef17a9a61e7@mail.gmail.com> In-Reply-To: <4AC99F1D.3040300@fsn.hu> References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu> <20091002184526.GA1660@garage.freebsd.pl> <4AC99F1D.3040300@fsn.hu>
next in thread | previous in thread | raw e-mail | index | archive | help
Your lockup is very similar (processes stuck sleeping on vmwait) to what I had when arc_min was set too high. With Pawel's patch ZFS would not give up any memory above arc_min. Try bringing vfs.zfs.arc_min down. --Artem 2009/10/5 Attila Nagy <bra@fsn.hu>: > On 10/02/09 20:45, Pawel Jakub Dawidek wrote: >> >> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote: >> >>> >>> Backing out this change from the 8-STABLE kernel: >>> >>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/ut= s/common/fs/zfs/arc.c?r1=3D191901&r2=3D191902 >>> >>> makes it survive about half and hour of IMAP searching. Of course only >>> time will tell whether this helps in the long run, but so far 10/10 tri= es >>> succeeded to kill the machine with this method... >>> >> >> Could you try this patch: >> >> =A0 =A0 =A0 =A0http://people.freebsd.org/~pjd/patches/arc.c.4.patch >> > > Sure. But before that, a report with the above modification: the machine = has > survived some days, then started to behave strangely. Meaning I could pin= g > it, I could log in to the IMAP service (running from ZFS), read some mail= s, > but not all. > I could not access it via ssh (which runs from UFS), but an already runni= ng > top from a different session was alive. It showed: > last pid: 11272; =A0load averages: =A00.00, =A00.00, =A00.00 =A0 =A0up 3+= 15:21:13 > =A009:11:43 > 149 processes: 1 running, 143 sleeping, 1 zombie, 4 waiting > CPU: =A00.0% user, =A00.0% nice, =A00.2% system, =A00.0% interrupt, 99.8%= idle > Mem: 234M Active, 197M Inact, 559M Wired, 111M Buf, 440K Free > Swap: 4096M Total, 976K Used, 4095M Free > > =A0PID USERNAME =A0THR PRI NICE =A0 SIZE =A0 =A0RES STATE =A0 C =A0 TIME = =A0 WCPU COMMAND > 78492 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A04700K =A02156K CPU1 =A0 =A0= 1 =A0 5:37 =A00.00% top > 92343 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A04132K =A01576K nanslp =A01 = =A0 4:12 =A00.00% gstat > 13401 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01528K =A0 456K piperd =A00 = =A0 2:19 =A00.00% > readproctitl > 12679 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A03932K =A01236K vmwait =A01 = =A0 2:12 =A00.00% zpool > 35988 =A0 =A0125 =A0 =A0 =A04 =A045 =A0 =A00 16892K =A05968K sigwai =A00 = =A0 1:53 =A00.00% > milter-greyl > 25656 root =A0 =A0 =A0 =A01 =A045 =A0 =A00 =A01536K =A0 564K getblk =A00 = =A0 1:45 =A00.00% supervise > 25798 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 564K vmwait =A00 = =A0 1:44 =A00.00% supervise > 28406 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 544K vmwait =A00 = =A0 1:43 =A00.00% supervise > 30226 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 544K vmwait =A00 = =A0 1:43 =A00.00% supervise > 35401 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 544K vmwait =A00 = =A0 1:42 =A00.00% supervise > 29203 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 544K vmwait =A00 = =A0 1:42 =A00.00% supervise > 21629 =A0 =A0389 =A0 =A0 =A06 =A044 =A0 =A00 91664K 41892K ucond =A0 0 = =A0 1:02 =A00.00% slapd > 72283 =A0 =A0 60 =A0 =A0 =A01 =A044 =A0 =A00 80972K =A01948K select =A01 = =A0 0:34 =A00.00% idled > 98960 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A09396K =A02544K select =A01 = =A0 0:32 =A00.00% sshd > 1550 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A03340K =A0 940K vmwait =A01 = =A0 0:32 =A00.00% syslogd > 5463 =A0 =A0125 =A0 =A0 =A01 =A044 =A0 =A00 =A06924K =A02036K vmwait =A00= =A0 0:27 =A00.00% qmgr > 54193 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A09396K =A02516K select =A00 = =A0 0:22 =A00.00% sshd > > I could not log into the console, it didn't even gave a "user name" filed > after hitting enter. Strange. > > I will try the patch. > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ed91d4a80910050851m3d599f7ai67a57ef17a9a61e7>