From owner-freebsd-fs@FreeBSD.ORG Mon Oct 5 15:51:36 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3266C106566B; Mon, 5 Oct 2009 15:51:36 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-yw0-f204.google.com (mail-yw0-f204.google.com [209.85.211.204]) by mx1.freebsd.org (Postfix) with ESMTP id C7B228FC08; Mon, 5 Oct 2009 15:51:35 +0000 (UTC) Received: by ywh42 with SMTP id 42so2540968ywh.28 for ; Mon, 05 Oct 2009 08:51:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=NhAOZf4zBczyhhG7nqHkyLvvGiDYj5bpF1afs2RvOXs=; b=IEU5FUGUpJz+SqyDN32X1kzWXGxPIL1Bjm6QgxRER0Tv6mrvn/oa0DNzjATDEWYDL2 z8F5qc0gzdmGXYfC4jX/Vi5JSrhRdzAdZLYFvYLmivb00apmDdj3tbThEp5rZ7TX5eNj JoQ19jc+6rcPC95hNqOh4cem8Nlcs46PUU8rI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=fdeVjLeS6Th8OhOTIi3ush1dPUjLzc5BXLEz6ZCtwJTkcRpH68lF9+gbUXHCwfQW+F dUKGhzauadelHyU4expzXQ2WetNryuuCmz2efeJRy+pPorbJCUcNgFCkZI+sVjCmRyYd s2DI+eSUJZMDrIYJeQEfX/gUnpgmQCBa5Oc+Y= MIME-Version: 1.0 Sender: artemb@gmail.com Received: by 10.90.22.18 with SMTP id 18mr88708agv.20.1254757894250; Mon, 05 Oct 2009 08:51:34 -0700 (PDT) In-Reply-To: <4AC99F1D.3040300@fsn.hu> References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu> <20091002184526.GA1660@garage.freebsd.pl> <4AC99F1D.3040300@fsn.hu> Date: Mon, 5 Oct 2009 08:51:34 -0700 X-Google-Sender-Auth: 4902a9a24d034a97 Message-ID: From: Artem Belevich To: Attila Nagy Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: ARC size constantly shrinks, then ZFS slows down extremely X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 15:51:36 -0000 Your lockup is very similar (processes stuck sleeping on vmwait) to what I had when arc_min was set too high. With Pawel's patch ZFS would not give up any memory above arc_min. Try bringing vfs.zfs.arc_min down. --Artem 2009/10/5 Attila Nagy : > On 10/02/09 20:45, Pawel Jakub Dawidek wrote: >> >> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote: >> >>> >>> Backing out this change from the 8-STABLE kernel: >>> >>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/ut= s/common/fs/zfs/arc.c?r1=3D191901&r2=3D191902 >>> >>> makes it survive about half and hour of IMAP searching. Of course only >>> time will tell whether this helps in the long run, but so far 10/10 tri= es >>> succeeded to kill the machine with this method... >>> >> >> Could you try this patch: >> >> =A0 =A0 =A0 =A0http://people.freebsd.org/~pjd/patches/arc.c.4.patch >> > > Sure. But before that, a report with the above modification: the machine = has > survived some days, then started to behave strangely. Meaning I could pin= g > it, I could log in to the IMAP service (running from ZFS), read some mail= s, > but not all. > I could not access it via ssh (which runs from UFS), but an already runni= ng > top from a different session was alive. It showed: > last pid: 11272; =A0load averages: =A00.00, =A00.00, =A00.00 =A0 =A0up 3+= 15:21:13 > =A009:11:43 > 149 processes: 1 running, 143 sleeping, 1 zombie, 4 waiting > CPU: =A00.0% user, =A00.0% nice, =A00.2% system, =A00.0% interrupt, 99.8%= idle > Mem: 234M Active, 197M Inact, 559M Wired, 111M Buf, 440K Free > Swap: 4096M Total, 976K Used, 4095M Free > > =A0PID USERNAME =A0THR PRI NICE =A0 SIZE =A0 =A0RES STATE =A0 C =A0 TIME = =A0 WCPU COMMAND > 78492 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A04700K =A02156K CPU1 =A0 =A0= 1 =A0 5:37 =A00.00% top > 92343 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A04132K =A01576K nanslp =A01 = =A0 4:12 =A00.00% gstat > 13401 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01528K =A0 456K piperd =A00 = =A0 2:19 =A00.00% > readproctitl > 12679 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A03932K =A01236K vmwait =A01 = =A0 2:12 =A00.00% zpool > 35988 =A0 =A0125 =A0 =A0 =A04 =A045 =A0 =A00 16892K =A05968K sigwai =A00 = =A0 1:53 =A00.00% > milter-greyl > 25656 root =A0 =A0 =A0 =A01 =A045 =A0 =A00 =A01536K =A0 564K getblk =A00 = =A0 1:45 =A00.00% supervise > 25798 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 564K vmwait =A00 = =A0 1:44 =A00.00% supervise > 28406 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 544K vmwait =A00 = =A0 1:43 =A00.00% supervise > 30226 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 544K vmwait =A00 = =A0 1:43 =A00.00% supervise > 35401 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 544K vmwait =A00 = =A0 1:42 =A00.00% supervise > 29203 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A01536K =A0 544K vmwait =A00 = =A0 1:42 =A00.00% supervise > 21629 =A0 =A0389 =A0 =A0 =A06 =A044 =A0 =A00 91664K 41892K ucond =A0 0 = =A0 1:02 =A00.00% slapd > 72283 =A0 =A0 60 =A0 =A0 =A01 =A044 =A0 =A00 80972K =A01948K select =A01 = =A0 0:34 =A00.00% idled > 98960 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A09396K =A02544K select =A01 = =A0 0:32 =A00.00% sshd > 1550 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A03340K =A0 940K vmwait =A01 = =A0 0:32 =A00.00% syslogd > 5463 =A0 =A0125 =A0 =A0 =A01 =A044 =A0 =A00 =A06924K =A02036K vmwait =A00= =A0 0:27 =A00.00% qmgr > 54193 root =A0 =A0 =A0 =A01 =A044 =A0 =A00 =A09396K =A02516K select =A00 = =A0 0:22 =A00.00% sshd > > I could not log into the console, it didn't even gave a "user name" filed > after hitting enter. Strange. > > I will try the patch. > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >