From owner-freebsd-stable@FreeBSD.ORG Thu Jun 2 22:39:42 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7ECD1106564A for ; Thu, 2 Jun 2011 22:39:42 +0000 (UTC) (envelope-from torfinn.ingolfsen@broadpark.no) Received: from thalia-smout.broadpark.no (thalia-smout.broadpark.no [80.202.8.21]) by mx1.freebsd.org (Postfix) with ESMTP id 38B9A8FC08 for ; Thu, 2 Jun 2011 22:39:42 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from ignis-smin.broadpark.no ([80.202.8.11]) by thalia-smout.broadpark.no (Sun Java(tm) System Messaging Server 7u3-15.01 64bit (built Feb 12 2010)) with ESMTP id <0LM600A8KPM4WR40@thalia-smout.broadpark.no> for freebsd-stable@freebsd.org; Fri, 03 Jun 2011 00:39:40 +0200 (CEST) Received: from kg-v2.kg4.no ([84.48.120.215]) by ignis-smin.broadpark.no (Sun Java(tm) System Messaging Server 7u3-15.01 64bit (built Feb 12 2010)) with SMTP id <0LM60083OPM4U470@ignis-smin.broadpark.no> for freebsd-stable@freebsd.org; Fri, 03 Jun 2011 00:39:40 +0200 (CEST) Date: Fri, 03 Jun 2011 00:39:40 +0200 From: Torfinn Ingolfsen To: freebsd-stable@freebsd.org Message-id: <20110603003940.d0b3821b.torfinn.ingolfsen@broadpark.no> In-reply-to: <20110602195026.GA54023@icarus.home.lan> References: <20110602213116.425400b6.torfinn.ingolfsen@broadpark.no> <20110602195026.GA54023@icarus.home.lan> X-Mailer: Sylpheed 3.1.0 (GTK+ 2.22.1; amd64-portbld-freebsd8.1) X-Face: "t9w2,-X@O^I`jVW\sonI3.,36KBLZE*AL[y9lL[PyFD*r_S:dIL9c[8Y>V42R0"!"yb_zN,f#%.[PYYNq; m"_0v; ~rUM2Yy!zmkh)3&U|u!=T(zyv,MHJv"nDH>OJ`t(@mil461d_B'Uo|'nMwlKe0Mv=kvV?Nh@>Hb<3s_z2jYgZhPb@?Wi^x1a~Hplz1.zH Subject: Re: Fileserver panic - FreeBSD 8.1-stable and zfs X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2011 22:39:42 -0000 On Thu, 02 Jun 2011 12:50:26 -0700 Jeremy Chadwick wrote: > > This is a well-known thing with ZFS on FreeBSD. Because you're running > 8.1-STABLE, this makes figuring out all the tunables and so on a lot > more difficult than if you were running 8.2-STABLE. FWIW, the machine has been quite stable for me for a long time. > Please provide: > > 1) Contents of /boot/loader.conf root@kg-f2# more /boot/loader.conf zfs_load="YES" vfs.root.mountfrom="zfs:zroot" siis_load="YES" amdtemp_load="YES" # testing without MSI hw.pci.enable_msix="0" hw.pci.enable_msi="0" > 2) Output from: sysctl hw.physmem hw.usermem hw.realmem (your hardware > page says 4GB, but I can't be bothered to sift through multi-pages > of wiki documents and links to find the answers) root@kg-f2# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 4141920256 hw.usermem: 3721527296 hw.realmem: 4966055936 > 3) Output from: sysctl vfs.zfs.zio.use_uma root@kg-f2# sysctl vfs.zfs.zio.use_uma vfs.zfs.zio.use_uma: 0 > The scrub itself was not ultimately responsible for this problem > (meaning "the bug is not in scrub"). The problem is that your kernel > effectively wanted more memory for ZFS operations than was available. Understood. I didn't mean to imply it was; I just tried to provide data about activity on the server that might have contributed to the failure. FWIW, the scrub finished fine: root@kg-f2# zpool status storage pool: storage state: ONLINE scrub: scrub completed after 307445734561825860h15m with 0 errors on Thu Jun 2 23:23:44 2011 config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad8 ONLINE 0 0 0 ad10 ONLINE 0 0 0 ad12 ONLINE 0 0 0 ad14 ONLINE 0 0 0 ada0 ONLINE 0 0 0 errors: No known data errors > The "trick" is to tune /boot/loader.conf until you can gain stability. Well, the server has been reasonably stable for me for about a year now (I had to replace a failing hard drive, but I count that as "wear" not "instability"). > Again, because you're running 8.1-STABLE, the tuning parameters here > will behave different than on 8.2-STABLE. We can go over those in a > follow-up thread. I have no trouble with upgrading the server to 8.2-stable, if now is a good time to do it. (I haven't watched closely for any zfs related problems on the mailing list lately.) > I've gotten to the point where I literally cannot remember all of the > different situations/conditions/tunings for each FreeBSD kernel build, > release, date, type, etc., so I tend to focus on the most recent > RELENG_8 build. Then someone comes along with an older build..... > Hehe. :-) I know what you mean. Keeping up with all this "stuff" is getting harder every year. :) -- Torfinn