From owner-freebsd-fs@FreeBSD.ORG Fri May 28 14:52:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 969CA1065677; Fri, 28 May 2010 14:52:01 +0000 (UTC) (envelope-from howard@leadmon.net) Received: from mail.leadmon.net (mail.leadmon.net [173.13.218.154]) by mx1.freebsd.org (Postfix) with ESMTP id E05AA8FC24; Fri, 28 May 2010 14:52:00 +0000 (UTC) Received: from HDLDESKTOP64 (hdl-desktop-64.leadmon.net [10.0.0.3]) (authenticated bits=0) by mail.leadmon.net (8.14.4/8.14.4/LNSG+SCOP+PSBL+LUBL+NJABL+SBL+DSBL+SORBS+CBL+RHSBL) with ESMTP id o4SEpxUE002645 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Fri, 28 May 2010 10:51:59 -0400 (EDT) (envelope-from howard@leadmon.net) X-DKIM: OpenDKIM Filter v2.0.1 mail.leadmon.net o4SEpxUE002645 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=leadmon.net; s=default; t=1275058319; bh=9WExckdNJqQDgTS2N2MbFN0SCHUtcpPzlUBu9J71Ha8=; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type:Content-Transfer-Encoding; b=HhjeHJICXY8gSyejutBpFFdIgk/0SeCv8Xbfa3RrLwf6QuQ3ywVZezKN1NVObvHcY iPBtTiI1hpm7cnMJJWXXu8sIet2Gfz8x5Q5YxlkM740PiLYeudZ7eGDEpgfupiHoJq CTGr5ojVL73K+RcxNvKTMlJiVQxFY7/UYDsLI3rw= X-DomainKeys: Sendmail DomainKeys Filter v1.0.2 mail.leadmon.net o4SEpxUE002645 DomainKey-Signature: a=rsa-sha1; s=default; d=leadmon.net; c=simple; q=dns; h=x-senderid:authentication-results:from:to:cc:references: in-reply-to:subject:date:message-id:mime-version:content-type: content-transfer-encoding:x-mailer:thread-index:content-language; b=AqvFviPoZCqcU2DRDhEJ0+fg+1NEpDWfVDHQ3m7h8o/VtvPRf52UjlaOIobM5g1Uu Z66ZOXqjOh1uHBfH67I6psQZqKIg1sx82rwWJ5GImjUuIeMZVH/86FFY8A5FxPS7Yla 1I1oSesuB85fPD4tzEgeLDBQCenGBt7tJM41tq8= X-SenderID: Sendmail Sender-ID Filter v1.0.0 mail.leadmon.net o4SEpxUE002645 Authentication-Results: mail.leadmon.net; sender-id=pass header.from=howard@leadmon.net; auth=pass (LOGIN); spf=pass smtp.mfrom=howard@leadmon.net From: "Howard Leadmon" To: "'Jeremy Chadwick'" , "'Andriy Gapon'" References: <060401cafe37$a411b240$ec3516c0$@net><4BFF894F.4010008@icyb.net. ua> <20100528134549.GA75411@icarus.home.lan> In-Reply-To: <20100528134549.GA75411@icarus.home.lan> Date: Fri, 28 May 2010 10:51:59 -0400 Message-ID: <064701cafe75$537c1530$fa743f90$@net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acr+bBxoGdlG04UPQlmLFByMWNN2YQABhuKA Content-Language: en-us X-Virus-Scanned: clamav-milter 0.96.1 at vorlon.leadmon.net X-Virus-Status: Clean Cc: amd64@freebsd.org, freebsd-fs@freebsd.org Subject: RE: FreeBSD 8.1-Prerelease Panic amd64 w/ZFS.. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 May 2010 14:52:01 -0000 > -----Original Message----- > From: Jeremy Chadwick [mailto:freebsd@jdc.parodius.com] > Sent: Friday, May 28, 2010 9:46 AM > To: Andriy Gapon > Cc: Howard Leadmon; amd64@freebsd.org; freebsd-fs@freebsd.org > Subject: Re: FreeBSD 8.1-Prerelease Panic amd64 w/ZFS.. > > On Fri, May 28, 2010 at 12:13:51PM +0300, Andriy Gapon wrote: > > on 28/05/2010 10:30 Howard Leadmon said the following: > > > I know there used to be some issues with this a while back, but > thought > > > with the 8.x FBSD servers most of this tuned itself, or then again > maybe > > > this is something different. > > > > > > > > > > > > For the first time I ever recall, I found my FreeBSD 8 server was > crashed > > > this past morning, with the following error: > > > > > > > > > > > > panic:kmem_malloc(131072):kmem_map to small: 1296826368 total > allocated > > > > > > cupid=4 > > > > O wow, this is an amd64 system (with 4G RAM) and you've got "kmem_map > too > > small". That's very very strange. > > It is? On amd64, vm.kmem_size (not vm.kmem_size_max) is what has to be > increased. I can point folks to the "official" statement from pjd@ and > some others if need be. For a very long time I questioned this because > for an even longer amount of time we were being told to increase > vm.kmem_size_max. > > vm.kmem_size_max, by default, is already huge on amd64 (~320GB or > something like that). Proof: > > vm.kmem_size_max: 329853485875 > > To the OP: you will need to increase vm.kmem_size in /boot/loader.conf > and reboot the system. "What value do I pick?" With 4GB, I would > recommend you use these two variables: > > vm.kmem_size="2048M" > vfs.zfs.arc_max="1536M" > > This will increase the available kmem, and also limit the ARC size > explicitly to nothing larger than 1.5GB. This should stabilise your > system. > > -- > | Jeremy Chadwick jdc@parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP: 4BD6C0CB | Thanks Jeremy, I will try your recommended settings provided above. To the other poster, as to the settings of kmem, I had nothing specific set, just whatever FBSD was using by default. In loader.conf all I had was: zfs_load="YES" vfs.root.mountfrom="zfs:tank/root" As to the setting of kmem and arc, I had the following which I will assume were defaults or auto-tunes: vfs.zfs.arc_max : 862653440 vm.kmem_size : 1380245504 vm.kmem_size_max: 329853485875 Like Jeremy said, it sure looks like vm.kmem_size_max is quite large by default. After applying Jeremy's recommendations, I now see: vfs.zfs.arc_max : 1610612736 vm.kmem_size : 2147483648 vm.kmem_size_max: 329853485875 I guess while we are all on the subject, I notice in the dmesg log the message: ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. Is this anything I want to enable, that's like a big performance win, or do I just not have enough RAM to support it? Always been kinda curious about it, but so far I am liking ZFS, well outside of the machine panic.. LOL --- Howard Leadmon