From owner-freebsd-stable@FreeBSD.ORG Mon Jun 14 14:42:54 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9632F1065678 for ; Mon, 14 Jun 2010 14:42:54 +0000 (UTC) (envelope-from fb-stable@psconsult.nl) Received: from mx1.psconsult.nl (psc11.adsl.iaf.nl [80.89.238.138]) by mx1.freebsd.org (Postfix) with ESMTP id 16CB48FC1A for ; Mon, 14 Jun 2010 14:42:53 +0000 (UTC) Received: from mx1.psconsult.nl (psc11.adsl.iaf.nl [80.89.238.138]) by mx1.psconsult.nl (8.14.4/8.14.4) with ESMTP id o5EEfVKQ090212 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 14 Jun 2010 16:41:36 +0200 (CEST) (envelope-from fb-stable@psconsult.nl) Received: (from paul@localhost) by mx1.psconsult.nl (8.14.4/8.14.4/Submit) id o5EEfVKg090211 for freebsd-stable@freebsd.org; Mon, 14 Jun 2010 16:41:31 +0200 (CEST) (envelope-from fb-stable@psconsult.nl) X-Authentication-Warning: mx1.psconsult.nl: paul set sender to fb-stable@psconsult.nl using -f Date: Mon, 14 Jun 2010 16:41:31 +0200 From: Paul Schenkeveld To: freebsd-stable@freebsd.org Message-ID: <20100614144131.GA90078@psconsult.nl> References: <4BE110E3.8040902@zirakzigil.org> <4BE82C5D.1080806@bit0.com> <20100608.181146.1224841629154014733.kasahara@nc.kyushu-u.ac.jp> <4C124462.30408@smeets.im> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C124462.30408@smeets.im> User-Agent: Mutt/1.5.19 (2009-01-05) Subject: Re: Freebsd 8.0 kmem map too small X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 14:42:54 -0000 On Fri, Jun 11, 2010 at 04:12:50PM +0200, Florian Smeets wrote: > On 6/8/10 11:11 AM, Yoshiaki Kasahara wrote: > > Hello, > > > > I'd like to add another instance of similar problems. I recently > > updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE > > (as of Mar 1st) to 8.1-PRERELEASE (as of May 27th). After that, my > > box started to crash every couple of days due to kmem_map too small. > > > > Here is a (last week) screenshot of Munin graph about the memory usage > > of the box: > > > > http://eron.info/munin-memory.png > > > > In "by month" graph, a white gap at the end of "Week 20" is the update > > period from 8-STABLE to 8.1-PRERELEASE I mentioned above. Before the > > upgrade, the system was rock solid without any kmem tuning in > > loader.conf (vm.kmem_size was around 2.7GB IIRC). > > > > After the update, I could see that more wired memory was assigned, and > > then steep drop (crash) occured. > > > > "by day" graph shows my experiment to bump vm.kmem_size=12G > > (recommended somewhere in this thread) and explicitly limit > > vfs.zfs.arc_max=2G. I was surprised because the wired memory quickly > > increased over 5GB... > > > > Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on > > amd64, so I turned it off and removed other memory tunings (kmem_size > > and arc_max) in loader.conf on Tuesday at 16:00 and rebooted. It > > seems that the usage of wired memory was stablized and no crash since > > then with the default kmem_size and arc_max. > > > > Does anyone have any idea about this behavior? > > > > My wild guess is that ZFS/UMA code is not stable on amd64 too, and it > > should be turned off by default for 8.1-RELEASE maybe.... > > > > I can confirm this. I just upgraded a system from late January to > yesterdays 8-STABLE. I had to set vfs.zfs.zio.use_uma="0", to get the > system survive buildworld again. > > This is amd64 with only 1GB, (i know this is considered not enough for > ZFS) with the kernel from January it was stable without any tuning. > After disabling ZFSs UMA usage, it seems to be working fine again. > > Just another data point. +1 for me, several of my servers crash *every night* on a script that does csup from cvs.freebsd.org (cvs mode if that matters) followed by cvs up -dP in some source trees and one ports tree. Hardware has 12 GB RAM, dual nehalem processors; kernel is amd64, no sysctl tuning related to ZFS. With kernels before april 18 everything is rock solid here, one production server has been up since april 17. Can do some limited testing on one of these servers on mondays if needed. > Cheers, > Florian Regards, Paul Schenkeveld