From owner-freebsd-stable@FreeBSD.ORG Tue Jun 8 09:54:46 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46A59106568E for ; Tue, 8 Jun 2010 09:54:46 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta11.emeryville.ca.mail.comcast.net (qmta11.emeryville.ca.mail.comcast.net [76.96.27.211]) by mx1.freebsd.org (Postfix) with ESMTP id 2C2318FC18 for ; Tue, 8 Jun 2010 09:54:45 +0000 (UTC) Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by qmta11.emeryville.ca.mail.comcast.net with comcast id TMrD1e00117UAYkABMul9n; Tue, 08 Jun 2010 09:54:45 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta13.emeryville.ca.mail.comcast.net with comcast id TMuk1e0063S48mS8ZMulLf; Tue, 08 Jun 2010 09:54:45 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 8EAC19B418; Tue, 8 Jun 2010 02:54:44 -0700 (PDT) Date: Tue, 8 Jun 2010 02:54:44 -0700 From: Jeremy Chadwick To: Yoshiaki Kasahara Message-ID: <20100608095444.GA86953@icarus.home.lan> References: <4BE110E3.8040902@zirakzigil.org> <4BE82C5D.1080806@bit0.com> <20100608.181146.1224841629154014733.kasahara@nc.kyushu-u.ac.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100608.181146.1224841629154014733.kasahara@nc.kyushu-u.ac.jp> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: pjd@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Freebsd 8.0 kmem map too small X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jun 2010 09:54:46 -0000 On Tue, Jun 08, 2010 at 06:11:46PM +0900, Yoshiaki Kasahara wrote: > Hello, > > I'd like to add another instance of similar problems. I recently > updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE > (as of Mar 1st) to 8.1-PRERELEASE (as of May 27th). After that, my > box started to crash every couple of days due to kmem_map too small. > > Here is a (last week) screenshot of Munin graph about the memory usage > of the box: > > http://eron.info/munin-memory.png > > In "by month" graph, a white gap at the end of "Week 20" is the update > period from 8-STABLE to 8.1-PRERELEASE I mentioned above. Before the > upgrade, the system was rock solid without any kmem tuning in > loader.conf (vm.kmem_size was around 2.7GB IIRC). > > After the update, I could see that more wired memory was assigned, and > then steep drop (crash) occured. > > "by day" graph shows my experiment to bump vm.kmem_size=12G > (recommended somewhere in this thread) and explicitly limit > vfs.zfs.arc_max=2G. I was surprised because the wired memory quickly > increased over 5GB... > > Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on > amd64, so I turned it off and removed other memory tunings (kmem_size > and arc_max) in loader.conf on Tuesday at 16:00 and rebooted. It > seems that the usage of wired memory was stablized and no crash since > then with the default kmem_size and arc_max. > > Does anyone have any idea about this behavior? > > My wild guess is that ZFS/UMA code is not stable on amd64 too, and it > should be turned off by default for 8.1-RELEASE maybe.... I realise you're talking about amd64, but I'm not sure how UMA is getting enabled on i386 to begin with. It does look like it's enabled on amd64 by default. src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c -- #if defined(__amd64__) static int zio_use_uma = 1; #else static int zio_use_uma = 0; #endif SYSCTL_DECL(_vfs_zfs); SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO"); TUNABLE_INT("vfs.zfs.zio.use_uma", &zio_use_uma); SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, &zio_use_uma, 0, "Use uma(9) for ZIO allocations"); Also note that this tunable does not appear in sysctl -a output; I believe this is caused by the first parameter to SYSCTL_INT() being incorrect -- should be _vfs_zfs, not _vfs_zfs_zio. Regardless of that, it's still adjustable in loader.conf. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |