From owner-freebsd-stable@FreeBSD.ORG Mon Apr 4 21:24:09 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCB2E106566B for ; Mon, 4 Apr 2011 21:24:09 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8BC948FC13 for ; Mon, 4 Apr 2011 21:24:09 +0000 (UTC) Received: by qyk27 with SMTP id 27so4469422qyk.13 for ; Mon, 04 Apr 2011 14:24:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=Kpe38f7Cr6WDYdhmHLnJvSVKcZNFP6Rr3bwcDl7Tfn8=; b=x0c5polilErEcIB53oE7BJgNMgnNLWScelrYyNLJx51yIZiopR6VzeSwBaA4vU6L17 B9g6wU9qtc0Fn4d+2A6whQJcNfF5jNAw7NOLd2WAVgHnEFHa6uzidMnPMgd2EhjgYdAX p0BnNDzmRu7h6/0sQGxXo7bkiERNadU31ftks= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=jW6dkm8wH3W3xQDxMTTcAfN99S5MBX9GxWNJFsqjcDgX5VJraHNeabdlc60KQUGn6D OMkMcITZnmIeg8TrnRhl72ER9CyRnEKKaPqyTesRjBNXLg9wqh7HKkqbA0Dil9cpgVP4 TQ3GGV81o8TxoFx32EA1LZ2dpGNXQ1WxZPHtE= MIME-Version: 1.0 Received: by 10.224.212.65 with SMTP id gr1mr5926255qab.382.1301952247723; Mon, 04 Apr 2011 14:24:07 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.229.233.195 with HTTP; Mon, 4 Apr 2011 14:24:07 -0700 (PDT) In-Reply-To: <4D9A307F.9070408@acm.poly.edu> References: <4D972FF7.6010901@acm.poly.edu> <20110402153315.GP78089@deviant.kiev.zoral.com.ua> <4D974393.80606@acm.poly.edu> <4D9A307F.9070408@acm.poly.edu> Date: Mon, 4 Apr 2011 14:24:07 -0700 X-Google-Sender-Auth: cNMb8B_vkvGXkFGmKv3RMNyNNZk Message-ID: From: Artem Belevich To: Boris Kochergin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Kostik Belousov , FreeBSD-STABLE Mailing List Subject: Re: Kernel memory leak in 8.2-PRERELEASE? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Apr 2011 21:24:09 -0000 On Mon, Apr 4, 2011 at 1:56 PM, Boris Kochergin wrote: > The problem persists, I'm afraid, and seems to have crept up a lot more > quickly than before: > > # uname -a > FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr =A02 > 11:48:43 EDT 2011 =A0 =A0 spawk@exodus.poly.edu:/usr/obj/usr/src/sys/EXOD= US > =A0amd64 > > Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free > > Any ideas for a diagnostic recourse? My bet would be that the wired memory is used by ZFS ARC. In your vmstat-m output you can see that ~2.2G were allocated for 'solaris' subsystem. Due to the fact that ARC allocations tend to be random, we do waste a lot of memory on that. There were few patches floating on stable@ adn fs@ that were supposed to mitigate the issue, but so far there's no good out of the box solution. General advice is to tune ARC so that it works in your case. Key loader tunables are vfs.zfs.arc_min and vfs.zfs.arc_max. Don't set min too high and experimentally set max to the maximum value that does not cause problems. One of the factors that makes things noticeably worse is presence of i/o actifity on non-ZFS filesystems. Regular filesystems cache competes with ZFS for RAM. In the past ZFS used to give up memory way too easily. Currently it's a bit more balanced, but is still far from perfect. By the way, if you don't have on-disk swap configured, I'd recommend adding some. It may help avoiding processes being killed during intermittent memory shortages. If would also help if you could post your /boot/loader.conf and output of "zfs-stats -a" (available in ports). --Artem