From owner-freebsd-stable@FreeBSD.ORG Tue Apr 5 01:10:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BF971065673 for ; Tue, 5 Apr 2011 01:10:27 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [128.238.9.200]) by mx1.freebsd.org (Postfix) with ESMTP id 029378FC19 for ; Tue, 5 Apr 2011 01:10:26 +0000 (UTC) Received: (qmail 13668 invoked from network); 5 Apr 2011 01:10:20 -0000 Received: from unknown (HELO ?192.168.0.2?) (spawk@96.224.221.101) by acm.poly.edu with CAMELLIA256-SHA encrypted SMTP; 5 Apr 2011 01:10:20 -0000 Message-ID: <4D9A6BF7.5000106@acm.poly.edu> Date: Mon, 04 Apr 2011 21:10:15 -0400 From: Boris Kochergin User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.12) Gecko/20101031 Thunderbird/3.1.6 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D972FF7.6010901@acm.poly.edu> <20110402153315.GP78089@deviant.kiev.zoral.com.ua> <4D974393.80606@acm.poly.edu> <4D9A307F.9070408@acm.poly.edu> <20110404224334.GA64297@icarus.home.lan> <4D9A68AA.6040803@acm.poly.edu> <20110405010148.GA67821@icarus.home.lan> In-Reply-To: <20110405010148.GA67821@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Kostik Belousov , FreeBSD-STABLE Mailing List Subject: Re: Kernel memory leak in 8.2-PRERELEASE? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Apr 2011 01:10:27 -0000 On 04/04/11 21:01, Jeremy Chadwick wrote: > On Mon, Apr 04, 2011 at 08:56:10PM -0400, Boris Kochergin wrote: >> On 04/04/11 18:43, Jeremy Chadwick wrote: >>> On Mon, Apr 04, 2011 at 04:56:31PM -0400, Boris Kochergin wrote: >>>> On 04/02/11 11:41, Boris Kochergin wrote: >>>>> On 04/02/11 11:33, Kostik Belousov wrote: >>>>>> On Sat, Apr 02, 2011 at 10:17:27AM -0400, Boris Kochergin wrote: >>>>>>> Ahoy. This morning, I awoke to the following on one of my servers: >>>>>>> >>>>>>> pid 59630 (httpd), uid 80, was killed: out of swap space >>>>>>> pid 59341 (find), uid 0, was killed: out of swap space >>>>>>> pid 23134 (irssi), uid 1001, was killed: out of swap space >>>>>>> pid 49332 (sshd), uid 1001, was killed: out of swap space >>>>>>> pid 69074 (httpd), uid 0, was killed: out of swap space >>>>>>> pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space >>>>>>> ... >>>>>>> >>>>>>> And so on. >>>>>>> >>>>>>> The machine is: >>>>>>> >>>>>>> FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu >>>>>>> Dec 2 11:39:21 EST 2010 >>>>>>> spawk@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 >>>>>>> >>>>>>> 10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00 >>>>>>> >>>>>>> The memory line from top intrigued me: >>>>>>> >>>>>>> Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf, >>>>>>> 605M Free >>>>>>> >>>>>>> The machine has 8 gigs of memory, and I don't know what all that wired >>>>>>> memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2 >>>>>>> on it which has had a disk in the UNAVAIL state for a few months: >>>>>>> >>>>>>> # zpool status >>>>>>> pool: home >>>>>>> state: DEGRADED >>>>>>> status: One or more devices could not be used because the label is >>>>>>> missing or >>>>>>> invalid. Sufficient replicas exist for the pool to continue >>>>>>> functioning in a degraded state. >>>>>>> action: Replace the device using 'zpool replace'. >>>>>>> see: http://www.sun.com/msg/ZFS-8000-4J >>>>>>> scrub: none requested >>>>>>> config: >>>>>>> >>>>>>> NAME STATE READ WRITE CKSUM >>>>>>> home DEGRADED 0 0 0 >>>>>>> raidz2 DEGRADED 0 0 0 >>>>>>> ada0 ONLINE 0 0 0 >>>>>>> ada1 ONLINE 0 0 0 >>>>>>> ada2 ONLINE 0 0 0 >>>>>>> ada3 ONLINE 0 0 0 >>>>>>> ada4 ONLINE 0 0 0 >>>>>>> ada5 UNAVAIL 0 85 11 experienced >>>>>>> I/O failures >>>>>>> >>>>>>> errors: No known data errors >>>>>>> >>>>>>> "vmstat -m" and "vmstat -z" output: >>>>>>> >>>>>>> http://acm.poly.edu/~spawk/vmstat-m.txt >>>>>>> http://acm.poly.edu/~spawk/vmstat-z.txt >>>>>>> >>>>>>> Anyone have a clue? I know it's just going to happen again if I reboot >>>>>>> the machine. It is still up in case there are diagnostics for >>>>>>> me to run. >>>>>> Try r218795. Most likely, your issue is not leak. >>>>> Thanks. Will update to today's 8-STABLE and report back. >>>>> >>>>> -Boris >>>> The problem persists, I'm afraid, and seems to have crept up a lot >>>> more quickly than before: >>>> >>>> # uname -a >>>> FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2 >>>> 11:48:43 EDT 2011 >>>> spawk@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 >>>> >>>> Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free >>>> >>>> Any ideas for a diagnostic recourse? >>> Can you please provide the details I requested here? Thanks. >>> >>> http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/062147.html >>> >> No swap, blank /boot/loader.conf, default /etc/sysctl.conf. I'm >> going to try this ARC tuning thing. I vaguely recall several claims >> that tuning wasn't necessary anymore on amd64 systems with the >> amount of memory mine has, but that's obviously not the case. > Given that you don't have swap (again: very, very bad idea), your > applications crashing due to there not being any swap space is expected: > no place to swap them out to. > > All you should need to set, in /boot/loader.conf, is: > > vfs.zfs.arc_max > > For example, if you want to limit the ARC to only use up to 2GB of RAM: > > vfs.zfs.arc_max="2048M" Thanks. I will attempt just this and report back. -Boris > This would reserve (on an 8GB machine) approximately ~6GB of RAM for > userland applications, the kernel, network buffers/mbufs, etc.. > > Finally, please note that most of the stuff you'll read online for ZFS > tuning on FreeBSD is outdated with 8.2. E.g. you should not need to set > vm.kmem_size and you should never need to adjust vm.kmem_size_max. >