From owner-freebsd-stable@FreeBSD.ORG Sat Apr 2 14:44:12 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A5BD106564A for ; Sat, 2 Apr 2011 14:44:12 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [128.238.9.200]) by mx1.freebsd.org (Postfix) with ESMTP id E12C18FC0A for ; Sat, 2 Apr 2011 14:44:11 +0000 (UTC) Received: (qmail 23531 invoked from network); 2 Apr 2011 14:17:30 -0000 Received: from unknown (HELO ?192.168.0.2?) (spawk@96.224.221.101) by acm.poly.edu with CAMELLIA256-SHA encrypted SMTP; 2 Apr 2011 14:17:30 -0000 Message-ID: <4D972FF7.6010901@acm.poly.edu> Date: Sat, 02 Apr 2011 10:17:27 -0400 From: Boris Kochergin User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.12) Gecko/20101031 Thunderbird/3.1.6 MIME-Version: 1.0 To: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Kernel memory leak in 8.2-PRERELEASE? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Apr 2011 14:44:12 -0000 Ahoy. This morning, I awoke to the following on one of my servers: pid 59630 (httpd), uid 80, was killed: out of swap space pid 59341 (find), uid 0, was killed: out of swap space pid 23134 (irssi), uid 1001, was killed: out of swap space pid 49332 (sshd), uid 1001, was killed: out of swap space pid 69074 (httpd), uid 0, was killed: out of swap space pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space ... And so on. The machine is: FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu Dec 2 11:39:21 EST 2010 spawk@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00 The memory line from top intrigued me: Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf, 605M Free The machine has 8 gigs of memory, and I don't know what all that wired memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2 on it which has had a disk in the UNAVAIL state for a few months: # zpool status pool: home state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAME STATE READ WRITE CKSUM home DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 ada5 UNAVAIL 0 85 11 experienced I/O failures errors: No known data errors "vmstat -m" and "vmstat -z" output: http://acm.poly.edu/~spawk/vmstat-m.txt http://acm.poly.edu/~spawk/vmstat-z.txt Anyone have a clue? I know it's just going to happen again if I reboot the machine. It is still up in case there are diagnostics for me to run. -Boris