From owner-freebsd-fs@FreeBSD.ORG Tue May 1 20:38:13 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BE24916A409; Tue, 1 May 2007 20:38:13 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from moe.cs.uoguelph.ca (moe.cs.uoguelph.ca [131.104.94.198]) by mx1.freebsd.org (Postfix) with ESMTP id 7887913C4BC; Tue, 1 May 2007 20:38:13 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.96.170]) by moe.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id l41Kc6ta012580; Tue, 1 May 2007 16:38:06 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id l41Kd9P16727; Tue, 1 May 2007 16:39:10 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 1 May 2007 16:39:09 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher To: Kris Kennaway In-Reply-To: <20070501160213.GA496@xor.obsecurity.org> Message-ID: References: <20070407165759.GG8831@cicely12.cicely.de> <20070407180319.GH8831@cicely12.cicely.de> <20070407191517.GN63916@garage.freebsd.pl> <20070407212413.GK8831@cicely12.cicely.de> <20070410003505.GA8189@nowhere> <46365F76.7090708@infidyne.com> <20070430213043.GF67738@garage.freebsd.pl> <463665F2.8090605@infidyne.com> <46373CAD.6000502@infidyne.com> <20070501160213.GA496@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.57 on 131.104.94.198 Cc: freebsd-fs@freebsd.org, Craig Boston , freebsd-current@freebsd.org, Pawel Jakub Dawidek Subject: Re: ZFS committed to the FreeBSD base. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 May 2007 20:38:13 -0000 On Tue, 1 May 2007, Kris Kennaway wrote: >> I don't know if it relevent, but I've seen "kmem_map: too small" panics >> when testing my NFSv4 server, ever since about FreeBSD5.4. There is no >> problem running the same server code on FreeBSD4 (which is what I still >> run in production mode) or OpenBSD3 or 4. If I increase the size of the >> map, I can delay the panic for up to about two weeks of hard testing, >> but it never goes away. I don't see any evidence of a memory leak during >> the several days of testing leading up to the panic. (NFSv4 uses >> MALLOC/FREE extensively for state related structures.) > > Sounds exactly like a memory leak to me. How did you rule it out? Well, I had a little program running on the server that grabbed the mti_stats[] out of the kernel and logged them. I had one client mounted running thousands of passes of the Connectathon basic tests (one client, same activity over and over and over again). For a week, the stats don't show any increase in allocation for any type (alloc - free doesn't get unreasonably big), then..."panic: kmem_map too small". How many days it took to happen would vary with the size of the kernel map, but no evidence of a leak prior to the crash. It seemed to be based on the number of times MALLOC and FREE were called. Also, the same server code (except for the port changes, which have nothing to do with the state handling where MALLOC/FREE get called a lot), works fine for months on FreeBSD4 and OpenBSD3.9. So, I won't say a "memory leak is ruled out", but if there was a leak why wouldn't it bite FreeBSD4 or show up in mti_stats[]? I first saw it on FreeBSD6.0, but went back to FreeBSD5.4 and tried the same test and got the same result. rick