Date: Wed, 2 May 2007 15:53:50 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Craig Boston <craig@xfoil.gank.org>, Pawel Jakub Dawidek <pjd@freebsd.org>, freebsd-fs@freebsd.org, freebsd-current@freebsd.org, Kris Kennaway <kris@obsecurity.org> Subject: Re: ZFS committed to the FreeBSD base. Message-ID: <20070502154934.E30345@fledge.watson.org> In-Reply-To: <Pine.GSO.4.63.0705011630010.15295@muncher> References: <20070407165759.GG8831@cicely12.cicely.de> <20070407180319.GH8831@cicely12.cicely.de> <20070407191517.GN63916@garage.freebsd.pl> <20070407212413.GK8831@cicely12.cicely.de> <20070410003505.GA8189@nowhere> <46365F76.7090708@infidyne.com> <20070430213043.GF67738@garage.freebsd.pl> <463665F2.8090605@infidyne.com> <46373CAD.6000502@infidyne.com> <Pine.GSO.4.63.0705011033410.23282@muncher> <20070501160213.GA496@xor.obsecurity.org> <Pine.GSO.4.63.0705011630010.15295@muncher>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 1 May 2007, Rick Macklem wrote:
> On Tue, 1 May 2007, Kris Kennaway wrote:
>
>>> I don't know if it relevent, but I've seen "kmem_map: too small" panics
>>> when testing my NFSv4 server, ever since about FreeBSD5.4. There is no
>>> problem running the same server code on FreeBSD4 (which is what I still
>>> run in production mode) or OpenBSD3 or 4. If I increase the size of the
>>> map, I can delay the panic for up to about two weeks of hard testing, but
>>> it never goes away. I don't see any evidence of a memory leak during the
>>> several days of testing leading up to the panic. (NFSv4 uses MALLOC/FREE
>>> extensively for state related structures.)
>>
>> Sounds exactly like a memory leak to me. How did you rule it out?
> Well, I had a little program running on the server that grabbed the
> mti_stats[] out of the kernel and logged them. I had one client mounted
> running thousands of passes of the Connectathon basic tests (one client,
> same activity over and over and over again). For a week, the stats don't
> show any increase in allocation for any type (alloc - free doesn't get
> unreasonably big), then..."panic: kmem_map too small". How many days it took
> to happen would vary with the size of the kernel map, but no evidence of a
> leak prior to the crash. It seemed to be based on the number of times MALLOC
> and FREE were called.
>
> Also, the same server code (except for the port changes, which have nothing
> to do with the state handling where MALLOC/FREE get called a lot), works
> fine for months on FreeBSD4 and OpenBSD3.9.
>
> So, I won't say a "memory leak is ruled out", but if there was a leak why
> wouldn't it bite FreeBSD4 or show up in mti_stats[]?
>
> I first saw it on FreeBSD6.0, but went back to FreeBSD5.4 and tried the same
> test and got the same result.
Historically, such panics have been a result of one of two things:
(1) An immediate resource leak in UMA(9) or malloc(9) allocated memory.
(2) Mis-tuning of a resource limit, perhaps due to sizing the limit based on
solely physical memory size, not taking available kernel address space
into account.
mti_stats reports only on malloc(9), you need to also look at uma(9), since
many frequently allocated types are allocated directly with the slab
allocator, and not from kernel malloc. Take a look at the output of "show
uma" or "show malloc" in DDB, or respectively "vmstat -z" and "vmstat -m" on a
core or on a live system. malloc(9) is actually implemented using two
different back-ends: UMA-managed fixed size memory buckets for small
allocations, and direct page allocation for large allocations.
The most frequent example of (2) is mis-tuning in the maximum vnode limit of
the system, resulting in the vnode cache exceeding available address space.
Try tuning down that limit. Notice that vnodes, inodes, and most frequently
used file system allocation data types are allocated using uma(9) and not
malloc(9).
Robert N M Watson
Computer Laboratory
University of Cambridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070502154934.E30345>