From owner-freebsd-net@FreeBSD.ORG Mon Aug 2 16:52:18 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21942106564A for ; Mon, 2 Aug 2010 16:52:18 +0000 (UTC) (envelope-from ddesimone@verio.net) Received: from relay1-bcrtfl2.verio.net (relay1-bcrtfl2.verio.net [131.103.218.142]) by mx1.freebsd.org (Postfix) with ESMTP id CB84B8FC15 for ; Mon, 2 Aug 2010 16:52:17 +0000 (UTC) Received: from iad-wprd-xchw02.corp.verio.net (iad-wprd-xchw02.corp.verio.net [198.87.7.165]) by relay1-bcrtfl2.verio.net (Postfix) with ESMTP id 8BAD5B038127 for ; Mon, 2 Aug 2010 12:52:16 -0400 (EDT) Thread-Index: AcsyYxAIxtEl9eFxQ9meKcbbEIxUnQ== Received: from dllstx1-8sst9f1.corp.verio.net ([10.144.2.52]) by iad-wprd-xchw02.corp.verio.net over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Mon, 2 Aug 2010 12:52:15 -0400 Received: by dllstx1-8sst9f1.corp.verio.net (sSMTP sendmail emulation); Mon, 02 Aug 2010 11:52:14 -0500 Content-Transfer-Encoding: 7bit Date: Mon, 2 Aug 2010 11:52:14 -0500 From: "David DeSimone" Content-class: urn:content-classes:message To: Importance: normal Priority: normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4657 Message-ID: <20100802165213.GD6052@verio.net> Mail-Followup-To: freebsd-net@freebsd.org References: <20100730181035.GI5168@verio.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20100730181035.GI5168@verio.net> Precedence: bulk User-Agent: Mutt/1.5.20 (2009-06-14) X-OriginalArrivalTime: 02 Aug 2010 16:52:15.0168 (UTC) FILETIME=[0F65EC00:01CB3263] Subject: Re: Kernel (7.3) crash due to mbuf leak? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Aug 2010 16:52:18 -0000 I will try asking this question again: Can anyone give me some pointers as to how I can analyze these crashdumps, or my running system, to determine what network subsystem is leaking these mbuf's? Before I start sifting through C code, I was wondering, does the mbuf subsystem have a built-in means of tracking which other subsystems are requesting memory, so that perhaps a clever gdb script can build a histogram of which subsystems are consuming large amounts of mbuf's? This would give me a pointer in the right direction to start investigating the cause of the leak. David DeSimone wrote: > > After upgrading a couple of our systems from 7.2-RELEASE to 7.3-RELEASE, > we have started to see them running out of mbuf's and crashing every > month or so. The panic string is: > > kmem_malloc(16384): kmem_map too small: 335233024 total allocated > > The actual panic signature (backtrace) shows a memory allocation failure > occurring in the filesystem code, but I do not think that is where the > problem lies. Instead, it is clear to me that the system is slowly > leaking mbuf's until there is no more kernel memory available, and the > filesystem is just the innocent bystander asking for memory and failing > to get it. > > Here's some netstat -m output on a couple of crashes: > > fs0# netstat -m -M vmcore.0 > > 882167/2902/885069 mbufs in use (current/cache/total) > 351/2041/2392/25600 mbuf clusters in use (current/cache/total/max) > 351/1569 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/199/199/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > 221249K/5603K/226853K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines -- David DeSimone == Network Admin == fox@verio.net "I don't like spinach, and I'm glad I don't, because if I liked it I'd eat it, and I just hate it." -- Clarence Darrow This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you.