From owner-freebsd-stable@FreeBSD.ORG Mon Apr 24 14:48:06 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4098E16A402 for ; Mon, 24 Apr 2006 14:48:06 +0000 (UTC) (envelope-from Stephen.Clark@seclark.us) Received: from smtpout09-04.prod.mesa1.secureserver.net (smtpout09-04.prod.mesa1.secureserver.net [64.202.165.17]) by mx1.FreeBSD.org (Postfix) with SMTP id 8004043D53 for ; Mon, 24 Apr 2006 14:48:05 +0000 (GMT) (envelope-from Stephen.Clark@seclark.us) Received: (qmail 13129 invoked from network); 24 Apr 2006 14:48:04 -0000 Received: from unknown (24.144.77.138) by smtpout09-04.prod.mesa1.secureserver.net (64.202.165.17) with ESMTP; 24 Apr 2006 14:48:02 -0000 Message-ID: <444CE51C.1070304@seclark.us> Date: Mon, 24 Apr 2006 10:47:56 -0400 From: Stephen Clark User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.16-22smp i686; en-US; m18) Gecko/20010110 Netscape6/6.5 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: <4444EE93.9050003@seclark.us> <44459286.1000008@seclark.us> <20060424141346.O44099@fledge.watson.org> In-Reply-To: <20060424141346.O44099@fledge.watson.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: stable@freebsd.org Subject: Re: FreeBSD 4.9 losing mbufs!!! X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Stephen.Clark@seclark.us List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Apr 2006 14:48:06 -0000 Robert Watson wrote: >On Tue, 18 Apr 2006, Stephen Clark wrote: > > > >>I have discovered that if I disable quaqqa/ospfd then I don't lose mbufs! >>This makes it appear that the mbuf leak is in the multicast routing logic. >>In fact I lose mbufs even with the both system basically idle but with a 100 >>vpn/gre with multicast going on thru the gre then the vpn. >> >>Any ideas on where to focus my continued investigation? >> >>Thanks to everybody who has responded. >> >> > >Steve, > >Sorry not to have caught this thread earlier; I've been on travel for the last >few weeks. My general suggestion would be to try to narrow the code paths >traversed to try to eliminate as much code as possible from the search. It >sounds like you've done that pretty effectively :-). > >Typically, memory leaks occur in edge error cases, where the memory is not >properly released, or ownership is unclear. My suggestion would be to add >counters (or look at existing counters where already present) and see if >there's an error case being triggered in about the same quantity that mbuf >leakage is occuring. Chances are, there's an error being returned and a >missing m_freem(). > >Based on your comments above, I might also pay attention to the routing socket >path -- the rate of leak could correspond to the routing daemons talking to >the network stack, rather than the rate of traffic. For example, it could be >that one of the routing messages is handled improperly resulting in a leak. > >Unfortunately, tracking down memory leaks can be quite difficult, and tends to >require a combination of dogged persistence and luck... > >Robert N M Watson > > > Robert, Thanks for your response. I am in the process of moving our app to 6. stable to see if the problem still exists. If it does then maybe I can't generate some enthusiasm form the FreeBSD community to take moew of an interest in the problem. I have a lot of C experience but not with the *BSD network stack, still trying to get a good understanding of the flow of the packets thru the stack. Our next release will be based on 6 but that is months away. We have some Athon 64 X2 we are putting in that will handling 100 to 200 vpn/gre tunnels and right now ipintrq slowly grows which eventually forces a reboot of the systems. Fortune 2000 companies don't like see that happen. Regards, Steve -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson)