From owner-freebsd-stable@FreeBSD.ORG Wed Apr 19 01:29:44 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A346F16A400 for ; Wed, 19 Apr 2006 01:29:44 +0000 (UTC) (envelope-from Stephen.Clark@seclark.us) Received: from smtpout10-04.prod.mesa1.secureserver.net (smtpout10-04.prod.mesa1.secureserver.net [64.202.165.238]) by mx1.FreeBSD.org (Postfix) with SMTP id 47E0643D45 for ; Wed, 19 Apr 2006 01:29:44 +0000 (GMT) (envelope-from Stephen.Clark@seclark.us) Received: (qmail 23669 invoked from network); 19 Apr 2006 01:29:43 -0000 Received: from unknown (24.144.77.138) by smtpout10-04.prod.mesa1.secureserver.net (64.202.165.238) with ESMTP; 19 Apr 2006 01:29:43 -0000 Message-ID: <44459286.1000008@seclark.us> Date: Tue, 18 Apr 2006 21:29:42 -0400 From: Stephen Clark User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.16-22smp i686; en-US; m18) Gecko/20010110 Netscape6/6.5 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Stephen.Clark@seclark.us, stable@freebsd.org References: <4444EE93.9050003@seclark.us> In-Reply-To: <4444EE93.9050003@seclark.us> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: FreeBSD 4.9 losing mbufs!!! X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Stephen.Clark@seclark.us List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Apr 2006 01:29:44 -0000 Stephen Clark wrote: >Hello List, > >I know 4.9 is ancient history, but unfortunately we have several >thousand sites installed. We are in the process of moving to 6.1 when it >is released. > >Right now I have an immediate problem where we are going to install two >system at a >HQ site. Each of the 2 systems will have two gre/vpn/ospf tunnels to a >100 remote sites in the >field. The broadband will be a T3 with failover to dialup actiontec >dualpc modems. We want >to use FreeBSD systems rather than put in Cisco equip which is what we >have done for other >large customers. > >The problem: > >I have been testing between an Athlon 64 3000+ (client) and an Athlon >64 X2 4800+ (server) across a dedicated 100mb lan. When I use nttcp, >which is a round trip tcp test, across the gre/vpn the client system, >(which goes to 0 percent idle), network stack will eventually stop >responding. In trying to track this down I find that >net.inet.ip.intr_queue_maxlen which is normally 50 has been reached (I >added a sysctl to be able to look at it), but it never drains down. If I >increase it things start working again. If I continue to hammer the >client I see the >intr_queue_maxlen continue to grow until it again reaches the new >maximum. Another datapoint if I don't send the data thru the gre tunnel, >but only thru the vpn I don't see this problem. > >I've looked at the gre code til I am blue in the face and can't see >where mbufs were not being freed when the quelen is full. > >If anybody could give some direction as where to look or how to better >trouble shoot this problem it would be greatly appreciated. > >Thanks for being such a great list, >Steve > > > I have discovered that if I disable quaqqa/ospfd then I don't lose mbufs! This makes it appear that the mbuf leak is in the multicast routing logic. In fact I lose mbufs even with the both system basically idle but with a 100 vpn/gre with multicast going on thru the gre then the vpn. Any ideas on where to focus my continued investigation? Thanks to everybody who has responded. Steve -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson)