From owner-freebsd-net@FreeBSD.ORG  Tue Aug 24 13:00:16 2010
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 0769310656A5
	for <freebsd-net@freebsd.org>; Tue, 24 Aug 2010 13:00:16 +0000 (UTC)
	(envelope-from andre@freebsd.org)
Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2])
	by mx1.freebsd.org (Postfix) with ESMTP id 6A4108FC1B
	for <freebsd-net@freebsd.org>; Tue, 24 Aug 2010 13:00:14 +0000 (UTC)
Received: (qmail 53245 invoked from network); 24 Aug 2010 12:59:10 -0000
Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1])
	(envelope-sender <andre@freebsd.org>)
	by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP
	for <adrian.chadd@gmail.com>; 24 Aug 2010 12:59:10 -0000
Message-ID: <4C73C25F.90903@freebsd.org>
Date: Tue, 24 Aug 2010 15:00:15 +0200
From: Andre Oppermann <andre@freebsd.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
	rv:1.9.2.8) Gecko/20100802 Thunderbird/3.1.2
MIME-Version: 1.0
To: Adrian Chadd <adrian.chadd@gmail.com>
References: <AANLkTikrbCFHz-CnuYcgH2JzpeH5hob0Aa2y5dwn3Hvv@mail.gmail.com>	<AANLkTikYMU=wML_z=HDnkUF1PGYMVa1q-QWTrkxD+7EP@mail.gmail.com>	<20100822222746.GC6013@michelle.cdnetworks.com>	<AANLkTi=t+nG8isp1nf2aBec+FwomApNt0NBPO8LqZ+=9@mail.gmail.com>	<4C724AD9.5020000@freebsd.org>
	<AANLkTikBHiQ15CFKhsP4Z=9bRJEP-1_RAJAS4Y3U1GLT@mail.gmail.com>
In-Reply-To: <AANLkTikBHiQ15CFKhsP4Z=9bRJEP-1_RAJAS4Y3U1GLT@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: pyunyh@gmail.com, freebsd-net@freebsd.org
Subject: Re: 8.0-RELEASE-p3: 4k jumbo mbuf cluster exhaustion
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Aug 2010 13:00:16 -0000

On 24.08.2010 14:37, Adrian Chadd wrote:
> On 23 August 2010 18:18, Andre Oppermann<andre@freebsd.org>  wrote:
>> It seems the 4k clusters do not get freed back to the pool after they've
>> been sent by the NIC and dropped from the socket buffer after the ACK has
>> arrived.  The leak must occur in one of these two places.  The socket
>> buffer is unlikely as it would affect not just you but everyone else too.
>> Thus the mbuf freeing after DMA/tx in the bce(4) driver is the prime
>> suspect.
>
> They don't stay leaked though. Killing the offending process sees
> mbuf's eventually returned.
> It isn't immediate though. It may be related to timing out existing
> socket connections or something?

Try "netstat -n -p tcp -x" to see whether one socket is holding on to
too much data.

> I haven't yet brought up the second box enough to start passing test
> traffic, so I can't provide any further details than this.

Testing with a different network card would help to narrow down the
area to look for the bug as well.

Can you describe your connection capturing setup some more?  Do you
use "ipfw fwd" or some form of NAT?

-- 
Andre