From owner-freebsd-current@FreeBSD.ORG Mon Nov 30 07:47:11 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17D2E106566C; Mon, 30 Nov 2009 07:47:11 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-iw0-f198.google.com (mail-iw0-f198.google.com [209.85.223.198]) by mx1.freebsd.org (Postfix) with ESMTP id AF1778FC0C; Mon, 30 Nov 2009 07:47:10 +0000 (UTC) Received: by iwn36 with SMTP id 36so1876033iwn.3 for ; Sun, 29 Nov 2009 23:47:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=MzY23SGk/8pRaQh+5nNrWwnC8oGsIKGfXmiyiK2DHwA=; b=HflEKXQjYK9ImBpbDgmcpo/QW2nAsdmwivWsfSaRgcQAvh9d9pbGrpMOXqc7RL3LR9 yVDWCg9wvJpOyMXSDEuRbKDpHHEQQ6zmY9OaMR/e3LGhBPSogVWYk03A0PWlOIi39GIa ECGalnCqbpFqtPzi8w+ocgIPEDi81RQZuloz4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=YLBUm3Jnho/0bD+GsmGJyped9tNHSMibirWn1v2/7YwsyNN1KOJgqcKnzI4qiLWaOC sEg+3rcdS1DOUILeCGFO1mTeprWbZxJ0LwBdxApqHix403tltPZBm/h8mQ2n1g6e4xV6 nZ1iYuqzoc4ihfH/i2rlgT86aR+VG5mha3cSo= MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.231.73.131 with SMTP id q3mr732256ibj.6.1259567230027; Sun, 29 Nov 2009 23:47:10 -0800 (PST) In-Reply-To: References: <20091129013026.GA1355@michelle.cdnetworks.com> <74BFE523-4BB3-4748-98BA-71FBD9829CD5@anduin.net> Date: Mon, 30 Nov 2009 15:47:09 +0800 X-Google-Sender-Auth: 7416efab38c77f60 Message-ID: From: Adrian Chadd To: =?ISO-8859-1?Q?Eirik_=D8verby?= Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: pyunyh@gmail.com, weldon@excelsusphoto.com, freebsd-current@freebsd.org, Robert Watson , Gavin Atkinson Subject: Re: FreeBSD 8.0 - network stack crashes? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Nov 2009 07:47:11 -0000 That URL works for me. So how much traffic is this box handling during peak times? I've seen this on the proxy boxes that I've setup. There's a lot of data being tied up in socket buffers as well as being routed between interfaces (ie, stuff that isn't being intercepted.) Take a look at "netstat -an" when things are locked up; see if there's any sockets which have full send/receive queues. I'm going to take a complete stab in the dark here and say this sounds a little like a livelock. Ie, something is queuing data and allocating mbufs for TX (and something else is generating mbufs - I dunno, packet headers?) far faster than the NIC is able to TX them out, and there's not enough backpressure on whatever (say, the stuff filling socket buffers) to stop the mbuf exhaustion. Again, I've seen this kind of crap on proxy boxes. See if you have full socket buffers showing up in netstat -an. Have you tweaked the socket/TCP send/receive sizes? I typically lock mine down to something small (32k-64k for the most part) so I don't hit mbuf exhaustion on very busy proxies. 2c, Adrian 2009/11/30 Eirik =D8verby : > On 29. nov. 2009, at 15.29, Robert Watson wrote: > >> On Sun, 29 Nov 2009, Eirik =D8verby wrote: >> >>> I just did that (-rxcsum -txcsum -tso), but the numbers still keep risi= ng. I'll wait and see if it goes down again, then reboot with those values = to see how it behaves. But right away it doesn't look too good .. >> >> It would be interesting to know if any of the counters in the output of = netstat -s grow linearly with the allocation count in netstat -m. =A0Often = times leaks are associated with edge cases in the stack (typically because = if they are in common cases the bug is detected really quickly!) -- usually= error handling, where in some error case the unwinding fails to free an mb= uf that it should free. =A0These are notoriously hard to track down, unfort= unately, but the stats output (especially where delta alloc is linear to de= lta stat) may inform the situation some more. > > From what I can tell, all that goes up with mbuf usage is traffic/packet = counts. I can't say I see anything fishy in there. > > From the last few samples in > http://anduin.net/~ltning/netstat.log > you can see the host stops receiving any packets, but does a few retransm= its before the session where this script ran timed out. > > /Eirik > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " >