From owner-freebsd-net@FreeBSD.ORG Mon Jan 26 17:53:13 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D602A106580D for ; Mon, 26 Jan 2009 17:53:13 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from proxy.meer.net (proxy.meer.net [64.13.141.13]) by mx1.freebsd.org (Postfix) with ESMTP id 4FB6F8FC16 for ; Mon, 26 Jan 2009 17:53:13 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mail.meer.net (mail.meer.net [64.13.141.3]) by proxy.meer.net (8.14.3/8.14.3) with ESMTP id n0QHr06f081266; Mon, 26 Jan 2009 09:53:07 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id n0QHqmUv063865; Mon, 26 Jan 2009 09:52:48 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from gnnbsd.hudson-trading.com.neville-neil.com (209.249.190.8.available.above.net [209.249.190.8] (may be forged)) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.3) with ESMTP id n0QHqlPC026905; Mon, 26 Jan 2009 09:52:48 -0800 (PST) (envelope-from gnn@neville-neil.com) Date: Mon, 26 Jan 2009 12:52:46 -0500 Message-ID: <7iocxu7y4x.wl%gnn@neville-neil.com> From: gnn@freebsd.org To: Rui Paulo In-Reply-To: References: <000001c97e23$0d81df20$39ed1aac@mtl.com> User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.7 Emacs/22.3 (amd64-portbld-freebsd7.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Bayes-Prob: 0.5 (Score 0, tokens from: ) X-Spam-Score: 0.00 () [Tag at 5.00] X-CanIt-Geo: No geolocation information available for 64.13.141.3 X-CanItPRO-Stream: default X-Canit-Stats-ID: 3127188 - 1fee5fc051ef X-Scanned-By: CanIt (www . roaringpenguin . com) on 64.13.141.13 Cc: Liran Liss , freebsd-net@freebsd.org, Yony Yossef , johan@nocrew.org, Amit Krig , Eitan Shefi Subject: Re: freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jan 2009 17:53:16 -0000 At Sat, 24 Jan 2009 16:20:06 +0000, Rui Paulo wrote: > > > On 24 Jan 2009, at 12:54, Yony Yossef wrote: > > > Hi All, > > > > I'm facing a temporary network hang on my interfaces following a flood > > ping/stress udp test. > > > > I'm running a netperf UDP test which is giving results but does not > > return > > to the shell. > > client output: > > > > UDP UNIDIRECTIONAL SEND TEST from fe80::202:c9ff:fe02:e1fe%mtnic0 > > (fe80::202:c9ff:fe02:e1fe) port 0 AF_INET6 to > > fe80::202:c9ff:fe02:e1f4%mt > > nic0 (fe80::202:c9ff:fe02:e1f4) port 0 AF_INET6 > > Socket Message Elapsed Messages > > Size Size Time Okay Errors Throughput > > bytes bytes secs # # 10^6bits/sec > > > > 32768 1472 10.02 547428 1694280 643.60 > > 32768 10.02 25089 29.50 > > > > > > (HANG) > > > > After a minute or two it returns to the shell with the following > > message: > > shutdown_control: no response received errno 55 > > > > 20 minutes later (!!) the interface is working again. > > > > netstat -m and vmstat -z outputs during the hang time: > > > > # netstat -m > > 25687/6578/32265 mbufs in use (current/cache/total) > > 17404/2438/19842/65536 mbuf clusters in use (current/cache/total/max) > > 0/1024 mbuf+clusters out of packet secondary zone in use (current/ > > cache) > > 2071/1369/3440/65536 4k (page size) jumbo clusters in use > > (current/cache/total/max) > > 0/0/0/65536 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > > 49513K/11996K/61510K bytes allocated to network (current/cache/total) > > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0/0/0 sfbufs in use (current/peak/max) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > I think there are too many mbufs in use. You're probably facing an > mbuf leakage and that causes an interface hang. > If this is a large memory machine try upping the number of clusters and mbufs. On 64 bit systems with large memories 1,000,000 mbufs is not unheard of. kern.ipc.nmbclusters: 1000000 Also, with UDP you can easily overrun different buffers within the system. You might also look at: netstat -id and see if the driver is dropping packets, and if so you might up its send queue. Best, George