From owner-freebsd-net@FreeBSD.ORG Mon Jan 26 19:13:20 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C49B106564A for ; Mon, 26 Jan 2009 19:13:20 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63907.mail.re1.yahoo.com (web63907.mail.re1.yahoo.com [69.147.97.122]) by mx1.freebsd.org (Postfix) with SMTP id 162D08FC17 for ; Mon, 26 Jan 2009 19:13:19 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: (qmail 56285 invoked by uid 60001); 26 Jan 2009 19:13:19 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Message-ID; b=qRMhsdZ797o/Roov0u/bRWMcxH10gnvrjYvBHroQjz7nhl65kQE25NcWgNqGKMvz6+xyuaps2CWHsxjeD3IUcEYuXele6t6HZ7fpwGHHjz4J25sk3VNQn2+jv3VxF8cnfiZoLbbTaM+dmlU+XKsx1urM3vjdPzGmXoSK4P7DSbk=; X-YMail-OSG: vAu5FnEVM1lX967bNTMzVlhshwB0SkNIm9LP7pTKwAKR8XV5AaKojE7pHNMBIh4aiue75yG.oeIN0uWiqbH9Fi6V5WJTKMU9_QyNouQDdJIuvqhEFMfv23LNjvDIYunRCZLsYXwiAzK7qOdmjeB5osqJEK9nQgllvJGrVrBQtu46ig_H8j0wlrf2yh8ZSMMd Received: from [98.242.222.229] by web63907.mail.re1.yahoo.com via HTTP; Mon, 26 Jan 2009 11:13:19 PST X-Mailer: YahooMailWebService/0.7.260.1 Date: Mon, 26 Jan 2009 11:13:19 -0800 (PST) From: Barney Cordoba To: Rui Paulo , gnn@freebsd.org In-Reply-To: <7iocxu7y4x.wl%gnn@neville-neil.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <618776.52345.qm@web63907.mail.re1.yahoo.com> Cc: Liran Liss , freebsd-net@freebsd.org, Yony Yossef , johan@nocrew.org, Amit Krig , Eitan Shefi Subject: Re: freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: barney_cordoba@yahoo.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jan 2009 19:13:20 -0000 --- On Mon, 1/26/09, gnn@freebsd.org wrote: > From: gnn@freebsd.org > Subject: Re: freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available > To: "Rui Paulo" > Cc: "Liran Liss" , freebsd-net@freebsd.org, "Yony Yossef" , johan@nocrew.org, "Amit Krig" , "Eitan Shefi" > Date: Monday, January 26, 2009, 12:52 PM > At Sat, 24 Jan 2009 16:20:06 +0000, > Rui Paulo wrote: > > > > > > On 24 Jan 2009, at 12:54, Yony Yossef wrote: > > > > > Hi All, > > > > > > I'm facing a temporary network hang on my > interfaces following a flood > > > ping/stress udp test. > > > > > > I'm running a netperf UDP test which is > giving results but does not > > > return > > > to the shell. > > > client output: > > > > > > UDP UNIDIRECTIONAL SEND TEST from > fe80::202:c9ff:fe02:e1fe%mtnic0 > > > (fe80::202:c9ff:fe02:e1fe) port 0 AF_INET6 to > > > fe80::202:c9ff:fe02:e1f4%mt > > > nic0 (fe80::202:c9ff:fe02:e1f4) port 0 AF_INET6 > > > Socket Message Elapsed Messages > > > Size Size Time Okay Errors > Throughput > > > bytes bytes secs # # > 10^6bits/sec > > > > > > 32768 1472 10.02 547428 1694280 > 643.60 > > > 32768 10.02 25089 > 29.50 > > > > > > > > > (HANG) > > > > > > After a minute or two it returns to the shell > with the following > > > message: > > > shutdown_control: no response received errno 55 > > > > > > 20 minutes later (!!) the interface is working > again. > > > > > > netstat -m and vmstat -z outputs during the hang > time: > > > > > > # netstat -m > > > 25687/6578/32265 mbufs in use > (current/cache/total) > > > 17404/2438/19842/65536 mbuf clusters in use > (current/cache/total/max) > > > 0/1024 mbuf+clusters out of packet secondary zone > in use (current/ > > > cache) > > > 2071/1369/3440/65536 4k (page size) jumbo > clusters in use > > > (current/cache/total/max) > > > 0/0/0/65536 9k jumbo clusters in use > (current/cache/total/max) > > > 0/0/0/3200 16k jumbo clusters in use > (current/cache/total/max) > > > 49513K/11996K/61510K bytes allocated to network > (current/cache/total) > > > 0/0/0 requests for mbufs denied > (mbufs/clusters/mbuf+clusters) > > > 0/0/0 requests for jumbo clusters denied > (4k/9k/16k) > > > 0/0/0 sfbufs in use (current/peak/max) > > > 0 requests for sfbufs denied > > > 0 requests for sfbufs delayed > > > 0 requests for I/O initiated by sendfile > > > 0 calls to protocol drain routines > > > > I think there are too many mbufs in use. You're > probably facing an > > mbuf leakage and that causes an interface hang. > > > If this is a large memory machine try upping the number of > clusters > and mbufs. On 64 bit systems with large memories 1,000,000 > mbufs is > not unheard of. > > kern.ipc.nmbclusters: 1000000 > > Also, with UDP you can easily overrun different buffers > within the > system. You might also look at: > > netstat -id > > and see if the driver is dropping packets, and if so you > might up its > send queue. > > Best, > George The clusters problem is an artifact of the buffers problem. You tend to get the "No buffer space" when a send queue gets clogged. Sometimes its difficult to get unclogged. For example if you look at some of the ethernet drivers, they will queue the packet when a send error occurs, hoping to send it when the problem clears. But if you have a problem that doesnt resolve those buffers get stuck in the queue. The only clean way I've found to free them in that case is to unload the driver and reload it. Barney