Date: Mon, 26 Jan 2009 11:13:19 -0800 (PST) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Rui Paulo <rpaulo@freebsd.org>, gnn@freebsd.org Cc: Liran Liss <liranl@mellanox.co.il>, freebsd-net@freebsd.org, Yony Yossef <yonyossef.lists@gmail.com>, johan@nocrew.org, Amit Krig <amitk@mellanox.co.il>, Eitan Shefi <eitans@mellanox.co.il> Subject: Re: freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available Message-ID: <618776.52345.qm@web63907.mail.re1.yahoo.com> In-Reply-To: <7iocxu7y4x.wl%gnn@neville-neil.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--- On Mon, 1/26/09, gnn@freebsd.org <gnn@freebsd.org> wrote:
> From: gnn@freebsd.org <gnn@freebsd.org>
> Subject: Re: freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available
> To: "Rui Paulo" <rpaulo@freebsd.org>
> Cc: "Liran Liss" <liranl@mellanox.co.il>, freebsd-net@freebsd.org, "Yony Yossef" <yonyossef.lists@gmail.com>, johan@nocrew.org, "Amit Krig" <amitk@mellanox.co.il>, "Eitan Shefi" <eitans@mellanox.co.il>
> Date: Monday, January 26, 2009, 12:52 PM
> At Sat, 24 Jan 2009 16:20:06 +0000,
> Rui Paulo wrote:
> >
> >
> > On 24 Jan 2009, at 12:54, Yony Yossef wrote:
> >
> > > Hi All,
> > >
> > > I'm facing a temporary network hang on my
> interfaces following a flood
> > > ping/stress udp test.
> > >
> > > I'm running a netperf UDP test which is
> giving results but does not
> > > return
> > > to the shell.
> > > client output:
> > >
> > > UDP UNIDIRECTIONAL SEND TEST from
> fe80::202:c9ff:fe02:e1fe%mtnic0
> > > (fe80::202:c9ff:fe02:e1fe) port 0 AF_INET6 to
> > > fe80::202:c9ff:fe02:e1f4%mt
> > > nic0 (fe80::202:c9ff:fe02:e1f4) port 0 AF_INET6
> > > Socket Message Elapsed Messages
> > > Size Size Time Okay Errors
> Throughput
> > > bytes bytes secs # #
> 10^6bits/sec
> > >
> > > 32768 1472 10.02 547428 1694280
> 643.60
> > > 32768 10.02 25089
> 29.50
> > >
> > >
> > > (HANG)
> > >
> > > After a minute or two it returns to the shell
> with the following
> > > message:
> > > shutdown_control: no response received errno 55
> > >
> > > 20 minutes later (!!) the interface is working
> again.
> > >
> > > netstat -m and vmstat -z outputs during the hang
> time:
> > >
> > > # netstat -m
> > > 25687/6578/32265 mbufs in use
> (current/cache/total)
> > > 17404/2438/19842/65536 mbuf clusters in use
> (current/cache/total/max)
> > > 0/1024 mbuf+clusters out of packet secondary zone
> in use (current/
> > > cache)
> > > 2071/1369/3440/65536 4k (page size) jumbo
> clusters in use
> > > (current/cache/total/max)
> > > 0/0/0/65536 9k jumbo clusters in use
> (current/cache/total/max)
> > > 0/0/0/3200 16k jumbo clusters in use
> (current/cache/total/max)
> > > 49513K/11996K/61510K bytes allocated to network
> (current/cache/total)
> > > 0/0/0 requests for mbufs denied
> (mbufs/clusters/mbuf+clusters)
> > > 0/0/0 requests for jumbo clusters denied
> (4k/9k/16k)
> > > 0/0/0 sfbufs in use (current/peak/max)
> > > 0 requests for sfbufs denied
> > > 0 requests for sfbufs delayed
> > > 0 requests for I/O initiated by sendfile
> > > 0 calls to protocol drain routines
> >
> > I think there are too many mbufs in use. You're
> probably facing an
> > mbuf leakage and that causes an interface hang.
> >
> If this is a large memory machine try upping the number of
> clusters
> and mbufs. On 64 bit systems with large memories 1,000,000
> mbufs is
> not unheard of.
>
> kern.ipc.nmbclusters: 1000000
>
> Also, with UDP you can easily overrun different buffers
> within the
> system. You might also look at:
>
> netstat -id
>
> and see if the driver is dropping packets, and if so you
> might up its
> send queue.
>
> Best,
> George
The clusters problem is an artifact of the buffers problem. You tend to
get the "No buffer space" when a send queue gets clogged. Sometimes
its difficult to get unclogged.
For example if you look at some of the ethernet drivers, they will queue
the packet when a send error occurs, hoping to send it when the problem clears. But if you have a problem that doesnt resolve those buffers get
stuck in the queue. The only clean way I've found to free them in that
case is to unload the driver and reload it.
Barney
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?618776.52345.qm>