From owner-freebsd-stable@FreeBSD.ORG Fri Dec 30 21:47:59 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 62D3F106564A for ; Fri, 30 Dec 2011 21:47:59 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 25ACF8FC12 for ; Fri, 30 Dec 2011 21:47:58 +0000 (UTC) Received: by iadj38 with SMTP id j38so32786465iad.13 for ; Fri, 30 Dec 2011 13:47:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=fvRC5wc+e6dciMUFLvfRWuT4Gq44uhGJAgURhWN2hM4=; b=amHKX063X7wpViEzQHqqEpCtj3XPyVO5LdF30fE3ZV9CaAqPqnrWdESQG5hdJ/UfaU y+P/WVkv2D+5wef7x8baCggcQ8w34lkdRf8uGDFR9fjqONMs9H6BsvqI/IXiHUUhAEwX wBWiIO0bkPmkTEa0vVfD+ZFoM9PAADqKONNKc= Received: by 10.42.147.72 with SMTP id m8mr43176800icv.56.1325281678582; Fri, 30 Dec 2011 13:47:58 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id i2sm56738856igq.6.2011.12.30.13.47.56 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 30 Dec 2011 13:47:57 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Fri, 30 Dec 2011 13:46:03 -0800 From: YongHyeon PYUN Date: Fri, 30 Dec 2011 13:46:03 -0800 To: Mike Andrews Message-ID: <20111230214603.GC14244@michelle.cdnetworks.com> References: <4ED154B6.2030304@bit0.com> <20111128013931.GC1830@michelle.cdnetworks.com> <4ED40D58.1030107@bit0.com> <20111128234212.GC1655@michelle.cdnetworks.com> <4EFD353D.1060900@bit0.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="BOKacYhQ+x31HxR3" Content-Disposition: inline In-Reply-To: <4EFD353D.1060900@bit0.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org Subject: Re: 9.0-RC2 re(4) "no memory for jumbo buffers" issue X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Dec 2011 21:47:59 -0000 --BOKacYhQ+x31HxR3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Dec 29, 2011 at 10:51:25PM -0500, Mike Andrews wrote: > On 11/28/2011 6:42 PM, YongHyeon PYUN wrote: > >On Mon, Nov 28, 2011 at 05:38:16PM -0500, Mike Andrews wrote: > >>On 11/27/11 8:39 PM, YongHyeon PYUN wrote: > >>>On Sat, Nov 26, 2011 at 04:05:58PM -0500, Mike Andrews wrote: > >>>>I have a Supermicro 5015A-H (Intel Atom 330) server with two Realtek > >>>>RTL8111C-GR gigabit NICs on it. As far as I can tell, these support > >>>>jumbo frames up to 7422 bytes. When running them at an MTU of 5000 on > >>>Actually the maximum size is 6KB for RTL8111C, not 7422. > >>>RTL8111C and newer PCIe based gigabit controllers no longer support > >>>scattering a jumbo frame into multiple RX buffers so a single RX > >>>buffer has to receive an entire jumbo frame. This adds more burden > >>>to system because it has to allocate a jumbo frame even when it > >>>receives a pure TCP ACK. > >>OK, that makes sense. > >> > >>>>FreeBSD 9.0-RC2, after a week or so of update, with fairly light network > >>>>activity, the interfaces die with "no memory for jumbo buffers" errors > >>>>on the console. Unloading and reloading the driver (via serial console) > >>>>doesn't help; only rebooting seems to clear it up. > >>>> > >>>The jumbo code path is the same as normal MTU sized one so I think > >>>possibility of leaking mbufs in driver is very low. And the > >>>message "no memory for jumbo RX buffers" can only happen either > >>>when you up the interface again or interface restart triggered by > >>>watchdog timeout handler. I don't think you're seeing watchdog > >>>timeouts though. > >>I'm fairly certain the interface isn't changing state when this happens > >>-- it just kinda spontaneously happens after a week or two, with no > >>interface up/down transitions. I don't see any watchdog messages when > >>this happens. > >There is another code path that causes controller reinitialization. > >If you change MTU or offloading configuration(TSO, VLAN tagging, > >checksum offloading etc) it will reinitialize the controller. So do > >you happen to trigger one of these code path during a week or two? > > > >>>When you see "no memory for jumbo RX buffers" message, did you > >>>check available mbuf pool? > >>Not yet, that's why I asked for debugging tips -- I'll do that the next > >>time this happens. > >> > >>>>What's the best way to go about debugging this... which sysctl's should > >>>>I be looking at first? I have already tried raising kern.ipc.nmbjumbo9 > >>>>to 16384 and it doesn't seem to help things... maybe prolonging it > >>>>slightly, but not by much. The problem is it takes a week or so to > >>>>reproduce the problem each time... > >>>> > >>>I vaguely guess it could be related with other subsystem which > >>>leaks mbufs such that driver was not able to get more jumbo RX > >>>buffers from system. For instance, r228016 would be worth to try on > >>>your box. I can't clearly explain why em(4) does not suffer from > >>>the issue though. > >>I've just this morning built a kernel with that fix, so we'll see how > >>that goes. > >Ok. > > OK, this just happened again with a 9.0-RC3 kernel rev r228247. > > > whitedog# ifconfig re0 down;ifconfig re0 up;ifconfig re1 down;ifconfig Ah, sorry. I should have spotted this issue earlier. Try attached patch and let me know whether it makes any difference. > re1 up > re0: no memory for jumbo RX buffers > re1: no memory for jumbo RX buffers > whitedog# netstat -m > 526/1829/2355 mbufs in use (current/cache/total) > 0/1278/1278/25600 mbuf clusters in use (current/cache/total/max) > 0/356 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/336/336/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 512/385/897/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 4739K/7822K/12561K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/4560/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines --BOKacYhQ+x31HxR3 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="re.jumbobuf.diff" Index: sys/dev/re/if_re.c =================================================================== --- sys/dev/re/if_re.c (revision 229006) +++ sys/dev/re/if_re.c (working copy) @@ -3558,7 +3558,6 @@ } /* Free the TX list buffers. */ - for (i = 0; i < sc->rl_ldata.rl_tx_desc_cnt; i++) { txd = &sc->rl_ldata.rl_tx_desc[i]; if (txd->tx_m != NULL) { @@ -3572,11 +3571,10 @@ } /* Free the RX list buffers. */ - for (i = 0; i < sc->rl_ldata.rl_rx_desc_cnt; i++) { rxd = &sc->rl_ldata.rl_rx_desc[i]; if (rxd->rx_m != NULL) { - bus_dmamap_sync(sc->rl_ldata.rl_tx_mtag, + bus_dmamap_sync(sc->rl_ldata.rl_rx_mtag, rxd->rx_dmamap, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(sc->rl_ldata.rl_rx_mtag, rxd->rx_dmamap); @@ -3584,6 +3582,20 @@ rxd->rx_m = NULL; } } + + if ((sc->rl_flags & RL_FLAG_JUMBOV2) != 0) { + for (i = 0; i < sc->rl_ldata.rl_rx_desc_cnt; i++) { + rxd = &sc->rl_ldata.rl_jrx_desc[i]; + if (rxd->rx_m != NULL) { + bus_dmamap_sync(sc->rl_ldata.rl_jrx_mtag, + rxd->rx_dmamap, BUS_DMASYNC_POSTREAD); + bus_dmamap_unload(sc->rl_ldata.rl_jrx_mtag, + rxd->rx_dmamap); + m_freem(rxd->rx_m); + rxd->rx_m = NULL; + } + } + } } /* --BOKacYhQ+x31HxR3--