From owner-freebsd-stable@FreeBSD.ORG Mon Nov 28 01:41:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C2A7F106564A for ; Mon, 28 Nov 2011 01:41:02 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8C2178FC12 for ; Mon, 28 Nov 2011 01:41:02 +0000 (UTC) Received: by iakl21 with SMTP id l21so12128899iak.13 for ; Sun, 27 Nov 2011 17:41:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=oV+ck0GmfMpx8ZuDzd29taEPwwS1eVK1dAFvvIr5X28=; b=SIeFKtXSZ0ZHdZYTKMObyT1byBKhfXiYa7JjpPx8C2M/jgvS36Q8AIpzNlBWRD3q12 drda6bv+S5CgRk4c3TeeM7+QwVMtblWu68Hl1LyK0zSH7FOe7GFMuMiMvgZyxtVYvg47 4FIhtJFKoc0Ju73mjS/2RXdpzvUNiLp/d6X28= Received: by 10.42.115.136 with SMTP id k8mr21543177icq.46.1322444461851; Sun, 27 Nov 2011 17:41:01 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id z10sm39036864ibv.9.2011.11.27.17.40.59 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 27 Nov 2011 17:41:01 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Sun, 27 Nov 2011 17:39:31 -0800 From: YongHyeon PYUN Date: Sun, 27 Nov 2011 17:39:31 -0800 To: Mike Andrews Message-ID: <20111128013931.GC1830@michelle.cdnetworks.com> References: <4ED154B6.2030304@bit0.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ED154B6.2030304@bit0.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org Subject: Re: 9.0-RC2 re(4) "no memory for jumbo buffers" issue X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2011 01:41:02 -0000 On Sat, Nov 26, 2011 at 04:05:58PM -0500, Mike Andrews wrote: > I have a Supermicro 5015A-H (Intel Atom 330) server with two Realtek > RTL8111C-GR gigabit NICs on it. As far as I can tell, these support > jumbo frames up to 7422 bytes. When running them at an MTU of 5000 on Actually the maximum size is 6KB for RTL8111C, not 7422. RTL8111C and newer PCIe based gigabit controllers no longer support scattering a jumbo frame into multiple RX buffers so a single RX buffer has to receive an entire jumbo frame. This adds more burden to system because it has to allocate a jumbo frame even when it receives a pure TCP ACK. > FreeBSD 9.0-RC2, after a week or so of update, with fairly light network > activity, the interfaces die with "no memory for jumbo buffers" errors > on the console. Unloading and reloading the driver (via serial console) > doesn't help; only rebooting seems to clear it up. > The jumbo code path is the same as normal MTU sized one so I think possibility of leaking mbufs in driver is very low. And the message "no memory for jumbo RX buffers" can only happen either when you up the interface again or interface restart triggered by watchdog timeout handler. I don't think you're seeing watchdog timeouts though. When you see "no memory for jumbo RX buffers" message, did you check available mbuf pool? > I don't have this issue with any of my em(4) based systems that are also > using a 5000 byte MTU -- and they push considerably more traffic. > > I don't really consider this a regression from FreeBSD 8.2 because 8.2 > didn't support jumbos at all on this hardware... :) > > What's the best way to go about debugging this... which sysctl's should > I be looking at first? I have already tried raising kern.ipc.nmbjumbo9 > to 16384 and it doesn't seem to help things... maybe prolonging it > slightly, but not by much. The problem is it takes a week or so to > reproduce the problem each time... > I vaguely guess it could be related with other subsystem which leaks mbufs such that driver was not able to get more jumbo RX buffers from system. For instance, r228016 would be worth to try on your box. I can't clearly explain why em(4) does not suffer from the issue though.