From owner-freebsd-stable@FreeBSD.ORG Mon Jun 7 23:11:19 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 850E91065670 for ; Mon, 7 Jun 2010 23:11:19 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 53CD48FC18 for ; Mon, 7 Jun 2010 23:11:19 +0000 (UTC) Received: by pvh11 with SMTP id 11so2119204pvh.13 for ; Mon, 07 Jun 2010 16:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=y4BNAw5RLBydo6EWgcU+n7PixvbBR+3y5ioq9JbmmwQ=; b=tyVG5da2novwO8qBBTMu6aL8JuvIIDjo4G3mMZLUvEhC46n6MHsQAKKZ46si2/QmVa el8wjb8A4Di/CpKn9AdtIBTbwFmXAqBFbMDgHD5B1C7ZDE6irm7VNFR9wsTZTnF0zels VdG8rLXCF7O7S6ZAiuqZf8rj6jgfOpDpBlwrM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=P2Z9B5AaHHl+6Q34uAin/BdWjOM/SQQtJTeo/LwKcVeG+7YsIS73YyaJbp0QxrrzVH m9PE0mUGDtu1jpbt9W1QZ+cQV63DkO2mgvbCxYBM9EstPMTPmpfnDfX8iDtq8ABLRr+A dQDuTjZY7gPmjzWZzo4fRVoDm7MG6DuJAXX9k= Received: by 10.114.215.30 with SMTP id n30mr12205875wag.27.1275952278797; Mon, 07 Jun 2010 16:11:18 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id c22sm42728420wam.18.2010.06.07.16.11.18 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 07 Jun 2010 16:11:18 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 7 Jun 2010 16:09:55 -0700 From: Pyun YongHyeon Date: Mon, 7 Jun 2010 16:09:55 -0700 To: Olaf Seibert Message-ID: <20100607230955.GD1369@michelle.cdnetworks.com> References: <20100527131310.GS883@twoquid.cs.ru.nl> <20100527174211.GC1211@michelle.cdnetworks.com> <20100607140611.GX883@twoquid.cs.ru.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100607140611.GX883@twoquid.cs.ru.nl> User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org Subject: Re: nfe0 loses network connectivity (8.0-RELEASE-p2) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jun 2010 23:11:19 -0000 On Mon, Jun 07, 2010 at 04:06:11PM +0200, Olaf Seibert wrote: > On Thu 27 May 2010 at 10:42:11 -0700, Pyun YongHyeon wrote: > > On Thu, May 27, 2010 at 03:13:10PM +0200, Olaf Seibert wrote: > > > Here is the output of netstat -m while the problem was going on: > > > > > > 25751/1774/27525 mbufs in use (current/cache/total) > > > 24985/615/25600/25600 mbuf clusters in use (current/cache/total/max) > > ^^^^^^^^^^^^^^^^^^^^^ > > As Jeremy said, it seems you're hitting mbuf shortage situation. I > > think nfe(4) is dropping received frames in that case. See how many > > packets were dropped due to mbuf shortage from the output of > > "netstat -ndI nfe0". You can also use "sysctl dev.nfe.0.stats" to > > see MAC statistics maintained in nfe(4) if your MCP controller > > supports hardware MAC counters. > > The sysctl command gives me (among other figures): > > dev.nfe.0.stats.rx.drops: 338180 > > so indeed frames seem to be dropped. > > Jeremy Chadwick mentioned that one can tune kern.ipc.nmbclusters in > boot.conf, but apparently it is also changeable at runtime with sysctl. > > Since the problem recurred today, I increased the value from 25600 to > 32768, the maximum recommended value in the Handbook. (I can probably go > higher if needed; the box has 8 GB of RAM, although up to half of it is > eaten by ZFS) > > I do get the impression there is a mbuf leak somehow. On a much older > file server (FreeBSD 6.1, serves a bit of NFS but has no ZFS) the mbuf > cluster useage is much lower, despite a longer uptime: > Yeah, it surely indicates mbuf leakage in kernel. There was no mbuf leakage report in nfe(4) so I think the leakage is in other part of kernel. As Mikolaj said, it would be great if you give the latest stable/8 try and let us know whether the issue was fixed or not. > 256/634/890/25600 mbuf clusters in use (current/cache/total/max) > > Also, it shows signs that measures are taken in case of mbuf shortage: > > 2259806/466391/598621 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 1016 calls to protocol drain routines > > whereas the FreeBSD 8.0 machine has zero or very low numbers: > > 0/3956/1959 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0 calls to protocol drain routines > > and useage keeps growing: > > 26122/1782/27904/32768 mbuf clusters in use (current/cache/total/max) > > -Olaf. > --