From owner-freebsd-net@FreeBSD.ORG Fri Aug 29 22:32:37 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9FB13AB for ; Fri, 29 Aug 2014 22:32:37 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 59E1E130C for ; Fri, 29 Aug 2014 22:32:37 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s7TMWZQG086874 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 29 Aug 2014 15:32:35 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s7TMWXQG086873; Fri, 29 Aug 2014 15:32:33 -0700 (PDT) (envelope-from jmg) Date: Fri, 29 Aug 2014 15:32:33 -0700 From: John-Mark Gurney To: "Eggert, Lars" Subject: Re: igb "requests for mbufs denied" Message-ID: <20140829223233.GL71691@funkthat.com> Mail-Followup-To: "Eggert, Lars" , FreeBSD Net References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Fri, 29 Aug 2014 15:32:35 -0700 (PDT) Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Aug 2014 22:32:37 -0000 Eggert, Lars wrote this message on Thu, Aug 28, 2014 at 08:31 +0000: > no matter what value I bump kern.ipc.nmbclusters and kern.ipc.nmbufs to, I still get "requests for mbufs denied" with igb interfaces, and the occasional connection stall, even when dialing down hw.igb.num_queues=1: Can you post the exact error message? Is this from dmesg? or? > [root@laurel: ~] netstat -m > 3070/1355/4425 mbufs in use (current/cache/total) > 3069/773/3842/8144896 mbuf clusters in use (current/cache/total/max) > 3069/772 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/6/6/1018097 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/301658 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/169682 16k jumbo clusters in use (current/cache/total/max) > 6905K/1908K/8814K bytes allocated to network (current/cache/total) > 73/3831/3091 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) > 6/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile The weird thing to me is that your number of in use and other related counts are close to my machine which only has two Intel gige (em) interfaces... The only difference is that I have no denied mbufs or clusters... > The box in question has six igb interfaces, 2x 'I210 Gigabit Network Connection' and 4x '82580 Gigabit Network Connection' and is running: > > FreeBSD laurel.muccbc.hq.netapp.com 10.0-RELEASE-p7 FreeBSD 10.0-RELEASE-p7 #0: Tue Jul 8 06:37:44 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > > Anyone have any ideas what else to try? Also, what does sysctl dev.em and sysctl dev.igb show? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."