From owner-freebsd-net@FreeBSD.ORG Sat Mar 9 01:40:03 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 95E148D8 for ; Sat, 9 Mar 2013 01:40:03 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 37E2436A for ; Sat, 9 Mar 2013 01:40:03 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r291e1kw010071; Fri, 8 Mar 2013 20:40:01 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r291e1O7010068; Fri, 8 Mar 2013 20:40:01 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20794.37617.822910.93537@hergotha.csail.mit.edu> Date: Fri, 8 Mar 2013 20:40:01 -0500 From: Garrett Wollman To: Rick Macklem Subject: Re: Limits on jumbo mbuf cluster allocation In-Reply-To: <2050712270.3721724.1362790033662.JavaMail.root@erie.cs.uoguelph.ca> References: <20794.7012.265887.99878@hergotha.csail.mit.edu> <2050712270.3721724.1362790033662.JavaMail.root@erie.cs.uoguelph.ca> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Fri, 08 Mar 2013 20:40:01 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 01:40:03 -0000 < said: > If reducing the size to 4K doesn't fix the problem, you might want to > consider shrinking the tunable vfs.nfsd.tcphighwater and suffering > the increased CPU overhead (and some increased mutex contention) of > calling nfsrv_trimcache() more frequently. Can't do that -- the system becomes intolerably slow when it gets into that state, and seems to get stuck that way, such that the only way to restore performance is to increase the size of the "cache". (Essentially all of the nfsd service threads end up spinning most of the time, load average goes to N, and goodput goes to nearly nil.) It does seem like a lot of effort for an extreme edge case that, in practical terms, never happens. > (I'm assuming that you are using drc2.patch + drc3.patch. I believe that's what I have. If my kernel coding skills were less rusty, I'd fix it to have a separate cache-trimming thread. One other weird thing that I've noticed is that netstat(1) reports the send and receive queues on NFS connections as being far higher than I have the limits configured. Does NFS do something to override this? -GAWollman