From owner-freebsd-net@FreeBSD.ORG Sun Mar 10 06:05:01 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3A012E3A for ; Sun, 10 Mar 2013 06:05:01 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id CE01AE02 for ; Sun, 10 Mar 2013 06:04:59 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r2A64wWI069227; Sun, 10 Mar 2013 01:04:58 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r2A64w19069224; Sun, 10 Mar 2013 01:04:58 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20796.8842.691923.899276@hergotha.csail.mit.edu> Date: Sun, 10 Mar 2013 01:04:58 -0500 From: Garrett Wollman To: Jack Vogel Subject: Re: Limits on jumbo mbuf cluster allocation In-Reply-To: References: <20793.36593.774795.720959@hergotha.csail.mit.edu> <51399926.6020201@freebsd.org> <20794.6692.191898.682241@hergotha.csail.mit.edu> <513A2887.2010408@freebsd.org> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Sun, 10 Mar 2013 01:04:59 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2013 06:05:01 -0000 < said: > Yes, in the past the code was in this form, it should work fine Garrett, > just make sure > the 4K pool is large enough. [Andre Oppermann's patch:] >> if (adapter->max_frame_size <= 2048) adapter-> rx_mbuf_sz = MCLBYTES; >> - else if (adapter->max_frame_size <= 4096) >> + else adapter-> rx_mbuf_sz = MJUMPAGESIZE; >> - else if (adapter->max_frame_size <= 9216) >> - adapter->rx_mbuf_sz = MJUM9BYTES; >> - else >> - adapter->rx_mbuf_sz = MJUM16BYTES; So I tried exactly this, and it certainly worked insofar as only 4k clusters were allocated, but NFS performance went down precipitously (to fewer than 100 ops/s where normally it would be doing 2000 ops/s). I took a tcpdump while it was in this state, which I will try to make some sense of when I get back to the office. (It wasn't livelocked; in fact, the server was mostly idle, but responses would take seconds rather than milliseconds -- assuming the client could even successfully mount the server at all, which the Debian automounter frequently refused to do.) I ended up reverting back to the old kernel (which I managed to lose the sources for), and once I get my second server up, I will try to do some more testing to see if I can identify the source of the problem. -GAWollman