From owner-freebsd-net@FreeBSD.ORG Sun Mar 10 22:22:30 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 110E7906 for ; Sun, 10 Mar 2013 22:22:30 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 60E86331 for ; Sun, 10 Mar 2013 22:22:29 +0000 (UTC) Received: (qmail 97356 invoked from network); 10 Mar 2013 23:35:20 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 10 Mar 2013 23:35:20 -0000 Message-ID: <513D07A4.4010503@freebsd.org> Date: Sun, 10 Mar 2013 23:22:28 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: Garrett Wollman Subject: Re: Limits on jumbo mbuf cluster allocation References: <20793.36593.774795.720959@hergotha.csail.mit.edu> <51399926.6020201@freebsd.org> <20794.6692.191898.682241@hergotha.csail.mit.edu> <513A2887.2010408@freebsd.org> <20796.8842.691923.899276@hergotha.csail.mit.edu> In-Reply-To: <20796.8842.691923.899276@hergotha.csail.mit.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Jack Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2013 22:22:30 -0000 On 10.03.2013 07:04, Garrett Wollman wrote: > < said: > >> Yes, in the past the code was in this form, it should work fine Garrett, >> just make sure >> the 4K pool is large enough. > > [Andre Oppermann's patch:] >>> if (adapter->max_frame_size <= 2048) > adapter-> rx_mbuf_sz = MCLBYTES; >>> - else if (adapter->max_frame_size <= 4096) >>> + else > adapter-> rx_mbuf_sz = MJUMPAGESIZE; >>> - else if (adapter->max_frame_size <= 9216) >>> - adapter->rx_mbuf_sz = MJUM9BYTES; >>> - else >>> - adapter->rx_mbuf_sz = MJUM16BYTES; > > So I tried exactly this, and it certainly worked insofar as only 4k > clusters were allocated, but NFS performance went down precipitously > (to fewer than 100 ops/s where normally it would be doing 2000 > ops/s). I took a tcpdump while it was in this state, which I will try > to make some sense of when I get back to the office. (It wasn't > livelocked; in fact, the server was mostly idle, but responses would > take seconds rather than milliseconds -- assuming the client could > even successfully mount the server at all, which the Debian > automounter frequently refused to do.) This is very weird and unlikely to come from the 4k mbufs by itself considering they are in heavy use in the write() path. Such a high delay smells like an issue in either the driver dealing with multiple mbufs per packet or nfs having a problem with it. > I ended up reverting back to the old kernel (which I managed to lose > the sources for), and once I get my second server up, I will try to do > some more testing to see if I can identify the source of the problem. -- Andre