From owner-freebsd-net@FreeBSD.ORG Sun Mar 10 22:06:55 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DC119280 for ; Sun, 10 Mar 2013 22:06:55 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 5352B283 for ; Sun, 10 Mar 2013 22:06:54 +0000 (UTC) Received: (qmail 97275 invoked from network); 10 Mar 2013 23:19:40 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 10 Mar 2013 23:19:40 -0000 Message-ID: <513D03F7.6090206@freebsd.org> Date: Sun, 10 Mar 2013 23:06:47 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: Rick Macklem Subject: Re: Limits on jumbo mbuf cluster allocation References: <2050712270.3721724.1362790033662.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <2050712270.3721724.1362790033662.JavaMail.root@erie.cs.uoguelph.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: jfv@freebsd.org, freebsd-net@freebsd.org, Garrett Wollman , Garrett Wollman X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2013 22:06:55 -0000 On 09.03.2013 01:47, Rick Macklem wrote: > Garrett Wollman wrote: >> <> said: >> >>> [stuff I wrote deleted] >>> You have an amd64 kernel running HEAD or 9.x? >> >> Yes, these are 9.1 with some patches to reduce mutex contention on the >> NFS server's replay "cache". >> > The cached replies are copies of the mbuf list done via m_copym(). > As such, the clusters in these replies won't be free'd (ref cnt -> 0) > until the cache is trimmed (nfsrv_trimcache() gets called after the > TCP layer has received an ACK for receipt of the reply from the client). If these are not received mbufs but locally generated with m_getm2() or so they won't be jumbo mbufs > PAGE_SIZE. > If reducing the size to 4K doesn't fix the problem, you might want to > consider shrinking the tunable vfs.nfsd.tcphighwater and suffering > the increased CPU overhead (and some increased mutex contention) of > calling nfsrv_trimcache() more frequently. > (I'm assuming that you are using drc2.patch + drc3.patch. If you are > using one of ivoras@'s variants of the patch, I'm not sure if the > tunable is called the same thing, although it should have basically > the same effect.) > > Good luck with it and thanks for running on the "bleeding edge" so > these issues get identified, rick -- Andre