From owner-freebsd-net@FreeBSD.ORG Mon Mar 11 20:24:25 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1190D316 for ; Mon, 11 Mar 2013 20:24:25 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 7C196E90 for ; Mon, 11 Mar 2013 20:24:24 +0000 (UTC) Received: (qmail 5619 invoked from network); 11 Mar 2013 21:37:04 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 11 Mar 2013 21:37:04 -0000 Message-ID: <513E3D75.7010803@freebsd.org> Date: Mon, 11 Mar 2013 21:24:21 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: Garrett Wollman Subject: Re: Limits on jumbo mbuf cluster allocation References: <1154859394.3748712.1362959165419.JavaMail.root@erie.cs.uoguelph.ca> <201303111605.r2BG5I6v073052@hergotha.csail.mit.edu> In-Reply-To: <201303111605.r2BG5I6v073052@hergotha.csail.mit.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, rmacklem@uoguelph.ca X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2013 20:24:25 -0000 On 11.03.2013 17:05, Garrett Wollman wrote: > In article <513DB550.5010004@freebsd.org>, andre@freebsd.org writes: > >> Garrett's problem is receive side specific and NFS can't do much about it. >> Unless, of course, NFS is holding on to received mbufs for a longer time. > > Well, I have two problems: one is running out of mbufs (caused, we > think, by ixgbe requiring 9k clusters when it doesn't actually need > them), and one is livelock. Allowing potentially hundreds of clients > to queue 2 MB of requests before TCP pushes back on them helps to > sustain the livelock once it gets started, and of course those packets > will be of the 9k jumbo variety, which makes the first problem worse > as well. I think that TCP, or rather the send socket buffer, currently doesn't push back at all but simply accepts everything that gets thrown at it. This obviously is a problem and the NFS server seems to depend somewhat on that by requiring atomicity on a RPC send. I have to trace the mbuf path through NFS to the socket to be sure. The code is slightly opaque though. -- Andre