From owner-freebsd-net@FreeBSD.ORG Tue Mar 12 00:39:34 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8C7B1C9F; Tue, 12 Mar 2013 00:39:34 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 3BB48CC3; Tue, 12 Mar 2013 00:39:33 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqEEAFJ4PlGDaFvO/2dsb2JhbABDiCm8O4FzdIIpAQEBBAEBASArIAsbGAICDRkCKQEJJgYIBwQBHASHcgytE5JegSOMOn00B4ItgRMDlBeCPoEej1eDJiAygQU1 X-IronPort-AV: E=Sophos;i="4.84,827,1355115600"; d="scan'208";a="18199196" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu.net.uoguelph.ca with ESMTP; 11 Mar 2013 20:39:26 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 9D5B1B3F14; Mon, 11 Mar 2013 20:39:26 -0400 (EDT) Date: Mon, 11 Mar 2013 20:39:26 -0400 (EDT) From: Rick Macklem To: Andre Oppermann Message-ID: <1475929068.3794602.1363048766615.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <513E3D75.7010803@freebsd.org> Subject: Re: Limits on jumbo mbuf cluster allocation MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-net@freebsd.org, Garrett Wollman X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Mar 2013 00:39:34 -0000 Andre Oppermann wrote: > On 11.03.2013 17:05, Garrett Wollman wrote: > > In article <513DB550.5010004@freebsd.org>, andre@freebsd.org writes: > > > >> Garrett's problem is receive side specific and NFS can't do much > >> about it. > >> Unless, of course, NFS is holding on to received mbufs for a longer > >> time. > > > > Well, I have two problems: one is running out of mbufs (caused, we > > think, by ixgbe requiring 9k clusters when it doesn't actually need > > them), and one is livelock. Allowing potentially hundreds of clients > > to queue 2 MB of requests before TCP pushes back on them helps to > > sustain the livelock once it gets started, and of course those > > packets > > will be of the 9k jumbo variety, which makes the first problem worse > > as well. > > I think that TCP, or rather the send socket buffer, currently doesn't > push back at all but simply accepts everything that gets thrown at it. > This obviously is a problem and the NFS server seems to depend > somewhat > on that by requiring atomicity on a RPC send. I have to trace the mbuf > path through NFS to the socket to be sure. The code is slightly opaque > though. > Yes, I think you are correct that when NFS sends RPC messages over TCP, they just get queued via sbappendstream(). For some reason I thought the krpc used sosend_generic(), whereas I just looked and it just uses sosend(), which does nothing except call tcp_user_send() { for TCP sockets, of course }. I tend to agree with Garrett that this is ok for the NFS server, since once it has done the work of generating a reply, why should the nfsd thread get stuck trying to queue the reply for the client. For the NFS client, it isn't quite so obvious, but after queuing a request to be sent, it will sit waiting for the reply, so it will "see" the NFS server's slow response. (Anyhow, Garrett couldn't care less about the FreeBSD NFS client, since he isn't using it.;-) rick > -- > Andre > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"