From owner-freebsd-net@FreeBSD.ORG Thu Mar 12 21:48:45 2015 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E0DD99D0 for ; Thu, 12 Mar 2015 21:48:45 +0000 (UTC) Received: from gw.catspoiler.org (cl-1657.chi-02.us.sixxs.net [IPv6:2001:4978:f:678::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 66D8EF64 for ; Thu, 12 Mar 2015 21:48:45 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id t2CLmCNM097193; Thu, 12 Mar 2015 13:48:18 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <201503122148.t2CLmCNM097193@gw.catspoiler.org> Date: Thu, 12 Mar 2015 14:48:11 -0700 (PDT) From: Don Lewis Subject: Re: A defensive NFS server (sbwait, flow control) To: timothy.borgeaud@framestore.com In-Reply-To: MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-net@FreeBSD.org, mark.hills@framestore.com X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2015 21:48:46 -0000 On 12 Mar, Tim Borgeaud wrote: > Hi again FreeBSD folks, > > A short while ago I sent a couple of emails regarding the idea of 'fair > share' NFS scheduling. Amongst others, Garrett Wollman replied, and also > sent a related email "Implementing backpressure in the NFS server". The > common theme: to provide a level of service in situations where requests > from some clients might tie up a lot of resources. > > There are various issues to consider. We might say that we're looking at > 'defensive' functionality, and we've made some experimental progress to: > > round robin requests from different users > > provide some limits (flow control) on the number of requests read > from a single transport > > Testing has highlighted the issue that Garrett mentioned. A client can make > a set of concurrent requests and then tie up nfsd threads as they attempt > to send replies. > > To be more specific, we seem to be hitting the situation in which an nfsd > thread sits in the "sbwait" state, waiting for a reply to be drained (by a > suspended client). Other threads subsequently pick up requests for the same > transport and then queue up waiting for a lock on the same socket. > > I'm not sure of the exact situation in which the sbwait arises. It's easily > repeatable only where there are multiple concurrent requests from the same > transport. > > Our testing is fairly synthetic, but it looks like this is a situation that > has been noticed before. Having a pool of spare threads doesn't seem like a > very robust solution. In fact, if more than the server minimum get tied up, > then, if load and threads fall, we end up with _all_ remaining threads > blocked (no more nfs service). > > How to address this particular issue is not obvious to me. I assume there > are options including: > > - prevent thread blocking when sending replies > > - timeouts for sending replies (NFS or RPC level?) > > - serialize the sending of nfs/rpc replies to avoid multiple > nfsd threads waiting on the same transport. > > Does anyone have any thoughts about this? Either this particular issue or > more general direction for a 'defensive' server? If a thread blocks when sending a reply, stop reading any new requests from that transport. That should limit the number of threads that get tied up.