From owner-freebsd-net@FreeBSD.ORG Thu Mar 12 14:04:48 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9658D70B for ; Thu, 12 Mar 2015 14:04:48 +0000 (UTC) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2AE2DD3C for ; Thu, 12 Mar 2015 14:04:46 +0000 (UTC) Received: by wiwl15 with SMTP id l15so47981524wiw.4 for ; Thu, 12 Mar 2015 07:04:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=zPpKVJ6e3BkptZBCVeDNuEVFWjSQIFiVfZtcjXmHjnc=; b=cm5dCb0Y8gP7pRCbiDihNMBETFoQAnGV8amjEEVB/rEUY77wrBS1oR5JHhVoeO67lx qRiRD1qUIGKBQ0o0eKcIX32t1AJmK4fUFVQFGaqtCkPRJIS3LNYiKg2h0DOf2BQq1OEj 5m9987MzXvbebai6xlyTkZyChGIidhM3P98xrvgKHLcno8A9NfoEZrpWP1H8Vhwf49Mq qHovqI9RXAWhmwM9q7NLu7xFLzRe2ewhV/ttUzRUhQRx5Mf+qrKP7EHNCre4jfvp8AW9 XpRcgVyMmPqQrytOPQi2HMcM/zMQ3t+wt2ymbnDh/byuSM5F6rXP/+vLA4SpnbGJntk3 PSLg== X-Gm-Message-State: ALoCoQk0xSNvfS0CozzaMDS7Yzv67F7v0+mYL4nTstW8RpdpX1xXwwlcam+56sO9OnuYX+7TByEp X-Received: by 10.194.93.165 with SMTP id cv5mr6075976wjb.24.1426169078969; Thu, 12 Mar 2015 07:04:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.27.126.132 with HTTP; Thu, 12 Mar 2015 07:04:18 -0700 (PDT) From: Tim Borgeaud Date: Thu, 12 Mar 2015 14:04:18 +0000 Message-ID: Subject: A defensive NFS server (sbwait, flow control) To: freebsd-net@freebsd.org, Mark Hills Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2015 14:04:48 -0000 Hi again FreeBSD folks, A short while ago I sent a couple of emails regarding the idea of 'fair share' NFS scheduling. Amongst others, Garrett Wollman replied, and also sent a related email "Implementing backpressure in the NFS server". The common theme: to provide a level of service in situations where requests from some clients might tie up a lot of resources. There are various issues to consider. We might say that we're looking at 'defensive' functionality, and we've made some experimental progress to: round robin requests from different users provide some limits (flow control) on the number of requests read from a single transport Testing has highlighted the issue that Garrett mentioned. A client can make a set of concurrent requests and then tie up nfsd threads as they attempt to send replies. To be more specific, we seem to be hitting the situation in which an nfsd thread sits in the "sbwait" state, waiting for a reply to be drained (by a suspended client). Other threads subsequently pick up requests for the same transport and then queue up waiting for a lock on the same socket. I'm not sure of the exact situation in which the sbwait arises. It's easily repeatable only where there are multiple concurrent requests from the same transport. Our testing is fairly synthetic, but it looks like this is a situation that has been noticed before. Having a pool of spare threads doesn't seem like a very robust solution. In fact, if more than the server minimum get tied up, then, if load and threads fall, we end up with _all_ remaining threads blocked (no more nfs service). How to address this particular issue is not obvious to me. I assume there are options including: - prevent thread blocking when sending replies - timeouts for sending replies (NFS or RPC level?) - serialize the sending of nfs/rpc replies to avoid multiple nfsd threads waiting on the same transport. Does anyone have any thoughts about this? Either this particular issue or more general direction for a 'defensive' server? -- Tim Borgeaud Systems Developer