From owner-freebsd-fs@freebsd.org Wed Mar 17 01:32:29 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1D55A5757C3 for ; Wed, 17 Mar 2021 01:32:29 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0XhY0Gtvz4Sx6 for ; Wed, 17 Mar 2021 01:32:28 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f51.google.com with SMTP id h6-20020a0568300346b02901b71a850ab4so290946ote.6 for ; Tue, 16 Mar 2021 18:32:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=b6UdMGq9B5EqhM2b14RACWAp4KJXWd+Q+x1zS3ZnXBo=; b=Bod4yDcWc36UnbpunlCkPtBFpX47x0Mz4wv8zBoVTh3c5l7oW9XpM6iJxNb9QsCF74 /8AzT7Ktfox47IXQUTnQHx99GuFGOhe8qhc7k7zj4oiipxevOus2QvnTkVyYlnRZ83i4 7ssAxfNq96tB4VDWriqbprSdREfaAiKepHe679+bt/nX3ZnsTTAyykCsH//1saIZY1dC 2BP0XEGEihFoRxW0UmT+lsZWqz3KOZkFkXu6YNub/j2lhFfFsCHKb2db1+CCYZORghK5 JQGspTklR3F0qcR+C+8u3gatjwidLljG/3DhtJovAyISNmloV8MBLCJobDqiaai6p2B6 N1Kg== X-Gm-Message-State: AOAM532GLT1v/f85Fm2imLfgKxRXD8BgQv+hwsW+HWyfIyQKLGOaCgAF JpKPnvPsomQa6ZysbAFUNM8lj6XYR9NtGACOfLs= X-Google-Smtp-Source: ABdhPJxAANVPfNKWdM8oeRBwa/M4Z/9awf70F1qcD08RF5v+rla+o/jrUZeSAo5fBoycqkjYGNL/AXIJ24DQEHGrM1k= X-Received: by 2002:a05:6830:148c:: with SMTP id s12mr1334689otq.251.1615944747110; Tue, 16 Mar 2021 18:32:27 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Tue, 16 Mar 2021 19:32:16 -0600 Message-ID: Subject: Re: Understanding the vfs.nfsd.request_space* sysctls To: Rick Macklem Cc: freebsd-fs X-Rspamd-Queue-Id: 4F0XhY0Gtvz4Sx6 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 01:32:29 -0000 On Tue, Mar 16, 2021 at 6:06 PM Rick Macklem wrote: > Alan Somers wrote: > >Could somebody help me understand the meaning of the various > >vfs.nfsd.request_space* sysctls, and what implication they have for tuning > >NFS? > > > >I have several NFS servers. Most are performing well, but one with a > >metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, > GetAttr > >and Lookup ops) is not. Total throughput is a not-unreasonable several > >hundred MB/s, but some clients are limited to very slow speeds, sometimes > >writing at < 10 MB/s. > > > >The request_space sysctls show some pretty stark divergence between the > >well-performing and poor-performing servers: > > > >sysctl well-performing poor-performing > >request_space_used < 6 M varies, but > >currently 40 M > >request_space_used_highest < 39 M 24 G > >request_space_throttle_count 0 35 > > > >So what does request_space_used measure, anyway? > It's the number of bytes of request message(s) received, > but not yet being processed by nfsd threads. > (They're actually in sys/rpc/svc.c, although named for the > nfs server.) > > Anytime used > high, it throttles, which means it leaves > the RPC messages on the socket receive queue. > This indicates the server is not keeping up with requests. > (ie Overloaded or ???) > That all makes sense. So having a high request_space_used basically just means that my storage is too slow. > > > And how can I either > >increase the available space or decrease the stuff that's using it? > Increasing vfs.nfsd.request_space_high avoids the throttling, > but it is hard to say that is a good idea, since there won't be > backpressure applied to the clients via TCP windows, etc. > > Fixing the server so that it isn't overloaded would be better, > I think? > --> I'm the last guy to take ZFS advice from, but I think there > is a tunable w.r.t. how much arc is used for metadata. > Getattrs will need cached (metadata) to reply quickly. > Lookups also depend on cached attributes for good > perf. as well. > I've added an L2ARC since I sent that original email. We'll see how well it works. I expect it to take 48 hours before results are apparent. > --> Make sure you have lots of nfsd threads. They are > just kernel threads, so they don't use a lot of resources > and having too many is much better than not enough. > -->I can't remember what the upper limit is these days, > but I'd set it to that for a busy nfs server. > For NFSv4.1, each client can send a maximum of > session_slot_table_size concurrent RPCs. FreeBSD > uses a single 64slot table for each mount. I think > Linux uses a 32slot table, but supports trunking. > I don't know if the Linux client uses a separate > session table for each trunk or not? > --> Something like #clients * 32 (or 64) nfsd > threads running on the server. > Experimentally, I determined that 768 threads is sufficient. But your formula would suggest > 2500, which is a lot of threads! I'll keep it in mind if I ever reach 768, though. > > rick > > -Alan > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >