From owner-freebsd-fs@freebsd.org Tue Mar 16 17:27:51 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CFDB85B61DB for ; Tue, 16 Mar 2021 17:27:51 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0KxM1GLgz4rLg for ; Tue, 16 Mar 2021 17:27:50 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f178.google.com with SMTP id u198so33945519oia.4 for ; Tue, 16 Mar 2021 10:27:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=uwP1LCsefw81DoC0u/ui0qf8kKoVXJmMaf5nObab0tM=; b=TXAFiqNVXSPjAsXOj+y5ll37a7DQ1Cio5OABY5vpH1xU+JV3REVZW5Fu4r6qPxrCx4 QbRekPXlctsgTDGiQes0uU7hTnf/ywur/dXC4WdCJh/FLtxhWhGoXIoV6G0pU1DUWp4V pLNh6A/jehwFPnh9e+uPJTtamVaK9KVGxqt23uImqs2AROUwj39tWtpqo5LX3+dzCsxe YXUM4JkpgZXM+4Vgu1MH/+ElBpa9yZvLYOiXchXSz2X47bosRCeX4xtz/YHzrf6stn3T twsXDjj7+NHawhYFlh2Kvc2QG+VndXFCBsC6B0SChsbpWauDFpim2bd5HSwnEi8Cji0k MqjA== X-Gm-Message-State: AOAM532NE8cB0UIEliF8Yj9E8T//oTe1Lr7n0pNcV7F+8HTguCzFqOpW lQ4iTxVc/dJ84EXGUdYT/2AjactZYU/4bw5RY5ZdMaFs X-Google-Smtp-Source: ABdhPJzLm1+CFgIDkCDu/TIwc6xA7FjjDR98U5P+u///m9upHVGCNBB7+xy4p7lDuH7iJcwgP1XN2FX/WSKxHYoT/Eg= X-Received: by 2002:aca:3046:: with SMTP id w67mr98669oiw.57.1615915669713; Tue, 16 Mar 2021 10:27:49 -0700 (PDT) MIME-Version: 1.0 From: Alan Somers Date: Tue, 16 Mar 2021 11:27:39 -0600 Message-ID: Subject: Understanding the vfs.nfsd.request_space* sysctls To: freebsd-fs X-Rspamd-Queue-Id: 4F0KxM1GLgz4rLg X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.178 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.167.178:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[asomers]; R_DKIM_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[209.85.167.178:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.178:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.178:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 17:27:51 -0000 Could somebody help me understand the meaning of the various vfs.nfsd.request_space* sysctls, and what implication they have for tuning NFS? I have several NFS servers. Most are performing well, but one with a metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAttr and Lookup ops) is not. Total throughput is a not-unreasonable several hundred MB/s, but some clients are limited to very slow speeds, sometimes writing at < 10 MB/s. The request_space sysctls show some pretty stark divergence between the well-performing and poor-performing servers: sysctl well-performing poor-performing request_space_used < 6 M varies, but currently 40 M request_space_used_highest < 39 M 24 G request_space_throttle_count 0 35 So what does request_space_used measure, anyway? And how can I either increase the available space or decrease the stuff that's using it? -Alan