From owner-freebsd-fs@freebsd.org Sun Mar 14 08:35:40 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3BEC5577AAA for ; Sun, 14 Mar 2021 08:35:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DytDC0Hctz3Cwl for ; Sun, 14 Mar 2021 08:35:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 12E8ZPZU019088 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 14 Mar 2021 10:35:28 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 12E8ZPZU019088 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 12E8ZOt8019080; Sun, 14 Mar 2021 10:35:24 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 14 Mar 2021 10:35:24 +0200 From: Konstantin Belousov To: Alexander Lochmann Cc: freebsd-fs@freebsd.org Subject: Re: [RFC] Understanding the locking of struct buf Message-ID: References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de> <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DytDC0Hctz3Cwl X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; R_SPF_SOFTFAIL(0.00)[~all]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2021 08:35:40 -0000 On Sat, Mar 13, 2021 at 05:29:39PM +0100, Alexander Lochmann wrote: > > > On 13.03.21 13:59, Konstantin Belousov wrote: > > What consistency? If you are talking about multithreading memory model > > as expected by the FreeBSD kernel, look at atomic(9). It has assumptions > > like atomicity of naturally-aligned word-sized integer accesses written > > out explicitly. > > > Yeah, I think that's what I meant. Thx! > > In sys/sys/buf.h, it says "All fields are protected by the buffer lock > except those marked [...]". How does this fit together with atomic(9)? > If I had to implement some extension to vfs_bio.c, for example, I would > follow those locking rules without knowing that I may access some fields > without any lock. You cannot write any working code that uses buffer cache, without studying both buffer cache code itself, and examples using it from filesystems. Short (even the whole-screen short) herald comment cannot explain all the nuances that grown in the 40+ years codebase. > Shouldn't that particular piece of documentation be updated? > For example: b_bcount /* w: b_lock, r: no lock needed */ Do you understand the purpose of the locking? What do you intent to do with the read value of b_bcount, if you do not own the buffer? What is you next action with that value? > >> In FreeBSD, do I have to use lock X in any case except Y and Z? > >> Or is it the other way round: Do I need no lock at all except for case X > >> and Y? > > I do not understand this question(?). > > > I'm sry. Maybe my question is just nuts.>> > >>> Are you reporting a bug or just asking about LK_KERNPROC. Lockmgr > >>> (kern_lock.c)is careful enough to handle LK_KERNPROC owner specially. In > >>> particular, it does not report unlock of such lock to witness. > >> First of all, I want to understand how things in FreeBSD work. > >> >From what I understand now: When setting up an asynchronous IO request, > >> buf.b_lock is taken by thread X. Later on LK_KERNPROC is used to hand > >> over the ownership to the kernel. The lock itself is still acquired. > > The lock is acquired in the sense that nobody else can take the buffer' > > lock until the call to lockmgr(LK_RELEASE). > So can we safely assume, for our analysis, that the b_lock is still > acquired during bufdone and the other related fns? LK_KERNPROC buffer is logically owned by the io completion thread(s). > But from this point, there > > is no thread owning the lock. Consider that the lock was converted to > > the 1-counting semaphore.I'm not sure. What do you mean bei 1-counting semaphore? > > > >> The completion of the IO job is performed in the kernel's context, which > >> releases buf.b_lock in brelse(). > > The completion for async IO is performed in the context of some other > > thread, typically either geom io up thread, or direct completion thread > > of some disk driver. This is why this somewhat strange LK_KERNPROC > > business is needed at all. > > > > For sync IO, that thread only signals original thread that io completed. > > In this case, no LK_KERNPROC trick is performed. > > > That is clear to me. > >> So there is no explicit lock call in the latter context, is it? > > No lock call, but there is an unlock. > Are you talking about BUF_UNLOCK()? > > > Assuming you wrote the stack bottom-up, this is exactly what I wrote above: > > xpt_done_td is CAM IO completion thread, and it performs actions after hw > > informed that io request (bio) was completed. > > > Thx. > >> allocbuf > > When brelse() notes that buffer was marked as 'no cache', it demolishes > > the buffer right after async io finishes. Perhaps this is the case that > > you observed. > > Yeah, but maybe our approach is just inaccurate. Due to the > WITNESS_UNLOCK() call in __lockmgr_disown(), we assume the b_lock has > already been released. > > - Alex > > -- > Technische Universität Dortmund > Alexander Lochmann PGP key: 0xBC3EF6FD > Otto-Hahn-Str. 16 phone: +49.231.7556141 > D-44227 Dortmund fax: +49.231.7556116 > http://ess.cs.tu-dortmund.de/Staff/al From owner-freebsd-fs@freebsd.org Sun Mar 14 21:00:24 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 757155B26BB for ; Sun, 14 Mar 2021 21:00:24 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DzBlX1Q9qz4l9N for ; Sun, 14 Mar 2021 21:00:24 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 0AFBF5B23FA; Sun, 14 Mar 2021 21:00:24 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 07C025B2719 for ; Sun, 14 Mar 2021 21:00:24 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DzBlW3d9tz4lCk for ; Sun, 14 Mar 2021 21:00:23 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 88F6723F58 for ; Sun, 14 Mar 2021 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12EL0M1q024926 for ; Sun, 14 Mar 2021 21:00:22 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12EL0Mi7024925 for fs@FreeBSD.org; Sun, 14 Mar 2021 21:00:22 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202103142100.12EL0Mi7024925@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 14 Mar 2021 21:00:22 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2021 21:00:25 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS Open | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 240831 | zfs: Panic during snapshot on 12.1-STABLE r352648 Open | 243973 | [zfs] rollback segmentation fault Open | 244656 | zfs: resilver doesn't provide enough information Open | 244692 | gjournal: Does not support TRIM Open | 244899 | [PATCH] zfs: xattr on a symlink target > 136 caus Open | 251035 | [zfs] Allow 64 bit ZFS to support 32 bit ioctls ( 11 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Tue Mar 16 00:07:52 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4255856CC59 for ; Tue, 16 Mar 2021 00:07:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DztsN1Cx6z4fCX for ; Tue, 16 Mar 2021 00:07:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 27FBA56CCD0; Tue, 16 Mar 2021 00:07:52 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 27C4656CC58 for ; Tue, 16 Mar 2021 00:07:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DztsN0Wpnz4f9C for ; Tue, 16 Mar 2021 00:07:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 04FB719D1F for ; Tue, 16 Mar 2021 00:07:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12G07pNH055976 for ; Tue, 16 Mar 2021 00:07:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12G07pgJ055975 for fs@FreeBSD.org; Tue, 16 Mar 2021 00:07:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 253158] Panic: snapacct_ufs2: bad block - Non-suJ mksnap_ffs(8) crash Date: Tue, 16 Mar 2021 00:07:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 00:07:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253158 --- Comment #55 from commit-hook@FreeBSD.org --- A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3Dcf0310dfefee8672680fb45b7ee25722e= 7630227 commit cf0310dfefee8672680fb45b7ee25722e7630227 Author: Kirk McKusick AuthorDate: 2021-02-12 05:31:16 +0000 Commit: Kirk McKusick CommitDate: 2021-03-16 00:11:29 +0000 Fix bug 253158 - Panic: snapacct_ufs2: bad block - mksnap_ffs(8) crash PR: 253158 (cherry picked from commit 8563de2f2799b2cb6f2f06e3c9dddd48dca2a986) (cherry picked from commit c31480a1f66537e59b02e935a547bcfc76715278) sys/ufs/ffs/ffs_snapshot.c | 145 ++++++++++++++++++++++++-----------------= ---- 1 file changed, 78 insertions(+), 67 deletions(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Mar 16 08:53:42 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 306B557CE4F for ; Tue, 16 Mar 2021 08:53:42 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "DFN-Verein Global Issuing CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F06X52DQGz3lrX for ; Tue, 16 Mar 2021 08:53:40 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from [129.217.43.55] (chulak.cs.uni-dortmund.de [129.217.43.55]) (authenticated bits=0) by unimail.uni-dortmund.de (8.16.1/8.16.1) with ESMTPSA id 12G8rdNZ009707 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Tue, 16 Mar 2021 09:53:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1615884819; bh=VZQT1WYJRnMfCFIneMtYVfnM1uFh+wkyAfK8Ylp2y1A=; h=To:Cc:References:From:Subject:Date:In-Reply-To; b=Ud7Z0NhER2K7jLHdiLWK7f9rwF8kBx5/wh7h/Gn8ymUyro1S+sYQ6wXeb2Em0cui+ Hs5XQCRB3EYODTNtokJ90zQBHhhktGynsWUeM0zj8anKrk0C8jKCS2wjOD2OhzsO+z l9cAeu1GtUkdDmqm81CTStIy3JB58aXGmBukdXHQ= To: Konstantin Belousov Cc: freebsd-fs@freebsd.org References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de> <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de> From: Alexander Lochmann Subject: Re: [RFC] Understanding the locking of struct buf Message-ID: <9e591bfe-1790-251e-a90c-79c6ec66b3ac@tu-dortmund.de> Date: Tue, 16 Mar 2021 09:53:39 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4F06X52DQGz3lrX X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tu-dortmund.de header.s=unimail header.b=Ud7Z0NhE; dmarc=none; spf=pass (mx1.freebsd.org: domain of alexander.lochmann@tu-dortmund.de designates 129.217.128.51 as permitted sender) smtp.mailfrom=alexander.lochmann@tu-dortmund.de X-Spamd-Result: default: False [-5.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:129.217.128.0/24]; RWL_MAILSPIKE_GOOD(0.00)[129.217.128.51:from]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[129.217.128.51:from]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[tu-dortmund.de:+]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; ASN(0.00)[asn:680, ipnet:129.217.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[tu-dortmund.de:s=unimail]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tu-dortmund.de]; DWL_DNSWL_LOW(-1.00)[tu-dortmund.de:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-Mailman-Approved-At: Tue, 16 Mar 2021 11:19:46 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 08:53:42 -0000 On 14.03.21 09:35, Konstantin Belousov wrote: > You cannot write any working code that uses buffer cache, without studying > both buffer cache code itself, and examples using it from filesystems. > Short (even the whole-screen short) herald comment cannot explain all the > nuances that grown in the 40+ years codebase. I see. > >> Shouldn't that particular piece of documentation be updated? >> For example: b_bcount /* w: b_lock, r: no lock needed */ > Do you understand the purpose of the locking? What do you intent to do > with the read value of b_bcount, if you do not own the buffer? What is > you next action with that value? > I do know the purpose of locking. I have no particular usage in mind at the moment. I'm trying to understand how locking is handled in FreeBSD. In the Linux kernel, for example, there are such exceptions: Read a ptr without a lock to perform a NULL check. All I'm asking was whether the documentation should reflect those exceptions. I assume the answer is no. Thank you, Konstantin, for lifting the fog. - Alex -- Technische Universität Dortmund Alexander Lochmann PGP key: 0xBC3EF6FD Otto-Hahn-Str. 16 phone: +49.231.7556141 D-44227 Dortmund fax: +49.231.7556116 http://ess.cs.tu-dortmund.de/Staff/al From owner-freebsd-fs@freebsd.org Tue Mar 16 17:27:51 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CFDB85B61DB for ; Tue, 16 Mar 2021 17:27:51 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0KxM1GLgz4rLg for ; Tue, 16 Mar 2021 17:27:50 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f178.google.com with SMTP id u198so33945519oia.4 for ; Tue, 16 Mar 2021 10:27:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=uwP1LCsefw81DoC0u/ui0qf8kKoVXJmMaf5nObab0tM=; b=TXAFiqNVXSPjAsXOj+y5ll37a7DQ1Cio5OABY5vpH1xU+JV3REVZW5Fu4r6qPxrCx4 QbRekPXlctsgTDGiQes0uU7hTnf/ywur/dXC4WdCJh/FLtxhWhGoXIoV6G0pU1DUWp4V pLNh6A/jehwFPnh9e+uPJTtamVaK9KVGxqt23uImqs2AROUwj39tWtpqo5LX3+dzCsxe YXUM4JkpgZXM+4Vgu1MH/+ElBpa9yZvLYOiXchXSz2X47bosRCeX4xtz/YHzrf6stn3T twsXDjj7+NHawhYFlh2Kvc2QG+VndXFCBsC6B0SChsbpWauDFpim2bd5HSwnEi8Cji0k MqjA== X-Gm-Message-State: AOAM532NE8cB0UIEliF8Yj9E8T//oTe1Lr7n0pNcV7F+8HTguCzFqOpW lQ4iTxVc/dJ84EXGUdYT/2AjactZYU/4bw5RY5ZdMaFs X-Google-Smtp-Source: ABdhPJzLm1+CFgIDkCDu/TIwc6xA7FjjDR98U5P+u///m9upHVGCNBB7+xy4p7lDuH7iJcwgP1XN2FX/WSKxHYoT/Eg= X-Received: by 2002:aca:3046:: with SMTP id w67mr98669oiw.57.1615915669713; Tue, 16 Mar 2021 10:27:49 -0700 (PDT) MIME-Version: 1.0 From: Alan Somers Date: Tue, 16 Mar 2021 11:27:39 -0600 Message-ID: Subject: Understanding the vfs.nfsd.request_space* sysctls To: freebsd-fs X-Rspamd-Queue-Id: 4F0KxM1GLgz4rLg X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.178 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.167.178:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[asomers]; R_DKIM_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[209.85.167.178:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.178:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.178:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 17:27:51 -0000 Could somebody help me understand the meaning of the various vfs.nfsd.request_space* sysctls, and what implication they have for tuning NFS? I have several NFS servers. Most are performing well, but one with a metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAttr and Lookup ops) is not. Total throughput is a not-unreasonable several hundred MB/s, but some clients are limited to very slow speeds, sometimes writing at < 10 MB/s. The request_space sysctls show some pretty stark divergence between the well-performing and poor-performing servers: sysctl well-performing poor-performing request_space_used < 6 M varies, but currently 40 M request_space_used_highest < 39 M 24 G request_space_throttle_count 0 35 So what does request_space_used measure, anyway? And how can I either increase the available space or decrease the stuff that's using it? -Alan From owner-freebsd-fs@freebsd.org Tue Mar 16 20:27:49 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CDFF756C418 for ; Tue, 16 Mar 2021 20:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F0Px15CnDz3MhD for ; Tue, 16 Mar 2021 20:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id B10A256C16E; Tue, 16 Mar 2021 20:27:49 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B0CA856C16C for ; Tue, 16 Mar 2021 20:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Px14YBmz3MhC for ; Tue, 16 Mar 2021 20:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8F29729B55 for ; Tue, 16 Mar 2021 20:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GKRn6c020667 for ; Tue, 16 Mar 2021 20:27:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GKRnY9020666 for fs@FreeBSD.org; Tue, 16 Mar 2021 20:27:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev Date: Tue, 16 Mar 2021 20:27:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: vangyzen@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 20:27:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349 Eric van Gyzen changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mjg@FreeBSD.org, | |mm@FreeBSD.org, | |mmacy@FreeBSD.org Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Mar 16 21:23:37 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B119156E339 for ; Tue, 16 Mar 2021 21:23:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F0R9P4RXKz3jHg for ; Tue, 16 Mar 2021 21:23:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 96C1056DEEC; Tue, 16 Mar 2021 21:23:37 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 968B956E703 for ; Tue, 16 Mar 2021 21:23:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0R9P3mzyz3j9Y for ; Tue, 16 Mar 2021 21:23:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 74C462ABB7 for ; Tue, 16 Mar 2021 21:23:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GLNbBj054477 for ; Tue, 16 Mar 2021 21:23:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GLNbJ0054476 for fs@FreeBSD.org; Tue, 16 Mar 2021 21:23:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev Date: Tue, 16 Mar 2021 21:23:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 21:23:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349 --- Comment #1 from Andriy Gapon --- Maybe that's because loader does not support booting from such a pool? Just a speculation on my part. Removing a vdev means that now there is vdev indirection and other complications that require explicit handling. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Mar 16 21:37:48 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DFA7E56F0A6 for ; Tue, 16 Mar 2021 21:37:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F0RTm5q6nz3kC2 for ; Tue, 16 Mar 2021 21:37:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id C7B4856F0A5; Tue, 16 Mar 2021 21:37:48 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C77FB56EC47 for ; Tue, 16 Mar 2021 21:37:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0RTm5DdDz3kFD for ; Tue, 16 Mar 2021 21:37:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A6BAB2AF55 for ; Tue, 16 Mar 2021 21:37:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GLbmne063043 for ; Tue, 16 Mar 2021 21:37:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GLbmTs063042 for fs@FreeBSD.org; Tue, 16 Mar 2021 21:37:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev Date: Tue, 16 Mar 2021 21:37:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: vangyzen@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 21:37:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349 --- Comment #2 from Eric van Gyzen --- mm@ filed an upstream issue for this. Thanks, Martin! https://github.com/openzfs/zfs/issues/11762 Andriy: Fair question. My workstation, running main, boots from such a po= ol.=20 That's how I found the issue. My old SSD was failing, so I migrated to a n= ew one. For historical reasons, my pool was split across two partitions on the old disk. I wanted to use only one on the new disk, so I didn't use mirror= ing for the migration. I simply added and removed vdevs. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Mar 16 21:46:18 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C4D3356F15C for ; Tue, 16 Mar 2021 21:46:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F0RgZ4vBLz3kvf for ; Tue, 16 Mar 2021 21:46:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id A806756F51B; Tue, 16 Mar 2021 21:46:18 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A7D4056F15A for ; Tue, 16 Mar 2021 21:46:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0RgZ4J7Nz3kqx for ; Tue, 16 Mar 2021 21:46:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 850952B18F for ; Tue, 16 Mar 2021 21:46:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GLkIPD067766 for ; Tue, 16 Mar 2021 21:46:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GLkIks067765 for fs@FreeBSD.org; Tue, 16 Mar 2021 21:46:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev Date: Tue, 16 Mar 2021 21:46:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mm@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 21:46:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349 --- Comment #3 from Martin Matuska --- It looks like VDEV_TYPE_INDIRECT is handled at stand/libsa/zfs/zfsimpl.c The code originates in the device removal feature: https://github.com/openzfs/zfs/commit/a1d477c24c7badc89c60955995fd84d311938= 486 By modifying the vdev_is_bootable() function, setting bootfs is not allowed anymore. I think this is an easy fix and really not necessary on FreeBSD. Thomas Soome has been dealing with support of indirect vdevs at end of 2019, see: https://svnweb.freebsd.org/base?view=3Drevision&revision=3D355786 It would be great if Thomas would tell us some more about the support of removed vdevs (=3Dindirect vdevs). --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Mar 16 21:53:18 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A35D56F698 for ; Tue, 16 Mar 2021 21:53:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F0Rqf0nrDz3l3R for ; Tue, 16 Mar 2021 21:53:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 1B1C856F5AE; Tue, 16 Mar 2021 21:53:18 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19E4456F627 for ; Tue, 16 Mar 2021 21:53:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Rqf0865z3l3Q for ; Tue, 16 Mar 2021 21:53:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ECF812B13E for ; Tue, 16 Mar 2021 21:53:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GLrHsJ071930 for ; Tue, 16 Mar 2021 21:53:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GLrHoj071929 for fs@FreeBSD.org; Tue, 16 Mar 2021 21:53:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev Date: Tue, 16 Mar 2021 21:53:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mm@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 21:53:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349 --- Comment #4 from Martin Matuska --- This is the most important commit I am referring to: https://svnweb.freebsd.org/base?view=3Drevision&revision=3D350772 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Mar 17 00:06:45 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1569E573131 for ; Wed, 17 Mar 2021 00:06:45 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670089.outbound.protection.outlook.com [40.107.67.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Vnc0qRNz3t4t; Wed, 17 Mar 2021 00:06:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mfyUJmofVacVpZyVnjICuLUaut3+vEsd8Qg+OK7y7a/vW4n97U0KrJBnIZXgsupVk7MAVGhshMIUJZcqCKagzTy/PlqLV8RU2VKU/nn3oRAAykFSLRec5BjP0pD86xxt1UCvIaK9cwXvy1rG2fO/Mg0kLFJ9u6RcdSrHO3e1GZaB0GzCok579isK2mlEcnCz0Gi9EGUQveO5NiTTfkbs88/V0QHAJvk7aHMTkIsogBhvcdCU6qgkE4TMUGRfRJvcpyt2aHw96zN+eIVzJBCdsgyFs11LqfXzFJWzk48K4fsLNsZ40VZCOQ5oRzO+kc9HY33fTp8pk3GTXFMMaDMwKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pHvjowwaioWXpHFZsKodIFjuLdc/7Wa3c3oCFpPgMhg=; b=jS65r+jZ5ZUB40dTTaXv5K0mV79Ej087/NBA046qdzcn5fqQSwqDR/F+OabWn1hbKU6uqKTjzOysgaPodiO9JtSIsbLCVtOI0xk3k7iYfQ6VjQGmZzcwiL5ROUNTBfvtfqFYag6SH9H5Wbovato76nYn8z0jeEHcsV2GWJGY05lPtn92FoKxOGCtvEJVPS9XNUrJYHh4e39T1oFAWf6/XoU1aUKcn50m75CfhazcqMtjDFLIhS8MFJpBsNZDR7OnsFHhFf+RHnl35FRiX5fjz04xe6YjCQDlvi2rOhV+jXd/d/Fa48qiVevClG21S0M2ihTy0QKKHgEMgTpfafaQ7g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pHvjowwaioWXpHFZsKodIFjuLdc/7Wa3c3oCFpPgMhg=; b=Lakji1Z+1bFmjZOkqetVV7a0wje3V4yYaXNR0xk+6DSMXWo+H7rUexikCvxafv3d+SwMHsMIC9t/b0IiaTRHix6n7C28QPWDnqqX1RflBmDciGIXDsZ20Q9gFVg+r+ISnWt/vQP4ODEpcmDO1o+P0FKshf2P7vuLeWsQM+pL6b4EtPuuTa2zlRSOMxnYCzNzA1MElDnYLaoUlt8eVzTmzltEOtA/R8C+b6Nzgh7C21tkGgRQM0dE+Ujt1D82Uk+ihDmsYbedf1dieQXKHLK3UiTBXEwWHKzZh+j+w8LVf2HBeXlW5yFVu6rYOHTeG5o+h06iyqAtm/VkjMLdyK76/A== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB1764.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:e::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar 2021 00:06:42 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021 00:06:42 +0000 From: Rick Macklem To: Alan Somers , freebsd-fs Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Thread-Topic: Understanding the vfs.nfsd.request_space* sysctls Thread-Index: AQHXGom0SMOUlewkdkSbAUM6yshgqaqHSCtT Date: Wed, 17 Mar 2021 00:06:42 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0f2d188b-be5a-4991-4a75-08d8e8d88b71 x-ms-traffictypediagnostic: YQBPR0101MB1764: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5797; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: eLNYg7dSW95q2AkE5IjWGzxnkzfOK6PB+mNSjViMvybs5dbNUH/NrdGSEXSaBugGPp79PBH9cLki8R0DemuHxXAM84S5juhecJL8MZrvTsZsrx7i8WGL1cccUVzujpTSDUqQjsMKbTp5gZo4Mb6PX3vsDBmjCg1TF0lyTfgTlrgh1YnOC0ihCqMbSWULZ58PFiFFuNLtt0TG89LnEtfEkivqxCXN6hqsh6v8MVSvQux4olcKglmO4uJGTugKXQvJNRAHs9KuzMGIyBeRDTs0ZQrqbzXAy9Fofml4HAy2ponbqRhVHIH7/kPcgizK2WrBd4XFWDHZ8P9NFKrZQwg3Pwz1ATmQS+AULkjEwzKsujkDQboIiAshhakMzTbhNeW7VGxm3DQlOoNXrQfRIc7aa0tYHd97SZ8jtIn1PRoywE/jQNMfO+KLp2rNRWSC/daUS+wpY9bNQVZGFafvhU1Ct9ZK3ZOGcmojkfwprE32LdHLS/Gsv4DwpArCAx7SB9k6vg6PwKKiAbjT+Mgc2c1qNH2y3xXvB4DmOI0M6WpJL4XzeA65LFuku4aJFbCih2OY2YZSwd9Taw8yr2wSkWfRLcwCyIBaQ74y2jTeLFGYjTBFQ0nrPi3l8YaIlIAVUqga4y9d2VZHjO7Sb9+xQs3F7w== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(39860400002)(376002)(136003)(366004)(346002)(396003)(966005)(71200400001)(66946007)(8676002)(9686003)(52536014)(6506007)(76116006)(7696005)(66476007)(478600001)(64756008)(5660300002)(91956017)(2906002)(316002)(55016002)(786003)(8936002)(66446008)(83380400001)(33656002)(450100002)(186003)(66556008)(86362001)(110136005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?C44i7Z9ehjPF9pzMdKd/hYkkYMycz2gTvdvLVrGdUqn3YJMFWz/uAIgZe0?= =?iso-8859-1?Q?piRuBuB6tjU3H7fh15Z4F1IITVTY55APGJWYMIAdIh+dpOe7lZW2KOTBeD?= =?iso-8859-1?Q?bsqL9lxwNExv2FLpuD/axE4qKL1wSzuYDyqyI3up9LtngBRDxzYVP3f3Rs?= =?iso-8859-1?Q?bfBbOkmaT2FTZbi1H/mSgxVS28YFYCouXX2oEsAQPXNwtFgNFjWvdzDrgw?= =?iso-8859-1?Q?BLgLzsLTXfBOUIm4ihnrE4YFpaTHLv6CE0X4BslY6nx1LRB/TXjv76qnLM?= =?iso-8859-1?Q?yzDposGEFB/QnNFFTFkG0zfcigV3nPtFHtzt5rYOO2ukySoIVibvu7CxBi?= =?iso-8859-1?Q?J6EGTixLv9YZipddhOpS8kNGQt2ewrG8Z10cHOMZxzwpiW0W9ScECnreyD?= =?iso-8859-1?Q?d2PSC6wWiUZ1ixzf3clR+2spl94WrHTLz0gTyhUkGI8J73RKpTP2xaQpsf?= =?iso-8859-1?Q?WKGBYnU+zfVDSxGzHXnc6RhQFrGd0atA1b50tWb8zDOAZZZ4DqpZhdmMUf?= =?iso-8859-1?Q?Lj6kH02T7eZG8+kILFu+J7XTRyVtxZ2D+mpA/gJsdRE2u/nltJIu16v/BO?= =?iso-8859-1?Q?H8tc9kYpuhZAfxtRLfWl5G7Aesmw2p6WDtkbTFnA9didQ+i0frmg+VX7ha?= =?iso-8859-1?Q?9pbA0/7Ie6v/hbgya8ibA3oJ2u0wFgq+x6/Un4ZcPcCbFb/1dKghzGwyK8?= =?iso-8859-1?Q?y6HoAZI3M+Ylk+zmBnLrAsfuMNN7u8wsef7851yD7UUF7NE/GD1aqlSWRz?= =?iso-8859-1?Q?8vPh/i0Z7oSy44s4c0BZWw4CGe+/RSntsfxx0EmJJxiNgQJ5F0bE7sbsM0?= =?iso-8859-1?Q?fYTw+rJgCGUuUi/4DZ7wkeLL0lGY1BXCFzVkavZA2khTx1nVBq7rTYPjDA?= =?iso-8859-1?Q?/Pm3QvqT/3YUGT5iSiZkLLBib6JTKJSIv9zRDhJUsxvGblsmUGvGJqajnk?= =?iso-8859-1?Q?YaoBM2Tpf7/N2yi6fSwM2PJjuHeC/RjJWT5JxZEwSHlGlxvBGpFZ9Kouch?= =?iso-8859-1?Q?rDYmGNgg5t3ILLquA0nSgS/zHK8gAu9PBD0D5qt9du+jlIFlwnTG/V/etY?= =?iso-8859-1?Q?aYVEkKbYTZfKRSkcrQ56Woj8l2dSNqyDx1l4W88DFJUOvsN+zR4j4bT6Sw?= =?iso-8859-1?Q?dZ30ZZdhdOnI+rGumDzaRoa+0iFqVLvdNOTHF+WMtNeGMzRKlgbOwe4rGQ?= =?iso-8859-1?Q?whGhATBfK+fHnmyPpDIYfogKRIfjHYTgpC/myXGxvr/sVfbKnfu4uMW6Rf?= =?iso-8859-1?Q?ghVSAkDmsnAR+V1HF7DMdoVk1umoxQjveXYaTQjtwGKhZUMjKc6uvF/0QS?= =?iso-8859-1?Q?T1ofprLSjqqnPJCW1+gHW7Og/weRTV0I+PNdkQET2mO3YLPhIIUqBvhwxP?= =?iso-8859-1?Q?SazMCe2NS3+lPs6l2xPSmIyo943AM/3v7KDpg7ygW17j1cJboQAupxmL1f?= =?iso-8859-1?Q?nLScc6fTHR51fV8a?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 0f2d188b-be5a-4991-4a75-08d8e8d88b71 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 00:06:42.1943 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: QF3g69QNjNpkb74iVwKpOW0wIbuBJKRXpcfXeBZ7BcHHaYe7AIpPx4xSjlPjo5edtAeo7k5viWYqmQ7HrPOBZg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1764 X-Rspamd-Queue-Id: 4F0Vnc0qRNz3t4t X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=Lakji1Z+; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.89 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.03 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.89:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.97)[0.968]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.89:from]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; SPAMHAUS_ZRD(0.00)[40.107.67.89:from:127.0.2.255]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-fs]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.89:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 00:06:45 -0000 Alan Somers wrote:=0A= >Could somebody help me understand the meaning of the various=0A= >vfs.nfsd.request_space* sysctls, and what implication they have for tuning= =0A= >NFS?=0A= >=0A= >I have several NFS servers. Most are performing well, but one with a=0A= >metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAtt= r=0A= >and Lookup ops) is not. Total throughput is a not-unreasonable several=0A= >hundred MB/s, but some clients are limited to very slow speeds, sometimes= =0A= >writing at < 10 MB/s.=0A= >=0A= >The request_space sysctls show some pretty stark divergence between the=0A= >well-performing and poor-performing servers:=0A= >=0A= >sysctl well-performing poor-performing=0A= >request_space_used < 6 M varies, but=0A= >currently 40 M=0A= >request_space_used_highest < 39 M 24 G=0A= >request_space_throttle_count 0 35=0A= >=0A= >So what does request_space_used measure, anyway?=0A= It's the number of bytes of request message(s) received,=0A= but not yet being processed by nfsd threads.=0A= (They're actually in sys/rpc/svc.c, although named for the=0A= nfs server.)=0A= =0A= Anytime used > high, it throttles, which means it leaves=0A= the RPC messages on the socket receive queue.=0A= This indicates the server is not keeping up with requests.=0A= (ie Overloaded or ???)=0A= =0A= > And how can I either=0A= >increase the available space or decrease the stuff that's using it?=0A= Increasing vfs.nfsd.request_space_high avoids the throttling,=0A= but it is hard to say that is a good idea, since there won't be=0A= backpressure applied to the clients via TCP windows, etc.=0A= =0A= Fixing the server so that it isn't overloaded would be better,=0A= I think?=0A= --> I'm the last guy to take ZFS advice from, but I think there=0A= is a tunable w.r.t. how much arc is used for metadata.=0A= Getattrs will need cached (metadata) to reply quickly.=0A= Lookups also depend on cached attributes for good=0A= perf. as well.=0A= --> Make sure you have lots of nfsd threads. They are=0A= just kernel threads, so they don't use a lot of resources=0A= and having too many is much better than not enough.=0A= -->I can't remember what the upper limit is these days,=0A= but I'd set it to that for a busy nfs server.=0A= For NFSv4.1, each client can send a maximum of=0A= session_slot_table_size concurrent RPCs. FreeBSD=0A= uses a single 64slot table for each mount. I think=0A= Linux uses a 32slot table, but supports trunking.=0A= I don't know if the Linux client uses a separate=0A= session table for each trunk or not?=0A= --> Something like #clients * 32 (or 64) nfsd=0A= threads running on the server.=0A= =0A= rick=0A= =0A= -Alan=0A= _______________________________________________=0A= freebsd-fs@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= =0A= From owner-freebsd-fs@freebsd.org Wed Mar 17 01:32:29 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1D55A5757C3 for ; Wed, 17 Mar 2021 01:32:29 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0XhY0Gtvz4Sx6 for ; Wed, 17 Mar 2021 01:32:28 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f51.google.com with SMTP id h6-20020a0568300346b02901b71a850ab4so290946ote.6 for ; Tue, 16 Mar 2021 18:32:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=b6UdMGq9B5EqhM2b14RACWAp4KJXWd+Q+x1zS3ZnXBo=; b=Bod4yDcWc36UnbpunlCkPtBFpX47x0Mz4wv8zBoVTh3c5l7oW9XpM6iJxNb9QsCF74 /8AzT7Ktfox47IXQUTnQHx99GuFGOhe8qhc7k7zj4oiipxevOus2QvnTkVyYlnRZ83i4 7ssAxfNq96tB4VDWriqbprSdREfaAiKepHe679+bt/nX3ZnsTTAyykCsH//1saIZY1dC 2BP0XEGEihFoRxW0UmT+lsZWqz3KOZkFkXu6YNub/j2lhFfFsCHKb2db1+CCYZORghK5 JQGspTklR3F0qcR+C+8u3gatjwidLljG/3DhtJovAyISNmloV8MBLCJobDqiaai6p2B6 N1Kg== X-Gm-Message-State: AOAM532GLT1v/f85Fm2imLfgKxRXD8BgQv+hwsW+HWyfIyQKLGOaCgAF JpKPnvPsomQa6ZysbAFUNM8lj6XYR9NtGACOfLs= X-Google-Smtp-Source: ABdhPJxAANVPfNKWdM8oeRBwa/M4Z/9awf70F1qcD08RF5v+rla+o/jrUZeSAo5fBoycqkjYGNL/AXIJ24DQEHGrM1k= X-Received: by 2002:a05:6830:148c:: with SMTP id s12mr1334689otq.251.1615944747110; Tue, 16 Mar 2021 18:32:27 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Tue, 16 Mar 2021 19:32:16 -0600 Message-ID: Subject: Re: Understanding the vfs.nfsd.request_space* sysctls To: Rick Macklem Cc: freebsd-fs X-Rspamd-Queue-Id: 4F0XhY0Gtvz4Sx6 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 01:32:29 -0000 On Tue, Mar 16, 2021 at 6:06 PM Rick Macklem wrote: > Alan Somers wrote: > >Could somebody help me understand the meaning of the various > >vfs.nfsd.request_space* sysctls, and what implication they have for tuning > >NFS? > > > >I have several NFS servers. Most are performing well, but one with a > >metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, > GetAttr > >and Lookup ops) is not. Total throughput is a not-unreasonable several > >hundred MB/s, but some clients are limited to very slow speeds, sometimes > >writing at < 10 MB/s. > > > >The request_space sysctls show some pretty stark divergence between the > >well-performing and poor-performing servers: > > > >sysctl well-performing poor-performing > >request_space_used < 6 M varies, but > >currently 40 M > >request_space_used_highest < 39 M 24 G > >request_space_throttle_count 0 35 > > > >So what does request_space_used measure, anyway? > It's the number of bytes of request message(s) received, > but not yet being processed by nfsd threads. > (They're actually in sys/rpc/svc.c, although named for the > nfs server.) > > Anytime used > high, it throttles, which means it leaves > the RPC messages on the socket receive queue. > This indicates the server is not keeping up with requests. > (ie Overloaded or ???) > That all makes sense. So having a high request_space_used basically just means that my storage is too slow. > > > And how can I either > >increase the available space or decrease the stuff that's using it? > Increasing vfs.nfsd.request_space_high avoids the throttling, > but it is hard to say that is a good idea, since there won't be > backpressure applied to the clients via TCP windows, etc. > > Fixing the server so that it isn't overloaded would be better, > I think? > --> I'm the last guy to take ZFS advice from, but I think there > is a tunable w.r.t. how much arc is used for metadata. > Getattrs will need cached (metadata) to reply quickly. > Lookups also depend on cached attributes for good > perf. as well. > I've added an L2ARC since I sent that original email. We'll see how well it works. I expect it to take 48 hours before results are apparent. > --> Make sure you have lots of nfsd threads. They are > just kernel threads, so they don't use a lot of resources > and having too many is much better than not enough. > -->I can't remember what the upper limit is these days, > but I'd set it to that for a busy nfs server. > For NFSv4.1, each client can send a maximum of > session_slot_table_size concurrent RPCs. FreeBSD > uses a single 64slot table for each mount. I think > Linux uses a 32slot table, but supports trunking. > I don't know if the Linux client uses a separate > session table for each trunk or not? > --> Something like #clients * 32 (or 64) nfsd > threads running on the server. > Experimentally, I determined that 768 threads is sufficient. But your formula would suggest > 2500, which is a lot of threads! I'll keep it in mind if I ever reach 768, though. > > rick > > -Alan > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > From owner-freebsd-fs@freebsd.org Wed Mar 17 02:31:39 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 20FBC576C46 for ; Wed, 17 Mar 2021 02:31:39 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670051.outbound.protection.outlook.com [40.107.67.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Z0j5Cbwz4WGX; Wed, 17 Mar 2021 02:31:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nYkIyxjWmIiWtM9EhJJneMkYUXukCA+PcZcXBrmchGMnp2tU/AKquVcr2ebTv54B0WBTZgQGgqYrZl1IxdPZaBFmhObl10ra8h3KFsBDnd57sHafCZjGNWvm3vttYOpOEFRvgiQ6sXtJLYZzxg8hcl8QlTPYCKWZCTmjhD8S/q2V+ZtUIbChl18A/4nEIiutVTDEjM6oVfIfRmhmxsyEPsl5sUbacH/dEaQQD01DLtLbhBsA31S0w1vSAfpJShg8B/TfQM0xRDjV7o14VwJzV9rNXfHkw5H7XEDPRjnbUb4YIIbyfzTAhAb/vcGnHnyTdkI5KolTQO5dxgkPBffPMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wLCP9g/PEdXdiOyqQqrfl32Sgu0Y8tOatQjHXXyPjZo=; b=g4MWHtFgukECLOUQ9hbel/TUKPX0imdyAokDIn1CxebzK6ECbZAVKJ0ZH1BlWDOW6JFjNRTsyl/JqPXeABI7kmmGYDn/G0w2M6KNDS0OBNrpQZtdbu9B0NLdMPZqrKC4XB9a9uG0zQ2HGIzbCGgTcyxVH1OErZ2F4YbXjwmWyqsmJ4lRniPeSDRyKoBL7dxPES7pLK4nKZdybb1KBdJ3IcT/3cc5tq7F3J3MN8WN1+QoBzHth8Y8UgMtHeIxubqw1NMquol0akCxm1z5l+a3wEFDjUFUH5yBKENp9Tw6LZw5Z06J2zntj+EfBnk7cHcG2941+e/ir48Y6fkTJo665w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wLCP9g/PEdXdiOyqQqrfl32Sgu0Y8tOatQjHXXyPjZo=; b=TZcAHbmwNB1Xpk31zXJG2dY8SiEWSrfcxlSsQlWEociYeCywP350vRCE8ambjQVxnmns9RjKIZN2e1Ix0P4NsAAYj/w0nYx/PfEbWJf0EVJR+JuVOZe8Dke81T71Y8f5Lj0cJXf5Z6YlmzzwAMnGE4A7YeyeSjIwshVFsl31X71nR76aTociT72HhAIeZT35DjXszSsNeUprsghz+xitph5xaYGe54m6IIEZI+ItxveGTe8WaUuwlMrqNoqJRHqF1WvFuxkoMitwZtfFQyk+qHk1b09EVtKVuulVdJFEwJJfqogft0NNGfDagxvmQGMhA+kXNNXrhExY/t9gapgdig== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB3430.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:4d::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar 2021 02:31:30 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021 02:31:30 +0000 From: Rick Macklem To: Alan Somers CC: freebsd-fs Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Thread-Topic: Understanding the vfs.nfsd.request_space* sysctls Thread-Index: AQHXGom0SMOUlewkdkSbAUM6yshgqaqHSCtTgAAdewCAAAz0qQ== Date: Wed, 17 Mar 2021 02:31:30 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b53ed26f-ae04-4060-727c-08d8e8ecc627 x-ms-traffictypediagnostic: YQXPR01MB3430: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: J/0wBns632WIQavUurYRQBArDZ2H0tPeu0lS5q6yoj7xKoVGvFGDf6i1eCcLMCUyIJ6ChQGodiHJ3zTOkFlaKrLld026HGHXo2/C1Nvz6dndgKkeczEE7KRdHZmho3kIhD2+xT0qWC5xvcPuqfydCRjdSo+aU+HmnNhg0iTTrFwpRhafIY3ovhK/3DIRNFXRaRlcy9ybJXTCZVIuC5Mqy9BfVlPrucR0QABQPI3Wf1d8QGfx9c+FgptN5MlhvkXbE09j8Wyzb4DUvX6rYrGksMpFbn3/RLP+018lE/8TzS7uYldZj+0UWRp3Y8qHs04GU8a7ltFDJ/qMbbjfL8WqgNzdvqP0T13eIHAs4mViBW2eS1gwE4sKDPbpOvkvdkeTYQpMlkzW1k/RoaEvNRCfhqCVdFAcHbvD4JReUMk7AqtN7UOHUM+Rarvgd0VOKBzlstkWDxUpPAdP5X+s9mhWu6PVPvp+MhsrIs71joxAPcqbdTmxMvVTe7/wI4Kztblm1OZTDUeAjx9fJGD4cd09hoDx6Si0T6dvn1ba9cIrGu1/VVuXLf1+/dWkr7jgXOL69SxENsum91X/JI1rRME34sN8RH7B214WYn92d0Pj+j3yh9mTy7fA6Sca8cGVZVLdtxKH2yZCrX4oc6uWP57F2A== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(376002)(396003)(136003)(346002)(39860400002)(366004)(66446008)(76116006)(66946007)(316002)(9686003)(8936002)(478600001)(64756008)(66476007)(6506007)(8676002)(6916009)(7696005)(450100002)(55016002)(4326008)(86362001)(5660300002)(66556008)(91956017)(52536014)(2906002)(83380400001)(33656002)(786003)(186003)(966005)(71200400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?tQKcv5fggIBPb4QCrT+K4tWsMOeK+etSa/C1d7r6S8fkiZc0EUiHobjwyo?= =?iso-8859-1?Q?VWcfAH7JGKcCESITsDfyHDSTkRusiwD17NQCczPsMXzIsoCbCf9NGnXFGZ?= =?iso-8859-1?Q?K3Gx9WXbk3UBLIzUZR/tb91symuHsRQBKViUmvb8iZHhvb/L7XfkkFHOVK?= =?iso-8859-1?Q?d+BX7yq3ZrvnwZ8HYPg4fKNTh1hgG6amNm0oOviBOzFxfXoPLG1kJD/9mo?= =?iso-8859-1?Q?DGVNjwhE4JG9uWIOyMc0SLS8Lfl3gcAPExh4R0c7K/2Qi9eDgoxT2Th5Dq?= =?iso-8859-1?Q?HCjvqdp+NhOV23n8N3KfsCoklWIi6cYMGvCkTD/y5tgmlE5PdIprhN29rH?= =?iso-8859-1?Q?M1F/ek7yhYqe0NnllNDwdbsA/hRgA/Bn6Z3R27PLtBcwVbHF99YCa59zDg?= =?iso-8859-1?Q?tpx4GgvMk+8zZCXnzWRHqh+Lqga1skKQtWQc9InjCweYOO3ESB1ayvLlCj?= =?iso-8859-1?Q?fvJzfv5AlxqdifHw2mPRt9Vzh1nRkyQ+qid9Lak1hqlWG4/tyS+RmU9T53?= =?iso-8859-1?Q?7r04A1Cz3s8aV7Y5b5ldRFckHb+tkMsikOeqoHYvsh89+HherWoiN3o6Ii?= =?iso-8859-1?Q?2QSYaalOqVx8PzS+Ze327WtcQ9sjH3K4mwl3P61qWDjbOMxaprupRzBgUa?= =?iso-8859-1?Q?ed5At+6xBuGMkndWGFnSAUVOeBYIzG33796CmVb4ZeUJfwh57jJe5bVnoZ?= =?iso-8859-1?Q?JnZEVqE060oEFVKXg9rlP0Dhs6khxiDNVTBnGWXjVQ6xTRfbgGKZF368Ke?= =?iso-8859-1?Q?aw5bgApqlxPD+31EuZ0oeTDz1uQNpbY1796ewpV/uIyQt60QJcxW1SCGZ7?= =?iso-8859-1?Q?zwLwdsHyCsy6fOHs/lSTUOiDH2cUH2cCjhBMS5tSRG6WjoU7B25blXBEG8?= =?iso-8859-1?Q?SNqpCtRd3knDLWmpbopbVxxljhH/VzR9Aqn7BhssYrVJF3iHnnC0tbCOlL?= =?iso-8859-1?Q?gswx4dJaDllmb0OvM3TPNFOwzeyLflS+8NMe0OyMbEH6ZL4+EYdnQST2ZJ?= =?iso-8859-1?Q?T1QtbgW9pna49mfxXHt1yEfAn74muKWNjyhvp2uaAYvzkDbirEDE37dMZU?= =?iso-8859-1?Q?vg4mIb1TjiV6grU+NiUIuaMwO4tfvNDIiZE0ljj5+4uE+QoxciAdeTa5ij?= =?iso-8859-1?Q?a1zA53RS6io4cRwuX339c+F8i9/SK9hWJ7fvP5P3a/UK7Xh9ABhV1+7VQf?= =?iso-8859-1?Q?ThNFiJspCGsR50v8B8BWj5YQCi/qUxUDNcBsuvTDFiQbR3KayjmgrlpA1B?= =?iso-8859-1?Q?kyIlu+5epv0Gzyfu09/3qe5qBk+9yaJlpHBFvT/TPa3jqnQP6VFFO7R+aY?= =?iso-8859-1?Q?s6Bavi0iCCl21IIXgYBgrgIs4mdBvIsygRHWYBbCVFnE2j0mL4aSSLt3ao?= =?iso-8859-1?Q?nHZE39s+Is8a/WwkIQ9Xa/SErtzRSQ2ClOdIfsLjroESZV4bi9rSU=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: b53ed26f-ae04-4060-727c-08d8e8ecc627 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 02:31:30.6365 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: kwnvC7ROnPdKxrwvQ9TYwduUBbIbPTc3GSspDKXZ63nLfrFMP/+4o/5drui8J+OCOn32qPKcWg0dS4a5SpwXVA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3430 X-Rspamd-Queue-Id: 4F0Z0j5Cbwz4WGX X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=TZcAHbmw; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=temperror (mx1.freebsd.org: error in processing during lookup of rmacklem@uoguelph.ca: DNS error) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.80 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.51:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; R_SPF_DNSFAIL(0.00)[temporary DNS error]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[40.107.67.51:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.51:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-fs]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.51:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 02:31:39 -0000 Alan Somers wrote:=0A= [stuff snipped]=0A= >That all makes sense. So having a high request_space_used basically just = means that >my storage is too slow.=0A= Since it appears that you have plenty of nfsd threads, that would imply=0A= that the bottleneck is probably the file system.=0A= (You might want to check that there is cpu available. ie. Some amount of=0A= idle cpu, just in case?)=0A= [more stuff snipped]=0A= >I've added an L2ARC since I sent that original email. We'll see how well = it works. I >expect it to take 48 hours before results are apparent.=0A= Others have tweaked something so that more of the arc is used for metadata.= =0A= [more stuff snipped]=0A= =0A= >Experimentally, I determined that 768 threads is sufficient. But your for= mula would >suggest > 2500, which is a lot of threads! I'll keep it in min= d if I ever reach 768, >though.=0A= 768 is probably plenty. My number would be an extreme upper bound.=0A= I did it because the default of 8 per cpu is very low (assumes a machine mo= stly=0A= doing other things).=0A= =0A= Good luck with it, rick=0A= =0A= =0A= rick=0A= =0A= -Alan=0A= _______________________________________________=0A= freebsd-fs@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= =0A= From owner-freebsd-fs@freebsd.org Wed Mar 17 14:47:06 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6DFC756B611 for ; Wed, 17 Mar 2021 14:47:06 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0tKP5Ymzz4SnV for ; Wed, 17 Mar 2021 14:47:05 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id E5C382458 for ; Wed, 17 Mar 2021 10:47:01 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 17 Mar 2021 10:47:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:mime-version:content-type; s= fm2; bh=bYuayMDbgYPH+pDH5rkq4QZkpAlVx7f4mpDuWsrt/xM=; b=Uw0LwOaK wJLNUnPYrw/i5WnSgZOv5eImUUgjUm4TMTIHlavnvU95GqXVJRivvNUtuopTsD1q 86tYLxm+g1acBnoLOyEXS3+48HXIY4V5AraZ/3/0/b+tBKMhx49aHUQU4MH5eA1F 5047ny0J8nK8HO6MwpElmdvPGiifJ+8O54J9+xncdyXiC35Qv0wklnEZ40UYI0SI tzH3Rls/08EuClZrh9sZI7UTlvpWSKZN2oiaFdclJabj5DPe4flCa11WsL9UnLTD cnrBmUCWsVZLVfTvkkgm+Mm7MnSkzZ0Tln3IUSI1pvBxkwvDq5MR8/NCoqTHQX5E S0rJoOi+EWJyrA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=bYuayMDbgYPH+pDH5rkq4QZkpAlVx 7f4mpDuWsrt/xM=; b=Md23+iUzLg5S/nuchhbp+8zyeAyVF7oGfM6zYp+/KXcXC kaFROZ6rQ6CoU5wi6gR62qNt70mg77Gzm4RueTd16FhRzMjBYy12H4V3p2qNkpO2 S9mppatq9AkxPpQlWItr/QNBa7rmaCzmPqYTUH4lFEQimmnxFdTeTdp2iyQ2eUHY ixWaEnyPXY+MK80U1Lt5lqudIlMymBthlKVu4NoRiZB7+k3/nfeqI6zEzORLDghY AGRQE7N+svuIjmj1dzIUkhXf1p54nXezataVlthUkD+4QYKnxb6usUDNBWH29KFY LbKba832QdJ5Jp+YIdAgIA03MCH14NTLn2PH2unRw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudefgedgieelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkgggtugesghdtreertd dtvdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseiihiig shhtrdhnvghtqeenucggtffrrghtthgvrhhnpeevgffhffdtfeekleelhedtjeelvdfhvd egieejveffgfduvdfhteegjeeujeeuieenucfkphepkedvrdejtddrledurddutddtnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthgvtghhqd hlihhsthhsseiihiigshhtrdhnvght X-ME-Proxy: Received: from ceres.zyxst.net (ceres.zyxst.net [82.70.91.100]) by mail.messagingengine.com (Postfix) with ESMTPA id 157C7240054 for ; Wed, 17 Mar 2021 10:47:01 -0400 (EDT) Date: Wed, 17 Mar 2021 14:46:59 +0000 From: tech-lists To: freebsd-fs@freebsd.org Subject: reference for zfs on freebsd stable/12 and stable/13 and main/14 Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="mgc8tl8SHwXPqumN" Content-Disposition: inline X-Rspamd-Queue-Id: 4F0tKP5Ymzz4SnV X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=Uw0LwOaK; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Md23+iUz; dmarc=none; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 64.147.123.20 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-5.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[64.147.123.20:from]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.20]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.20:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 14:47:06 -0000 --mgc8tl8SHwXPqumN Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I know from reading around the lists that zfs is changing to OpenZFS. What I'd like to know is, is there *recent* literature (website or pdf)=20 available for reference for zfs backup/incremental backup with zfs=20 send to a remote system in the following situations: 1. 12.2-p4 system sending to recent stable/12 system 2. 12.2-p4 ststem sending to recent stable/13 system I realise some of this is in the handbook but the handbook is out-of-date in many areas and references versions (of freebsd) that are=20 no longer supported.=20 thanks, --=20 J. --mgc8tl8SHwXPqumN Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAmBSFloACgkQs8o7QhFz NAXkHg//fUTcKvngfyYdCSUk56VADt1rQ+rRRjLQBAjoP8+sERP5R7jr0GyXaWg7 5Z7J7c0+creYVAVJPxXh9ODlW/fW3ELhTvtBuYO/DK/7wtZh+hzSoi/kq5/g3wnH Acm9vTU+8FxiV+JMbsuoXUejxoG5HE4hipFVnd+setf5n+Xhb8CEdC3j7q8Q63R1 SlRuW45P4xnh1QEKZStSsddQB6/VuI2/ZlRqKVjr3YmvBGbvErW/lF2cHXp7fugB LVCj0DSftkYTMxwHkSRTCr9yOcqN6vbRKeF2Sr+dYq5UXBMVd8lWcZWDp7AUCFhO 1WNWRYDmkSCfrrrDOv3qMljrW9OBIemiGD97XCAK6subbj38F7mRpa7DzQ7IFmmv UPh0qFT9ivwjlM0xjgqFukf+zRlnX5eIzO5s+rGCnBWZymu31GH5nDPXbWvIg610 NKI+EnIT/VU9gZXs15dwDgWKb/14+EA1TSAwo4/1PUxOBORpUzL8JqJa+8n07NKj 6/aF0prxR64sNz5gD6riDm0YtKFOjF9F6TG/xSnftezTs3tGrACMgs/sf2D0R+qu 3bYDpCSX2hj1pZrHulZ4UJQ2CtkbYyhDBax3tv6bci6Ruu0AwTJbs2MnE7Iegp11 zrEWfLL20OkarTPydvC0tneXUZxQlWsS7KPHHV5mYfqHWcnhXi0= =MptM -----END PGP SIGNATURE----- --mgc8tl8SHwXPqumN-- From owner-freebsd-fs@freebsd.org Wed Mar 17 21:48:44 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A798579B6F for ; Wed, 17 Mar 2021 21:48:44 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670082.outbound.protection.outlook.com [40.107.67.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F13gv0gqQz3Mrt; Wed, 17 Mar 2021 21:48:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XSlJe0vq+wm4TA80eFJoz7zI89mYBhshe9zPKQKz9MOQbnC8ZJehM/+nr/+XXSvWNBq0PGD8WR6u2hSMHqGIe5JnLO2HG/dwvCnYTOc4hJXVXb9pq9vUAIsXFvOwuAOzzICUQ8fyntUU6H9hg9jTy0WUmPkLj8VOGNIlsqQ2x3uKiLE6H7JLCP/esSyAzcMo5CuM9OD/kftO98lk0ANjJMNP0DEnnz9ChnZx/V6NNl5D2vdcFjaVT5XZB16QmIsqTqgGMvO2xVOYCzVxtdccIWk52PZKi2OVx5JyPuZpVh0aDeELvpL+qaKBjsqgOtRg2LmAl2zWkwO4VxkGFGgffA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DPEvm0/qKlhDJlzhuiDcziP+SZAPTJvTqWhnuAVz9SI=; b=hyt39gLoSh4N+JSwCSzEAPQ4CXChBvk4o6XQWjsfglFC7aTYJLxTXPtrU+gRgwKJItDn4pXzU4tdH1atP1YtX1J0wMD4BiiPKYIKLUSNQIBU05cHAwFjT1bF/6we9KSXR9I/3m4Q9hVZdUrKfwCpj6zQO7MixMX/tVPokF1n4rC6UnFXqzE1TWPtTYzfy2UxC6oUqpQoHn4/fCdim8uK27h4jmCzoTKVYs756CM4iSs8YIH5ishNvsFkZ3KRrbfFvksit8k9LsCIVNITctL/8co+pfKTE7vAL/IKwCiWlGpz5r4/9cf01RomfuK2T14hl6ti1LwoHwPIb4Y+v1//XA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DPEvm0/qKlhDJlzhuiDcziP+SZAPTJvTqWhnuAVz9SI=; b=ZpfgDTZQwiEtBEjmpmaDiZlWB9MHKYDGa6D5YxTdFP3ctbgrvA42tgnGhrEqq9T93t4t8g2UXzJk6PMflJReM27xtpmEe8mdX/0pxNRJL3i2wjbCq+5EwQPzjK6padTYojQUV+4ZGmdb3gCEkRkMzJDOAMGSn/tf2o3MtBGNig5tJlRfPEIaadYSxB62hknzWkveCfwx29gp0JeIiTX5n1LBo7fryqCyZR0Xmzn1oBguRVjq2gv9ulbGjDtiJJdL+LQV6zz3ZU99PpmB462tf+Yb0Jy+67X7fbxIrQ/8HQ1Dqe0ZsYL/Lv+y23ORPmwBzmOWUCvwA+JEToBUbYmdRQ== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB3432.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:52::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar 2021 21:48:41 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021 21:48:41 +0000 From: Rick Macklem To: Alan Somers CC: freebsd-fs Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Thread-Topic: Understanding the vfs.nfsd.request_space* sysctls Thread-Index: AQHXGom0SMOUlewkdkSbAUM6yshgqaqHSCtTgAAdewCAAVEfeg== Date: Wed, 17 Mar 2021 21:48:41 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1c9e5862-1dc7-4773-db70-08d8e98e6e45 x-ms-traffictypediagnostic: YQXPR01MB3432: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: QgeRAAo+d1w0uz/+9hzSEtkpm/NjyyuW0aPpCfkeshZVe743SG/nN/eJGETzSsTKKflFq3gL4wXM/Ht4LQgFCiJNshm4Q60/S8eQN+5yL2+t9CJGqv/utVRmOqpFIgrhfDZN7FYjj/f8aiwHc2C/l0VK63z/GAtP/4QmZYB7WQVYB+PWHK5M7xJJnTXoI9Qdc4jHqvmKbnmKAF3HuHP0wqbRwF1tI9QVmmiH74QJhQz/RXgbjqu6l5MvCcUPtE2ZMCmuThPSQq/gruTlm2bSbFPI6vBY9rfr9LWnBXRHaS41gZfxxslbAEg9ojPp1dhJnTVBhL83tYOBK6Hvh+VP7qHfi/ObWS07kbD3TSyRb4qlCNTwk6o3et3Dawh5gTaZHwNMhehYZR6BZSE+rmGNTmHAS80dYDjRMDxErhY8q9whPzpjSuufuBIQIOnrtSEv3ulBGB4HX9rjKDfebWznT+jFm5/5dLMRhNHdF6LrUU8zr1u7Xq6GsiY427cUu083/6sCd8WLvNAlU74vjIkg8Rv40lkmAlFIpRjsQZV/9AaKpxXppHLxYpiHqnfBYbwmV2uQhZrq+mxY1747QQ+Q9JmKEatN/En+NKbtp6njC52dYrews4h5BFRDDA90KVbn0iB/jofDbyr/DEanZt+cZQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(136003)(376002)(366004)(396003)(39860400002)(346002)(8936002)(2906002)(33656002)(9686003)(8676002)(86362001)(316002)(52536014)(76116006)(55016002)(186003)(478600001)(83380400001)(6916009)(71200400001)(786003)(6506007)(7696005)(66946007)(66476007)(66446008)(66556008)(64756008)(91956017)(5660300002)(4326008)(450100002)(966005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?JdoC7+FN/ZSrf8SWKJn7fiLnownfmWMV1nNlgULrSEY9EcXM0kw+Q44IEa?= =?iso-8859-1?Q?2eZPXgCNQbkKbhWyrDrreu41emQcpQZaSBo6oUrGmob89+XfrDGuhwMoTY?= =?iso-8859-1?Q?lOzxze8I5GyOrm7VA3Xa6muqNRUPZVf/CWwOW/jIHfVep8ao6eJ/OGd1bR?= =?iso-8859-1?Q?QL0ZayAWvKNMtzQcVTFf8URffJRnwHzj9ME6l69yMnblq8MFCgOX5z/I4b?= =?iso-8859-1?Q?prvcU8UMMuACBqKV2c8ndkUeAJXz58g8JfUl3hgvMhaI/NBf92vuaZ39//?= =?iso-8859-1?Q?clXGqslcfhsHH4nwwgYMY2WQt/8Ge8ElJ63pQh8B8z16dwDEpzEo9k626s?= =?iso-8859-1?Q?JQ1tJhEm4XTa0vIAhCnE4ghzwt3ZLjt6WrBIU96BuCHi2a80WBr4Afkwxe?= =?iso-8859-1?Q?mt64oFYd9DmreErgb4AsrHVoU7uN9ToS/TPSYIRlfNIsqX2LeElSZSk4EZ?= =?iso-8859-1?Q?yZkj8zkAdFHsj06Jm/cELg3RFiaRLe1RCsZnOa9Fc1jYkUGO+SN8F0E265?= =?iso-8859-1?Q?H0gDuS+VIlA/gDOGcWcbGXNsTFsy1HUPW/36jCw1X4xUdVqnWBKOt0L0eB?= =?iso-8859-1?Q?7vJvGg2peUOdOKjiAZeKLpM4q7g8jL/cHLVhJLGUvZs8J3vB3FDm3z88bD?= =?iso-8859-1?Q?VYw6Ea+7v9kwC3l7gdNq9wBqehUAzDH2OqaFjippVQVF2uczZ6r6QpnF6s?= =?iso-8859-1?Q?cMmHqytm1GIWnjOfLln6lv3lXC8ZK201dnUjbnMYjRiR3nVuOxu5sJbGSU?= =?iso-8859-1?Q?tWAAZkNp7a2xrqCHuk+z7ijGef72ulYToKKb7/tZOwJ3aYYv3zzxYDCFhD?= =?iso-8859-1?Q?tqf20v1aeUsvqtc7+ixVLTOoXQyXll1d06VwHpYd74XeY6Ggw0+6p+qrJ6?= =?iso-8859-1?Q?f2xoBgTRM6rFDACXI3UJNoR2j3DLqz0wtdK/DhzIoqov6oov1zKXwqKWgk?= =?iso-8859-1?Q?ya1Dupbi2uv/+Z3F0qYHNx3AsjkzYu/OvG5TPjGs+ViVhf0NqJcr9iHs7Z?= =?iso-8859-1?Q?XXd2AqUfK5CeKJbrwKFwvsc8q1eIIfc6D8qor+5JAHrKqNq6Dyagejq1g7?= =?iso-8859-1?Q?r8jCYy24vVdgHHQz3tJ2DPqjav4o1hx/OBwhxiSVfcfgJj2NwKVhsvWwrF?= =?iso-8859-1?Q?T2QT0MLfitvaHWj3DaNUKt8226QcFOJvuflQ7OMs8yCilohh3jkQGOMfhb?= =?iso-8859-1?Q?GfekHnQrGUZOPsGrrTjRwC4dprzauSnoHgfgDhY0XWb0HlJk6B6VZtzrHt?= =?iso-8859-1?Q?x9M7DAv4/I84TfJUddt2tLZVXdRvZ+8QBRI1pRrqlc131bHWHp/BKSsrly?= =?iso-8859-1?Q?sSL+7AWwyoUE/qgiR0/FjO+ULY1dZInd0t96wEQ59LSfeVH8mpmAZfviXr?= =?iso-8859-1?Q?HKNofDtF2M8ncQh4Tz1LtiTKySPQaCT1I7BC8sTcdXFxLWVdvrv7E=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 1c9e5862-1dc7-4773-db70-08d8e98e6e45 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 21:48:41.6975 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +3Jl6bY+r8jMnnWOZVfc8VT/YEjRrXY0ot0S1UQV383wL76VZDq707Ppqe6b1hrOmClV/Qh39ETIX2UKNyubmg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3432 X-Rspamd-Queue-Id: 4F13gv0gqQz3Mrt X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=ZpfgDTZQ; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.82 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.82:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[40.107.67.82:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.82:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-fs]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.82:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 21:48:44 -0000 Alan Somers wrote:=0A= >Rick Macklem wrote:=0A= [stuff snipped]=0A= >>Anytime used > high, it throttles, which means it leaves=0A= >>the RPC messages on the socket receive queue.=0A= >>This indicates the server is not keeping up with requests.=0A= >>(ie Overloaded or ???)=0A= >=0A= >That all makes sense. So having a high request_space_used basically just = means that >my storage is too slow.=0A= One other thing you might want to look at is rcv interrupt moderation=0A= in the net device driver.=0A= --> This *might* result in received RPC requests being "batched", resulting= in=0A= a burst of RPCs that overloads the server. (Except for write requests= , each=0A= RPC is a small message typically in a single small packet.)=0A= --> Disabling rcv interrupt moderation will result in more interrupts= , but=0A= then each RPC request will be handled right away. This might r= esult=0A= in more evenly spaced RPCs (in a temporal sense).=0A= Also, delaying processing of RPC requests increases RPC RTT an= d that=0A= always has a negative impact on NFS performance.=0A= =0A= rick=0A= =0A= > And how can I either=0A= >increase the available space or decrease the stuff that's using it?=0A= Increasing vfs.nfsd.request_space_high avoids the throttling,=0A= but it is hard to say that is a good idea, since there won't be=0A= backpressure applied to the clients via TCP windows, etc.=0A= =0A= Fixing the server so that it isn't overloaded would be better,=0A= I think?=0A= --> I'm the last guy to take ZFS advice from, but I think there=0A= is a tunable w.r.t. how much arc is used for metadata.=0A= Getattrs will need cached (metadata) to reply quickly.=0A= Lookups also depend on cached attributes for good=0A= perf. as well.=0A= =0A= I've added an L2ARC since I sent that original email. We'll see how well i= t works. I expect it to take 48 hours before results are apparent.=0A= =0A= --> Make sure you have lots of nfsd threads. They are=0A= just kernel threads, so they don't use a lot of resources=0A= and having too many is much better than not enough.=0A= -->I can't remember what the upper limit is these days,=0A= but I'd set it to that for a busy nfs server.=0A= For NFSv4.1, each client can send a maximum of=0A= session_slot_table_size concurrent RPCs. FreeBSD=0A= uses a single 64slot table for each mount. I think=0A= Linux uses a 32slot table, but supports trunking.=0A= I don't know if the Linux client uses a separate=0A= session table for each trunk or not?=0A= --> Something like #clients * 32 (or 64) nfsd=0A= threads running on the server.=0A= =0A= Experimentally, I determined that 768 threads is sufficient. But your form= ula would suggest > 2500, which is a lot of threads! I'll keep it in mind = if I ever reach 768, though.=0A= =0A= =0A= rick=0A= =0A= -Alan=0A= _______________________________________________=0A= freebsd-fs@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= =0A=