From owner-freebsd-fs@freebsd.org Sun Mar 14 08:35:40 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3BEC5577AAA
for <freebsd-fs@mailman.nyi.freebsd.org>; Sun, 14 Mar 2021 08:35:40 +0000 (UTC)
(envelope-from kostikbel@gmail.com)
Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 4DytDC0Hctz3Cwl
for <freebsd-fs@freebsd.org>; Sun, 14 Mar 2021 08:35:38 +0000 (UTC)
(envelope-from kostikbel@gmail.com)
Received: from tom.home (kib@localhost [127.0.0.1])
by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 12E8ZPZU019088
(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO);
Sun, 14 Mar 2021 10:35:28 +0200 (EET)
(envelope-from kostikbel@gmail.com)
DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 12E8ZPZU019088
Received: (from kostik@localhost)
by tom.home (8.16.1/8.16.1/Submit) id 12E8ZOt8019080;
Sun, 14 Mar 2021 10:35:24 +0200 (EET)
(envelope-from kostikbel@gmail.com)
X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com
using -f
Date: Sun, 14 Mar 2021 10:35:24 +0200
From: Konstantin Belousov <kostikbel@gmail.com>
To: Alexander Lochmann <alexander.lochmann@tu-dortmund.de>
Cc: freebsd-fs@freebsd.org
Subject: Re: [RFC] Understanding the locking of struct buf
Message-ID: <YE3KzAiZwcSZZLWS@kib.kiev.ua>
References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de>
<YEwx0WdSuJpzhONL@kib.kiev.ua>
<c49dc72a-da0f-a089-7e93-e4e54d0c03eb@tu-dortmund.de>
<YEy3JET6Lx7BkJP3@kib.kiev.ua>
<48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de>
X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00,
DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM,
NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4
X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home
X-Rspamd-Queue-Id: 4DytDC0Hctz3Cwl
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org; dkim=none;
dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com
(policy=none);
spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor
denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com
X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_TLS_ALL(0.00)[];
ARC_NA(0.00)[];
DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none];
NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[];
TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com];
NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain];
HAS_XAW(0.00)[];
RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from];
R_SPF_SOFTFAIL(0.00)[~all];
SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255];
TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000];
RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[];
R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com];
ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US];
MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-fs];
RCVD_COUNT_TWO(0.00)[2]
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 14 Mar 2021 08:35:40 -0000
On Sat, Mar 13, 2021 at 05:29:39PM +0100, Alexander Lochmann wrote:
>
>
> On 13.03.21 13:59, Konstantin Belousov wrote:
> > What consistency? If you are talking about multithreading memory model
> > as expected by the FreeBSD kernel, look at atomic(9). It has assumptions
> > like atomicity of naturally-aligned word-sized integer accesses written
> > out explicitly.
> >
> Yeah, I think that's what I meant. Thx!
>
> In sys/sys/buf.h, it says "All fields are protected by the buffer lock
> except those marked [...]". How does this fit together with atomic(9)?
> If I had to implement some extension to vfs_bio.c, for example, I would
> follow those locking rules without knowing that I may access some fields
> without any lock.
You cannot write any working code that uses buffer cache, without studying
both buffer cache code itself, and examples using it from filesystems.
Short (even the whole-screen short) herald comment cannot explain all the
nuances that grown in the 40+ years codebase.
> Shouldn't that particular piece of documentation be updated?
> For example: b_bcount /* w: b_lock, r: no lock needed */
Do you understand the purpose of the locking? What do you intent to do
with the read value of b_bcount, if you do not own the buffer? What is
you next action with that value?
> >> In FreeBSD, do I have to use lock X in any case except Y and Z?
> >> Or is it the other way round: Do I need no lock at all except for case X
> >> and Y?
> > I do not understand this question(?).
> >
> I'm sry. Maybe my question is just nuts.>>
> >>> Are you reporting a bug or just asking about LK_KERNPROC. Lockmgr
> >>> (kern_lock.c)is careful enough to handle LK_KERNPROC owner specially. In
> >>> particular, it does not report unlock of such lock to witness.
> >> First of all, I want to understand how things in FreeBSD work.
> >> >From what I understand now: When setting up an asynchronous IO request,
> >> buf.b_lock is taken by thread X. Later on LK_KERNPROC is used to hand
> >> over the ownership to the kernel. The lock itself is still acquired.
> > The lock is acquired in the sense that nobody else can take the buffer'
> > lock until the call to lockmgr(LK_RELEASE).
> So can we safely assume, for our analysis, that the b_lock is still
> acquired during bufdone and the other related fns?
LK_KERNPROC buffer is logically owned by the io completion thread(s).
> But from this point, there
> > is no thread owning the lock. Consider that the lock was converted to
> > the 1-counting semaphore.I'm not sure. What do you mean bei 1-counting semaphore?
> >
> >> The completion of the IO job is performed in the kernel's context, which
> >> releases buf.b_lock in brelse().
> > The completion for async IO is performed in the context of some other
> > thread, typically either geom io up thread, or direct completion thread
> > of some disk driver. This is why this somewhat strange LK_KERNPROC
> > business is needed at all.
> >
> > For sync IO, that thread only signals original thread that io completed.
> > In this case, no LK_KERNPROC trick is performed.
> >
> That is clear to me.
> >> So there is no explicit lock call in the latter context, is it?
> > No lock call, but there is an unlock.
> Are you talking about BUF_UNLOCK()?
>
> > Assuming you wrote the stack bottom-up, this is exactly what I wrote above:
> > xpt_done_td is CAM IO completion thread, and it performs actions after hw
> > informed that io request (bio) was completed.
> >
> Thx.
> >> allocbuf
> > When brelse() notes that buffer was marked as 'no cache', it demolishes
> > the buffer right after async io finishes. Perhaps this is the case that
> > you observed.
>
> Yeah, but maybe our approach is just inaccurate. Due to the
> WITNESS_UNLOCK() call in __lockmgr_disown(), we assume the b_lock has
> already been released.
>
> - Alex
>
> --
> Technische Universität Dortmund
> Alexander Lochmann PGP key: 0xBC3EF6FD
> Otto-Hahn-Str. 16 phone: +49.231.7556141
> D-44227 Dortmund fax: +49.231.7556116
> http://ess.cs.tu-dortmund.de/Staff/al
From owner-freebsd-fs@freebsd.org Sun Mar 14 21:00:24 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 757155B26BB
for <freebsd-fs@mailman.nyi.freebsd.org>; Sun, 14 Mar 2021 21:00:24 +0000 (UTC)
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3])
by mx1.freebsd.org (Postfix) with ESMTP id 4DzBlX1Q9qz4l9N
for <freebsd-fs@freebsd.org>; Sun, 14 Mar 2021 21:00:24 +0000 (UTC)
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: by mailman.nyi.freebsd.org (Postfix)
id 0AFBF5B23FA; Sun, 14 Mar 2021 21:00:24 +0000 (UTC)
Delivered-To: fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 07C025B2719
for <fs@mailman.nyi.freebsd.org>; Sun, 14 Mar 2021 21:00:24 +0000 (UTC)
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4DzBlW3d9tz4lCk
for <fs@FreeBSD.org>; Sun, 14 Mar 2021 21:00:23 +0000 (UTC)
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
[IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 88F6723F58
for <fs@FreeBSD.org>; Sun, 14 Mar 2021 21:00:22 +0000 (UTC)
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12EL0M1q024926
for <fs@FreeBSD.org>; Sun, 14 Mar 2021 21:00:22 GMT
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: (from bugzilla@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12EL0Mi7024925
for fs@FreeBSD.org; Sun, 14 Mar 2021 21:00:22 GMT
(envelope-from bugzilla-noreply@FreeBSD.org)
Message-Id: <202103142100.12EL0Mi7024925@kenobi.freebsd.org>
X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to
bugzilla-noreply@FreeBSD.org using -f
From: bugzilla-noreply@FreeBSD.org
To: fs@FreeBSD.org
Subject: Problem reports for fs@FreeBSD.org that need special attention
Date: Sun, 14 Mar 2021 21:00:22 +0000
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Content-Filtered-By: Mailman/MimeDel 2.1.34
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 14 Mar 2021 21:00:25 -0000
To view an individual PR, use:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id).
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.
Status | Bug Id | Description
------------+-----------+---------------------------------------------------
New | 203492 | mount_unionfs -o below causes panic
Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f
Open | 211491 | System hangs after "Uptime" on reboot with ZFS
Open | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data
Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w
Open | 240831 | zfs: Panic during snapshot on 12.1-STABLE r352648
Open | 243973 | [zfs] rollback segmentation fault
Open | 244656 | zfs: resilver doesn't provide enough information
Open | 244692 | gjournal: Does not support TRIM
Open | 244899 | [PATCH] zfs: xattr on a symlink target > 136 caus
Open | 251035 | [zfs] Allow 64 bit ZFS to support 32 bit ioctls (
11 problems total for which you should take action.
From owner-freebsd-fs@freebsd.org Tue Mar 16 00:07:52 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4255856CC59
for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 00:07:52 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::50:13])
by mx1.freebsd.org (Postfix) with ESMTP id 4DztsN1Cx6z4fCX
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 00:07:52 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: by mailman.nyi.freebsd.org (Postfix)
id 27FBA56CCD0; Tue, 16 Mar 2021 00:07:52 +0000 (UTC)
Delivered-To: fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 27C4656CC58
for <fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 00:07:52 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4DztsN0Wpnz4f9C
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 00:07:52 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
[IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 04FB719D1F
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 00:07:52 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12G07pNH055976
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 00:07:51 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from bugzilla@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12G07pgJ055975
for fs@FreeBSD.org; Tue, 16 Mar 2021 00:07:51 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to
bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 253158] Panic: snapacct_ufs2: bad block - Non-suJ mksnap_ffs(8)
crash
Date: Tue, 16 Mar 2021 00:07:52 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: Unspecified
X-Bugzilla-Keywords: panic
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: commit-hook@FreeBSD.org
X-Bugzilla-Status: Closed
X-Bugzilla-Resolution: FIXED
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-253158-3630-vTkMDuRfkc@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-253158-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-253158-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 00:07:52 -0000
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253158
--- Comment #55 from commit-hook@FreeBSD.org ---
A commit in branch stable/12 references this bug:
URL:
https://cgit.FreeBSD.org/src/commit/?id=3Dcf0310dfefee8672680fb45b7ee25722e=
7630227
commit cf0310dfefee8672680fb45b7ee25722e7630227
Author: Kirk McKusick <mckusick@FreeBSD.org>
AuthorDate: 2021-02-12 05:31:16 +0000
Commit: Kirk McKusick <mckusick@FreeBSD.org>
CommitDate: 2021-03-16 00:11:29 +0000
Fix bug 253158 - Panic: snapacct_ufs2: bad block - mksnap_ffs(8) crash
PR: 253158
(cherry picked from commit 8563de2f2799b2cb6f2f06e3c9dddd48dca2a986)
(cherry picked from commit c31480a1f66537e59b02e935a547bcfc76715278)
sys/ufs/ffs/ffs_snapshot.c | 145 ++++++++++++++++++++++++-----------------=
----
1 file changed, 78 insertions(+), 67 deletions(-)
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From owner-freebsd-fs@freebsd.org Tue Mar 16 08:53:42 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 306B557CE4F
for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 08:53:42 +0000 (UTC)
(envelope-from alexander.lochmann@tu-dortmund.de)
Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de
[129.217.128.51])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "unimail.tu-dortmund.de",
Issuer "DFN-Verein Global Issuing CA" (not verified))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F06X52DQGz3lrX
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 08:53:40 +0000 (UTC)
(envelope-from alexander.lochmann@tu-dortmund.de)
Received: from [129.217.43.55] (chulak.cs.uni-dortmund.de [129.217.43.55])
(authenticated bits=0)
by unimail.uni-dortmund.de (8.16.1/8.16.1) with ESMTPSA id 12G8rdNZ009707
(version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT);
Tue, 16 Mar 2021 09:53:39 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de;
s=unimail; t=1615884819;
bh=VZQT1WYJRnMfCFIneMtYVfnM1uFh+wkyAfK8Ylp2y1A=;
h=To:Cc:References:From:Subject:Date:In-Reply-To;
b=Ud7Z0NhER2K7jLHdiLWK7f9rwF8kBx5/wh7h/Gn8ymUyro1S+sYQ6wXeb2Em0cui+
Hs5XQCRB3EYODTNtokJ90zQBHhhktGynsWUeM0zj8anKrk0C8jKCS2wjOD2OhzsO+z
l9cAeu1GtUkdDmqm81CTStIy3JB58aXGmBukdXHQ=
To: Konstantin Belousov <kostikbel@gmail.com>
Cc: freebsd-fs@freebsd.org
References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de>
<YEwx0WdSuJpzhONL@kib.kiev.ua>
<c49dc72a-da0f-a089-7e93-e4e54d0c03eb@tu-dortmund.de>
<YEy3JET6Lx7BkJP3@kib.kiev.ua>
<48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de>
<YE3KzAiZwcSZZLWS@kib.kiev.ua>
From: Alexander Lochmann <alexander.lochmann@tu-dortmund.de>
Subject: Re: [RFC] Understanding the locking of struct buf
Message-ID: <9e591bfe-1790-251e-a90c-79c6ec66b3ac@tu-dortmund.de>
Date: Tue, 16 Mar 2021 09:53:39 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
Thunderbird/78.7.1
MIME-Version: 1.0
In-Reply-To: <YE3KzAiZwcSZZLWS@kib.kiev.ua>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 4F06X52DQGz3lrX
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=tu-dortmund.de header.s=unimail header.b=Ud7Z0NhE;
dmarc=none;
spf=pass (mx1.freebsd.org: domain of alexander.lochmann@tu-dortmund.de
designates 129.217.128.51 as permitted sender)
smtp.mailfrom=alexander.lochmann@tu-dortmund.de
X-Spamd-Result: default: False [-5.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[];
TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:129.217.128.0/24];
RWL_MAILSPIKE_GOOD(0.00)[129.217.128.51:from];
RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
RCVD_IN_DNSWL_MED(-0.20)[129.217.128.51:from];
RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[tu-dortmund.de:+];
FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[];
MIME_TRACE(0.00)[0:+]; NEURAL_HAM_SHORT(-1.00)[-1.000];
ASN(0.00)[asn:680, ipnet:129.217.0.0/16, country:DE];
MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[];
NEURAL_HAM_MEDIUM(-1.00)[-1.000];
R_DKIM_ALLOW(-0.20)[tu-dortmund.de:s=unimail];
FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000];
MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tu-dortmund.de];
DWL_DNSWL_LOW(-1.00)[tu-dortmund.de:dkim];
TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2];
RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]
X-Mailman-Approved-At: Tue, 16 Mar 2021 11:19:46 +0000
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 08:53:42 -0000
On 14.03.21 09:35, Konstantin Belousov wrote:
> You cannot write any working code that uses buffer cache, without studying
> both buffer cache code itself, and examples using it from filesystems.
> Short (even the whole-screen short) herald comment cannot explain all the
> nuances that grown in the 40+ years codebase.
I see.
>
>> Shouldn't that particular piece of documentation be updated?
>> For example: b_bcount /* w: b_lock, r: no lock needed */
> Do you understand the purpose of the locking? What do you intent to do
> with the read value of b_bcount, if you do not own the buffer? What is
> you next action with that value?
>
I do know the purpose of locking. I have no particular usage in mind at
the moment. I'm trying to understand how locking is handled in FreeBSD.
In the Linux kernel, for example, there are such exceptions: Read a ptr
without a lock to perform a NULL check.
All I'm asking was whether the documentation should reflect those
exceptions. I assume the answer is no.
Thank you, Konstantin, for lifting the fog.
- Alex
--
Technische Universität Dortmund
Alexander Lochmann PGP key: 0xBC3EF6FD
Otto-Hahn-Str. 16 phone: +49.231.7556141
D-44227 Dortmund fax: +49.231.7556116
http://ess.cs.tu-dortmund.de/Staff/al
From owner-freebsd-fs@freebsd.org Tue Mar 16 17:27:51 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id CFDB85B61DB
for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 17:27:51 +0000 (UTC)
(envelope-from asomers@gmail.com)
Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com
[209.85.167.178])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0KxM1GLgz4rLg
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 17:27:50 +0000 (UTC)
(envelope-from asomers@gmail.com)
Received: by mail-oi1-f178.google.com with SMTP id u198so33945519oia.4
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 10:27:50 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
bh=uwP1LCsefw81DoC0u/ui0qf8kKoVXJmMaf5nObab0tM=;
b=TXAFiqNVXSPjAsXOj+y5ll37a7DQ1Cio5OABY5vpH1xU+JV3REVZW5Fu4r6qPxrCx4
QbRekPXlctsgTDGiQes0uU7hTnf/ywur/dXC4WdCJh/FLtxhWhGoXIoV6G0pU1DUWp4V
pLNh6A/jehwFPnh9e+uPJTtamVaK9KVGxqt23uImqs2AROUwj39tWtpqo5LX3+dzCsxe
YXUM4JkpgZXM+4Vgu1MH/+ElBpa9yZvLYOiXchXSz2X47bosRCeX4xtz/YHzrf6stn3T
twsXDjj7+NHawhYFlh2Kvc2QG+VndXFCBsC6B0SChsbpWauDFpim2bd5HSwnEi8Cji0k
MqjA==
X-Gm-Message-State: AOAM532NE8cB0UIEliF8Yj9E8T//oTe1Lr7n0pNcV7F+8HTguCzFqOpW
lQ4iTxVc/dJ84EXGUdYT/2AjactZYU/4bw5RY5ZdMaFs
X-Google-Smtp-Source: ABdhPJzLm1+CFgIDkCDu/TIwc6xA7FjjDR98U5P+u///m9upHVGCNBB7+xy4p7lDuH7iJcwgP1XN2FX/WSKxHYoT/Eg=
X-Received: by 2002:aca:3046:: with SMTP id w67mr98669oiw.57.1615915669713;
Tue, 16 Mar 2021 10:27:49 -0700 (PDT)
MIME-Version: 1.0
From: Alan Somers <asomers@freebsd.org>
Date: Tue, 16 Mar 2021 11:27:39 -0600
Message-ID: <CAOtMX2geF7Tv7VYc9WtK23a+nDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com>
Subject: Understanding the vfs.nfsd.request_space* sysctls
To: freebsd-fs <freebsd-fs@freebsd.org>
X-Rspamd-Queue-Id: 4F0KxM1GLgz4rLg
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none;
spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates
209.85.167.178 as permitted sender) smtp.mailfrom=asomers@gmail.com
X-Spamd-Result: default: False [-3.00 / 15.00];
R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_DN_ALL(0.00)[];
NEURAL_HAM_SHORT(-1.00)[-1.000];
FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com];
MIME_TRACE(0.00)[0:+,1:+,2:~];
RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.167.178:from];
FREEMAIL_ENVFROM(0.00)[gmail.com];
ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US];
FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com];
TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[];
NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[asomers];
R_DKIM_NA(0.00)[]; FROM_HAS_DN(0.00)[];
TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000];
MIME_GOOD(-0.10)[multipart/alternative,text/plain];
PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1];
SPAMHAUS_ZRD(0.00)[209.85.167.178:from:127.0.2.255];
RCVD_IN_DNSWL_NONE(0.00)[209.85.167.178:from];
RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.178:from];
RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[];
MAILMAN_DEST(0.00)[freebsd-fs]
Content-Type: text/plain; charset="UTF-8"
X-Content-Filtered-By: Mailman/MimeDel 2.1.34
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 17:27:51 -0000
Could somebody help me understand the meaning of the various
vfs.nfsd.request_space* sysctls, and what implication they have for tuning
NFS?
I have several NFS servers. Most are performing well, but one with a
metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAttr
and Lookup ops) is not. Total throughput is a not-unreasonable several
hundred MB/s, but some clients are limited to very slow speeds, sometimes
writing at < 10 MB/s.
The request_space sysctls show some pretty stark divergence between the
well-performing and poor-performing servers:
sysctl well-performing poor-performing
request_space_used < 6 M varies, but
currently 40 M
request_space_used_highest < 39 M 24 G
request_space_throttle_count 0 35
So what does request_space_used measure, anyway? And how can I either
increase the available space or decrease the stuff that's using it?
-Alan
From owner-freebsd-fs@freebsd.org Tue Mar 16 20:27:49 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id CDFF756C418
for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 20:27:49 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::50:13])
by mx1.freebsd.org (Postfix) with ESMTP id 4F0Px15CnDz3MhD
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 20:27:49 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: by mailman.nyi.freebsd.org (Postfix)
id B10A256C16E; Tue, 16 Mar 2021 20:27:49 +0000 (UTC)
Delivered-To: fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id B0CA856C16C
for <fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 20:27:49 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Px14YBmz3MhC
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 20:27:49 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
[IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8F29729B55
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 20:27:49 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GKRn6c020667
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 20:27:49 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GKRnY9020666
for fs@FreeBSD.org; Tue, 16 Mar 2021 20:27:49 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to
bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev
Date: Tue, 16 Mar 2021 20:27:49 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: CURRENT
X-Bugzilla-Keywords:
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: vangyzen@FreeBSD.org
X-Bugzilla-Status: New
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields: cc assigned_to
Message-ID: <bug-254349-3630-MdrNhKw5xT@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 20:27:49 -0000
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349
Eric van Gyzen <vangyzen@FreeBSD.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |mjg@FreeBSD.org,
| |mm@FreeBSD.org,
| |mmacy@FreeBSD.org
Assignee|bugs@FreeBSD.org |fs@FreeBSD.org
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From owner-freebsd-fs@freebsd.org Tue Mar 16 21:23:37 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id B119156E339
for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 21:23:37 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3])
by mx1.freebsd.org (Postfix) with ESMTP id 4F0R9P4RXKz3jHg
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 21:23:37 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: by mailman.nyi.freebsd.org (Postfix)
id 96C1056DEEC; Tue, 16 Mar 2021 21:23:37 +0000 (UTC)
Delivered-To: fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 968B956E703
for <fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 21:23:37 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0R9P3mzyz3j9Y
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:23:37 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
[IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 74C462ABB7
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:23:37 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GLNbBj054477
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:23:37 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GLNbJ0054476
for fs@FreeBSD.org; Tue, 16 Mar 2021 21:23:37 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to
bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev
Date: Tue, 16 Mar 2021 21:23:37 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: CURRENT
X-Bugzilla-Keywords:
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: avg@FreeBSD.org
X-Bugzilla-Status: New
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-254349-3630-rMiejZ5AaL@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 21:23:37 -0000
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349
--- Comment #1 from Andriy Gapon <avg@FreeBSD.org> ---
Maybe that's because loader does not support booting from such a pool?
Just a speculation on my part.
Removing a vdev means that now there is vdev indirection and other
complications that require explicit handling.
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From owner-freebsd-fs@freebsd.org Tue Mar 16 21:37:48 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id DFA7E56F0A6
for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 21:37:48 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3])
by mx1.freebsd.org (Postfix) with ESMTP id 4F0RTm5q6nz3kC2
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 21:37:48 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: by mailman.nyi.freebsd.org (Postfix)
id C7B4856F0A5; Tue, 16 Mar 2021 21:37:48 +0000 (UTC)
Delivered-To: fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id C77FB56EC47
for <fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 21:37:48 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0RTm5DdDz3kFD
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:37:48 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
[IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A6BAB2AF55
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:37:48 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GLbmne063043
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:37:48 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GLbmTs063042
for fs@FreeBSD.org; Tue, 16 Mar 2021 21:37:48 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to
bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev
Date: Tue, 16 Mar 2021 21:37:48 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: CURRENT
X-Bugzilla-Keywords:
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: vangyzen@FreeBSD.org
X-Bugzilla-Status: New
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-254349-3630-tzewFe1Rgo@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 21:37:48 -0000
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349
--- Comment #2 from Eric van Gyzen <vangyzen@FreeBSD.org> ---
mm@ filed an upstream issue for this. Thanks, Martin!
https://github.com/openzfs/zfs/issues/11762
Andriy: Fair question. My workstation, running main, boots from such a po=
ol.=20
That's how I found the issue. My old SSD was failing, so I migrated to a n=
ew
one. For historical reasons, my pool was split across two partitions on the
old disk. I wanted to use only one on the new disk, so I didn't use mirror=
ing
for the migration. I simply added and removed vdevs.
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From owner-freebsd-fs@freebsd.org Tue Mar 16 21:46:18 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id C4D3356F15C
for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 21:46:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::50:13])
by mx1.freebsd.org (Postfix) with ESMTP id 4F0RgZ4vBLz3kvf
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 21:46:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: by mailman.nyi.freebsd.org (Postfix)
id A806756F51B; Tue, 16 Mar 2021 21:46:18 +0000 (UTC)
Delivered-To: fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id A7D4056F15A
for <fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 21:46:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0RgZ4J7Nz3kqx
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:46:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
[IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 850952B18F
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:46:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GLkIPD067766
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:46:18 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GLkIks067765
for fs@FreeBSD.org; Tue, 16 Mar 2021 21:46:18 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to
bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev
Date: Tue, 16 Mar 2021 21:46:18 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: CURRENT
X-Bugzilla-Keywords:
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: mm@FreeBSD.org
X-Bugzilla-Status: New
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-254349-3630-g9VeU8xPVc@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 21:46:18 -0000
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349
--- Comment #3 from Martin Matuska <mm@FreeBSD.org> ---
It looks like VDEV_TYPE_INDIRECT is handled at stand/libsa/zfs/zfsimpl.c
The code originates in the device removal feature:
https://github.com/openzfs/zfs/commit/a1d477c24c7badc89c60955995fd84d311938=
486
By modifying the vdev_is_bootable() function, setting bootfs is not allowed
anymore. I think this is an easy fix and really not necessary on FreeBSD.
Thomas Soome has been dealing with support of indirect vdevs at end of 2019,
see:
https://svnweb.freebsd.org/base?view=3Drevision&revision=3D355786
It would be great if Thomas would tell us some more about the support of
removed vdevs (=3Dindirect vdevs).
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From owner-freebsd-fs@freebsd.org Tue Mar 16 21:53:18 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A35D56F698
for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 21:53:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3])
by mx1.freebsd.org (Postfix) with ESMTP id 4F0Rqf0nrDz3l3R
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 21:53:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: by mailman.nyi.freebsd.org (Postfix)
id 1B1C856F5AE; Tue, 16 Mar 2021 21:53:18 +0000 (UTC)
Delivered-To: fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19E4456F627
for <fs@mailman.nyi.freebsd.org>; Tue, 16 Mar 2021 21:53:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Rqf0865z3l3Q
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:53:18 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
[IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ECF812B13E
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:53:17 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12GLrHsJ071930
for <fs@FreeBSD.org>; Tue, 16 Mar 2021 21:53:17 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12GLrHoj071929
for fs@FreeBSD.org; Tue, 16 Mar 2021 21:53:17 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to
bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 254349] cannot set bootfs on pool after removing a vdev
Date: Tue, 16 Mar 2021 21:53:18 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: CURRENT
X-Bugzilla-Keywords:
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: mm@FreeBSD.org
X-Bugzilla-Status: New
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-254349-3630-dmrJ8RlEr5@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-254349-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 21:53:18 -0000
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254349
--- Comment #4 from Martin Matuska <mm@FreeBSD.org> ---
This is the most important commit I am referring to:
https://svnweb.freebsd.org/base?view=3Drevision&revision=3D350772
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From owner-freebsd-fs@freebsd.org Wed Mar 17 00:06:45 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1569E573131
for <freebsd-fs@mailman.nyi.freebsd.org>; Wed, 17 Mar 2021 00:06:45 +0000 (UTC)
(envelope-from rmacklem@uoguelph.ca)
Received: from CAN01-TO1-obe.outbound.protection.outlook.com
(mail-eopbgr670089.outbound.protection.outlook.com [40.107.67.89])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.protection.outlook.com",
Issuer "DigiCert Cloud Services CA-1" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Vnc0qRNz3t4t;
Wed, 17 Mar 2021 00:06:43 +0000 (UTC)
(envelope-from rmacklem@uoguelph.ca)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=mfyUJmofVacVpZyVnjICuLUaut3+vEsd8Qg+OK7y7a/vW4n97U0KrJBnIZXgsupVk7MAVGhshMIUJZcqCKagzTy/PlqLV8RU2VKU/nn3oRAAykFSLRec5BjP0pD86xxt1UCvIaK9cwXvy1rG2fO/Mg0kLFJ9u6RcdSrHO3e1GZaB0GzCok579isK2mlEcnCz0Gi9EGUQveO5NiTTfkbs88/V0QHAJvk7aHMTkIsogBhvcdCU6qgkE4TMUGRfRJvcpyt2aHw96zN+eIVzJBCdsgyFs11LqfXzFJWzk48K4fsLNsZ40VZCOQ5oRzO+kc9HY33fTp8pk3GTXFMMaDMwKQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=pHvjowwaioWXpHFZsKodIFjuLdc/7Wa3c3oCFpPgMhg=;
b=jS65r+jZ5ZUB40dTTaXv5K0mV79Ej087/NBA046qdzcn5fqQSwqDR/F+OabWn1hbKU6uqKTjzOysgaPodiO9JtSIsbLCVtOI0xk3k7iYfQ6VjQGmZzcwiL5ROUNTBfvtfqFYag6SH9H5Wbovato76nYn8z0jeEHcsV2GWJGY05lPtn92FoKxOGCtvEJVPS9XNUrJYHh4e39T1oFAWf6/XoU1aUKcn50m75CfhazcqMtjDFLIhS8MFJpBsNZDR7OnsFHhFf+RHnl35FRiX5fjz04xe6YjCQDlvi2rOhV+jXd/d/Fa48qiVevClG21S0M2ihTy0QKKHgEMgTpfafaQ7g==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca;
dkim=pass header.d=uoguelph.ca; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=pHvjowwaioWXpHFZsKodIFjuLdc/7Wa3c3oCFpPgMhg=;
b=Lakji1Z+1bFmjZOkqetVV7a0wje3V4yYaXNR0xk+6DSMXWo+H7rUexikCvxafv3d+SwMHsMIC9t/b0IiaTRHix6n7C28QPWDnqqX1RflBmDciGIXDsZ20Q9gFVg+r+ISnWt/vQP4ODEpcmDO1o+P0FKshf2P7vuLeWsQM+pL6b4EtPuuTa2zlRSOMxnYCzNzA1MElDnYLaoUlt8eVzTmzltEOtA/R8C+b6Nzgh7C21tkGgRQM0dE+Ujt1D82Uk+ihDmsYbedf1dieQXKHLK3UiTBXEwWHKzZh+j+w8LVf2HBeXlW5yFVu6rYOHTeG5o+h06iyqAtm/VkjMLdyK76/A==
Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
(2603:10b6:c00:19::29) by YQBPR0101MB1764.CANPRD01.PROD.OUTLOOK.COM
(2603:10b6:c00:e::20) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar
2021 00:06:42 +0000
Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021
00:06:42 +0000
From: Rick Macklem <rmacklem@uoguelph.ca>
To: Alan Somers <asomers@freebsd.org>, freebsd-fs <freebsd-fs@freebsd.org>
Subject: Re: Understanding the vfs.nfsd.request_space* sysctls
Thread-Topic: Understanding the vfs.nfsd.request_space* sysctls
Thread-Index: AQHXGom0SMOUlewkdkSbAUM6yshgqaqHSCtT
Date: Wed, 17 Mar 2021 00:06:42 +0000
Message-ID: <YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
References: <CAOtMX2geF7Tv7VYc9WtK23a+nDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com>
In-Reply-To: <CAOtMX2geF7Tv7VYc9WtK23a+nDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 0f2d188b-be5a-4991-4a75-08d8e8d88b71
x-ms-traffictypediagnostic: YQBPR0101MB1764:
x-microsoft-antispam-prvs: <YQBPR0101MB1764C1CBFFC65409EDBBCE61DD6A9@YQBPR0101MB1764.CANPRD01.PROD.OUTLOOK.COM>
x-ms-oob-tlc-oobclassifiers: OLM:5797;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: eLNYg7dSW95q2AkE5IjWGzxnkzfOK6PB+mNSjViMvybs5dbNUH/NrdGSEXSaBugGPp79PBH9cLki8R0DemuHxXAM84S5juhecJL8MZrvTsZsrx7i8WGL1cccUVzujpTSDUqQjsMKbTp5gZo4Mb6PX3vsDBmjCg1TF0lyTfgTlrgh1YnOC0ihCqMbSWULZ58PFiFFuNLtt0TG89LnEtfEkivqxCXN6hqsh6v8MVSvQux4olcKglmO4uJGTugKXQvJNRAHs9KuzMGIyBeRDTs0ZQrqbzXAy9Fofml4HAy2ponbqRhVHIH7/kPcgizK2WrBd4XFWDHZ8P9NFKrZQwg3Pwz1ATmQS+AULkjEwzKsujkDQboIiAshhakMzTbhNeW7VGxm3DQlOoNXrQfRIc7aa0tYHd97SZ8jtIn1PRoywE/jQNMfO+KLp2rNRWSC/daUS+wpY9bNQVZGFafvhU1Ct9ZK3ZOGcmojkfwprE32LdHLS/Gsv4DwpArCAx7SB9k6vg6PwKKiAbjT+Mgc2c1qNH2y3xXvB4DmOI0M6WpJL4XzeA65LFuku4aJFbCih2OY2YZSwd9Taw8yr2wSkWfRLcwCyIBaQ74y2jTeLFGYjTBFQ0nrPi3l8YaIlIAVUqga4y9d2VZHjO7Sb9+xQs3F7w==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE;
SFS:(39860400002)(376002)(136003)(366004)(346002)(396003)(966005)(71200400001)(66946007)(8676002)(9686003)(52536014)(6506007)(76116006)(7696005)(66476007)(478600001)(64756008)(5660300002)(91956017)(2906002)(316002)(55016002)(786003)(8936002)(66446008)(83380400001)(33656002)(450100002)(186003)(66556008)(86362001)(110136005);
DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?C44i7Z9ehjPF9pzMdKd/hYkkYMycz2gTvdvLVrGdUqn3YJMFWz/uAIgZe0?=
=?iso-8859-1?Q?piRuBuB6tjU3H7fh15Z4F1IITVTY55APGJWYMIAdIh+dpOe7lZW2KOTBeD?=
=?iso-8859-1?Q?bsqL9lxwNExv2FLpuD/axE4qKL1wSzuYDyqyI3up9LtngBRDxzYVP3f3Rs?=
=?iso-8859-1?Q?bfBbOkmaT2FTZbi1H/mSgxVS28YFYCouXX2oEsAQPXNwtFgNFjWvdzDrgw?=
=?iso-8859-1?Q?BLgLzsLTXfBOUIm4ihnrE4YFpaTHLv6CE0X4BslY6nx1LRB/TXjv76qnLM?=
=?iso-8859-1?Q?yzDposGEFB/QnNFFTFkG0zfcigV3nPtFHtzt5rYOO2ukySoIVibvu7CxBi?=
=?iso-8859-1?Q?J6EGTixLv9YZipddhOpS8kNGQt2ewrG8Z10cHOMZxzwpiW0W9ScECnreyD?=
=?iso-8859-1?Q?d2PSC6wWiUZ1ixzf3clR+2spl94WrHTLz0gTyhUkGI8J73RKpTP2xaQpsf?=
=?iso-8859-1?Q?WKGBYnU+zfVDSxGzHXnc6RhQFrGd0atA1b50tWb8zDOAZZZ4DqpZhdmMUf?=
=?iso-8859-1?Q?Lj6kH02T7eZG8+kILFu+J7XTRyVtxZ2D+mpA/gJsdRE2u/nltJIu16v/BO?=
=?iso-8859-1?Q?H8tc9kYpuhZAfxtRLfWl5G7Aesmw2p6WDtkbTFnA9didQ+i0frmg+VX7ha?=
=?iso-8859-1?Q?9pbA0/7Ie6v/hbgya8ibA3oJ2u0wFgq+x6/Un4ZcPcCbFb/1dKghzGwyK8?=
=?iso-8859-1?Q?y6HoAZI3M+Ylk+zmBnLrAsfuMNN7u8wsef7851yD7UUF7NE/GD1aqlSWRz?=
=?iso-8859-1?Q?8vPh/i0Z7oSy44s4c0BZWw4CGe+/RSntsfxx0EmJJxiNgQJ5F0bE7sbsM0?=
=?iso-8859-1?Q?fYTw+rJgCGUuUi/4DZ7wkeLL0lGY1BXCFzVkavZA2khTx1nVBq7rTYPjDA?=
=?iso-8859-1?Q?/Pm3QvqT/3YUGT5iSiZkLLBib6JTKJSIv9zRDhJUsxvGblsmUGvGJqajnk?=
=?iso-8859-1?Q?YaoBM2Tpf7/N2yi6fSwM2PJjuHeC/RjJWT5JxZEwSHlGlxvBGpFZ9Kouch?=
=?iso-8859-1?Q?rDYmGNgg5t3ILLquA0nSgS/zHK8gAu9PBD0D5qt9du+jlIFlwnTG/V/etY?=
=?iso-8859-1?Q?aYVEkKbYTZfKRSkcrQ56Woj8l2dSNqyDx1l4W88DFJUOvsN+zR4j4bT6Sw?=
=?iso-8859-1?Q?dZ30ZZdhdOnI+rGumDzaRoa+0iFqVLvdNOTHF+WMtNeGMzRKlgbOwe4rGQ?=
=?iso-8859-1?Q?whGhATBfK+fHnmyPpDIYfogKRIfjHYTgpC/myXGxvr/sVfbKnfu4uMW6Rf?=
=?iso-8859-1?Q?ghVSAkDmsnAR+V1HF7DMdoVk1umoxQjveXYaTQjtwGKhZUMjKc6uvF/0QS?=
=?iso-8859-1?Q?T1ofprLSjqqnPJCW1+gHW7Og/weRTV0I+PNdkQET2mO3YLPhIIUqBvhwxP?=
=?iso-8859-1?Q?SazMCe2NS3+lPs6l2xPSmIyo943AM/3v7KDpg7ygW17j1cJboQAupxmL1f?=
=?iso-8859-1?Q?nLScc6fTHR51fV8a?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: uoguelph.ca
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-Network-Message-Id: 0f2d188b-be5a-4991-4a75-08d8e8d88b71
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 00:06:42.1943 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: QF3g69QNjNpkb74iVwKpOW0wIbuBJKRXpcfXeBZ7BcHHaYe7AIpPx4xSjlPjo5edtAeo7k5viWYqmQ7HrPOBZg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1764
X-Rspamd-Queue-Id: 4F0Vnc0qRNz3t4t
X-Spamd-Bar: ----
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=Lakji1Z+;
arc=pass (microsoft.com:s=arcselector9901:i=1);
dmarc=pass (policy=none) header.from=uoguelph.ca;
spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates
40.107.67.89 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca
X-Spamd-Result: default: False [-4.03 / 15.00];
NEURAL_HAM_MEDIUM(-1.00)[-1.000];
RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.89:from];
R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1];
FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[];
TO_MATCH_ENVRCPT_ALL(0.00)[];
R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16];
MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.97)[0.968];
NEURAL_HAM_LONG(-1.00)[-1.000];
DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim];
RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[];
DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2];
RCVD_IN_DNSWL_NONE(0.00)[40.107.67.89:from];
DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none];
SPAMHAUS_ZRD(0.00)[40.107.67.89:from:127.0.2.255];
FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+];
RCVD_TLS_LAST(0.00)[];
ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US];
ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
MAILMAN_DEST(0.00)[freebsd-fs];
RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.89:from]
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Mar 2021 00:06:45 -0000
Alan Somers wrote:=0A=
>Could somebody help me understand the meaning of the various=0A=
>vfs.nfsd.request_space* sysctls, and what implication they have for tuning=
=0A=
>NFS?=0A=
>=0A=
>I have several NFS servers. Most are performing well, but one with a=0A=
>metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAtt=
r=0A=
>and Lookup ops) is not. Total throughput is a not-unreasonable several=0A=
>hundred MB/s, but some clients are limited to very slow speeds, sometimes=
=0A=
>writing at < 10 MB/s.=0A=
>=0A=
>The request_space sysctls show some pretty stark divergence between the=0A=
>well-performing and poor-performing servers:=0A=
>=0A=
>sysctl well-performing poor-performing=0A=
>request_space_used < 6 M varies, but=0A=
>currently 40 M=0A=
>request_space_used_highest < 39 M 24 G=0A=
>request_space_throttle_count 0 35=0A=
>=0A=
>So what does request_space_used measure, anyway?=0A=
It's the number of bytes of request message(s) received,=0A=
but not yet being processed by nfsd threads.=0A=
(They're actually in sys/rpc/svc.c, although named for the=0A=
nfs server.)=0A=
=0A=
Anytime used > high, it throttles, which means it leaves=0A=
the RPC messages on the socket receive queue.=0A=
This indicates the server is not keeping up with requests.=0A=
(ie Overloaded or ???)=0A=
=0A=
> And how can I either=0A=
>increase the available space or decrease the stuff that's using it?=0A=
Increasing vfs.nfsd.request_space_high avoids the throttling,=0A=
but it is hard to say that is a good idea, since there won't be=0A=
backpressure applied to the clients via TCP windows, etc.=0A=
=0A=
Fixing the server so that it isn't overloaded would be better,=0A=
I think?=0A=
--> I'm the last guy to take ZFS advice from, but I think there=0A=
is a tunable w.r.t. how much arc is used for metadata.=0A=
Getattrs will need cached (metadata) to reply quickly.=0A=
Lookups also depend on cached attributes for good=0A=
perf. as well.=0A=
--> Make sure you have lots of nfsd threads. They are=0A=
just kernel threads, so they don't use a lot of resources=0A=
and having too many is much better than not enough.=0A=
-->I can't remember what the upper limit is these days,=0A=
but I'd set it to that for a busy nfs server.=0A=
For NFSv4.1, each client can send a maximum of=0A=
session_slot_table_size concurrent RPCs. FreeBSD=0A=
uses a single 64slot table for each mount. I think=0A=
Linux uses a 32slot table, but supports trunking.=0A=
I don't know if the Linux client uses a separate=0A=
session table for each trunk or not?=0A=
--> Something like #clients * 32 (or 64) nfsd=0A=
threads running on the server.=0A=
=0A=
rick=0A=
=0A=
-Alan=0A=
_______________________________________________=0A=
freebsd-fs@freebsd.org mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A=
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A=
=0A=
From owner-freebsd-fs@freebsd.org Wed Mar 17 01:32:29 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1D55A5757C3
for <freebsd-fs@mailman.nyi.freebsd.org>; Wed, 17 Mar 2021 01:32:29 +0000 (UTC)
(envelope-from asomers@gmail.com)
Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com
[209.85.210.51])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0XhY0Gtvz4Sx6
for <freebsd-fs@freebsd.org>; Wed, 17 Mar 2021 01:32:28 +0000 (UTC)
(envelope-from asomers@gmail.com)
Received: by mail-ot1-f51.google.com with SMTP id
h6-20020a0568300346b02901b71a850ab4so290946ote.6
for <freebsd-fs@freebsd.org>; Tue, 16 Mar 2021 18:32:28 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc;
bh=b6UdMGq9B5EqhM2b14RACWAp4KJXWd+Q+x1zS3ZnXBo=;
b=Bod4yDcWc36UnbpunlCkPtBFpX47x0Mz4wv8zBoVTh3c5l7oW9XpM6iJxNb9QsCF74
/8AzT7Ktfox47IXQUTnQHx99GuFGOhe8qhc7k7zj4oiipxevOus2QvnTkVyYlnRZ83i4
7ssAxfNq96tB4VDWriqbprSdREfaAiKepHe679+bt/nX3ZnsTTAyykCsH//1saIZY1dC
2BP0XEGEihFoRxW0UmT+lsZWqz3KOZkFkXu6YNub/j2lhFfFsCHKb2db1+CCYZORghK5
JQGspTklR3F0qcR+C+8u3gatjwidLljG/3DhtJovAyISNmloV8MBLCJobDqiaai6p2B6
N1Kg==
X-Gm-Message-State: AOAM532GLT1v/f85Fm2imLfgKxRXD8BgQv+hwsW+HWyfIyQKLGOaCgAF
JpKPnvPsomQa6ZysbAFUNM8lj6XYR9NtGACOfLs=
X-Google-Smtp-Source: ABdhPJxAANVPfNKWdM8oeRBwa/M4Z/9awf70F1qcD08RF5v+rla+o/jrUZeSAo5fBoycqkjYGNL/AXIJ24DQEHGrM1k=
X-Received: by 2002:a05:6830:148c:: with SMTP id
s12mr1334689otq.251.1615944747110;
Tue, 16 Mar 2021 18:32:27 -0700 (PDT)
MIME-Version: 1.0
References: <CAOtMX2geF7Tv7VYc9WtK23a+nDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com>
<YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
In-Reply-To: <YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
From: Alan Somers <asomers@freebsd.org>
Date: Tue, 16 Mar 2021 19:32:16 -0600
Message-ID: <CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ+Y3PG82pN0s2eVusRooRF_g@mail.gmail.com>
Subject: Re: Understanding the vfs.nfsd.request_space* sysctls
To: Rick Macklem <rmacklem@uoguelph.ca>
Cc: freebsd-fs <freebsd-fs@freebsd.org>
X-Rspamd-Queue-Id: 4F0XhY0Gtvz4Sx6
X-Spamd-Bar: ----
Authentication-Results: mx1.freebsd.org;
none
X-Spamd-Result: default: False [-4.00 / 15.00];
REPLY(-4.00)[]
Content-Type: text/plain; charset="UTF-8"
X-Content-Filtered-By: Mailman/MimeDel 2.1.34
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Mar 2021 01:32:29 -0000
On Tue, Mar 16, 2021 at 6:06 PM Rick Macklem <rmacklem@uoguelph.ca> wrote:
> Alan Somers wrote:
> >Could somebody help me understand the meaning of the various
> >vfs.nfsd.request_space* sysctls, and what implication they have for tuning
> >NFS?
> >
> >I have several NFS servers. Most are performing well, but one with a
> >metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence,
> GetAttr
> >and Lookup ops) is not. Total throughput is a not-unreasonable several
> >hundred MB/s, but some clients are limited to very slow speeds, sometimes
> >writing at < 10 MB/s.
> >
> >The request_space sysctls show some pretty stark divergence between the
> >well-performing and poor-performing servers:
> >
> >sysctl well-performing poor-performing
> >request_space_used < 6 M varies, but
> >currently 40 M
> >request_space_used_highest < 39 M 24 G
> >request_space_throttle_count 0 35
> >
> >So what does request_space_used measure, anyway?
> It's the number of bytes of request message(s) received,
> but not yet being processed by nfsd threads.
> (They're actually in sys/rpc/svc.c, although named for the
> nfs server.)
>
> Anytime used > high, it throttles, which means it leaves
> the RPC messages on the socket receive queue.
> This indicates the server is not keeping up with requests.
> (ie Overloaded or ???)
>
That all makes sense. So having a high request_space_used basically just
means that my storage is too slow.
>
> > And how can I either
> >increase the available space or decrease the stuff that's using it?
> Increasing vfs.nfsd.request_space_high avoids the throttling,
> but it is hard to say that is a good idea, since there won't be
> backpressure applied to the clients via TCP windows, etc.
>
> Fixing the server so that it isn't overloaded would be better,
> I think?
> --> I'm the last guy to take ZFS advice from, but I think there
> is a tunable w.r.t. how much arc is used for metadata.
> Getattrs will need cached (metadata) to reply quickly.
> Lookups also depend on cached attributes for good
> perf. as well.
>
I've added an L2ARC since I sent that original email. We'll see how well
it works. I expect it to take 48 hours before results are apparent.
> --> Make sure you have lots of nfsd threads. They are
> just kernel threads, so they don't use a lot of resources
> and having too many is much better than not enough.
> -->I can't remember what the upper limit is these days,
> but I'd set it to that for a busy nfs server.
> For NFSv4.1, each client can send a maximum of
> session_slot_table_size concurrent RPCs. FreeBSD
> uses a single 64slot table for each mount. I think
> Linux uses a 32slot table, but supports trunking.
> I don't know if the Linux client uses a separate
> session table for each trunk or not?
> --> Something like #clients * 32 (or 64) nfsd
> threads running on the server.
>
Experimentally, I determined that 768 threads is sufficient. But your
formula would suggest > 2500, which is a lot of threads! I'll keep it in
mind if I ever reach 768, though.
>
> rick
>
> -Alan
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>
>
From owner-freebsd-fs@freebsd.org Wed Mar 17 02:31:39 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 20FBC576C46
for <freebsd-fs@mailman.nyi.freebsd.org>; Wed, 17 Mar 2021 02:31:39 +0000 (UTC)
(envelope-from rmacklem@uoguelph.ca)
Received: from CAN01-TO1-obe.outbound.protection.outlook.com
(mail-eopbgr670051.outbound.protection.outlook.com [40.107.67.51])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.protection.outlook.com",
Issuer "DigiCert Cloud Services CA-1" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Z0j5Cbwz4WGX;
Wed, 17 Mar 2021 02:31:32 +0000 (UTC)
(envelope-from rmacklem@uoguelph.ca)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=nYkIyxjWmIiWtM9EhJJneMkYUXukCA+PcZcXBrmchGMnp2tU/AKquVcr2ebTv54B0WBTZgQGgqYrZl1IxdPZaBFmhObl10ra8h3KFsBDnd57sHafCZjGNWvm3vttYOpOEFRvgiQ6sXtJLYZzxg8hcl8QlTPYCKWZCTmjhD8S/q2V+ZtUIbChl18A/4nEIiutVTDEjM6oVfIfRmhmxsyEPsl5sUbacH/dEaQQD01DLtLbhBsA31S0w1vSAfpJShg8B/TfQM0xRDjV7o14VwJzV9rNXfHkw5H7XEDPRjnbUb4YIIbyfzTAhAb/vcGnHnyTdkI5KolTQO5dxgkPBffPMQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=wLCP9g/PEdXdiOyqQqrfl32Sgu0Y8tOatQjHXXyPjZo=;
b=g4MWHtFgukECLOUQ9hbel/TUKPX0imdyAokDIn1CxebzK6ECbZAVKJ0ZH1BlWDOW6JFjNRTsyl/JqPXeABI7kmmGYDn/G0w2M6KNDS0OBNrpQZtdbu9B0NLdMPZqrKC4XB9a9uG0zQ2HGIzbCGgTcyxVH1OErZ2F4YbXjwmWyqsmJ4lRniPeSDRyKoBL7dxPES7pLK4nKZdybb1KBdJ3IcT/3cc5tq7F3J3MN8WN1+QoBzHth8Y8UgMtHeIxubqw1NMquol0akCxm1z5l+a3wEFDjUFUH5yBKENp9Tw6LZw5Z06J2zntj+EfBnk7cHcG2941+e/ir48Y6fkTJo665w==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca;
dkim=pass header.d=uoguelph.ca; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=wLCP9g/PEdXdiOyqQqrfl32Sgu0Y8tOatQjHXXyPjZo=;
b=TZcAHbmwNB1Xpk31zXJG2dY8SiEWSrfcxlSsQlWEociYeCywP350vRCE8ambjQVxnmns9RjKIZN2e1Ix0P4NsAAYj/w0nYx/PfEbWJf0EVJR+JuVOZe8Dke81T71Y8f5Lj0cJXf5Z6YlmzzwAMnGE4A7YeyeSjIwshVFsl31X71nR76aTociT72HhAIeZT35DjXszSsNeUprsghz+xitph5xaYGe54m6IIEZI+ItxveGTe8WaUuwlMrqNoqJRHqF1WvFuxkoMitwZtfFQyk+qHk1b09EVtKVuulVdJFEwJJfqogft0NNGfDagxvmQGMhA+kXNNXrhExY/t9gapgdig==
Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
(2603:10b6:c00:19::29) by YQXPR01MB3430.CANPRD01.PROD.OUTLOOK.COM
(2603:10b6:c00:4d::25) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar
2021 02:31:30 +0000
Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021
02:31:30 +0000
From: Rick Macklem <rmacklem@uoguelph.ca>
To: Alan Somers <asomers@freebsd.org>
CC: freebsd-fs <freebsd-fs@freebsd.org>
Subject: Re: Understanding the vfs.nfsd.request_space* sysctls
Thread-Topic: Understanding the vfs.nfsd.request_space* sysctls
Thread-Index: AQHXGom0SMOUlewkdkSbAUM6yshgqaqHSCtTgAAdewCAAAz0qQ==
Date: Wed, 17 Mar 2021 02:31:30 +0000
Message-ID: <YQXPR0101MB096871CFED5ADA41FC22713FDD6A9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
References: <CAOtMX2geF7Tv7VYc9WtK23a+nDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com>
<YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>,
<CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ+Y3PG82pN0s2eVusRooRF_g@mail.gmail.com>
In-Reply-To: <CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ+Y3PG82pN0s2eVusRooRF_g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: b53ed26f-ae04-4060-727c-08d8e8ecc627
x-ms-traffictypediagnostic: YQXPR01MB3430:
x-microsoft-antispam-prvs: <YQXPR01MB3430F6266616D513465D658FDD6A9@YQXPR01MB3430.CANPRD01.PROD.OUTLOOK.COM>
x-ms-oob-tlc-oobclassifiers: OLM:8882;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: J/0wBns632WIQavUurYRQBArDZ2H0tPeu0lS5q6yoj7xKoVGvFGDf6i1eCcLMCUyIJ6ChQGodiHJ3zTOkFlaKrLld026HGHXo2/C1Nvz6dndgKkeczEE7KRdHZmho3kIhD2+xT0qWC5xvcPuqfydCRjdSo+aU+HmnNhg0iTTrFwpRhafIY3ovhK/3DIRNFXRaRlcy9ybJXTCZVIuC5Mqy9BfVlPrucR0QABQPI3Wf1d8QGfx9c+FgptN5MlhvkXbE09j8Wyzb4DUvX6rYrGksMpFbn3/RLP+018lE/8TzS7uYldZj+0UWRp3Y8qHs04GU8a7ltFDJ/qMbbjfL8WqgNzdvqP0T13eIHAs4mViBW2eS1gwE4sKDPbpOvkvdkeTYQpMlkzW1k/RoaEvNRCfhqCVdFAcHbvD4JReUMk7AqtN7UOHUM+Rarvgd0VOKBzlstkWDxUpPAdP5X+s9mhWu6PVPvp+MhsrIs71joxAPcqbdTmxMvVTe7/wI4Kztblm1OZTDUeAjx9fJGD4cd09hoDx6Si0T6dvn1ba9cIrGu1/VVuXLf1+/dWkr7jgXOL69SxENsum91X/JI1rRME34sN8RH7B214WYn92d0Pj+j3yh9mTy7fA6Sca8cGVZVLdtxKH2yZCrX4oc6uWP57F2A==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE;
SFS:(376002)(396003)(136003)(346002)(39860400002)(366004)(66446008)(76116006)(66946007)(316002)(9686003)(8936002)(478600001)(64756008)(66476007)(6506007)(8676002)(6916009)(7696005)(450100002)(55016002)(4326008)(86362001)(5660300002)(66556008)(91956017)(52536014)(2906002)(83380400001)(33656002)(786003)(186003)(966005)(71200400001);
DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?tQKcv5fggIBPb4QCrT+K4tWsMOeK+etSa/C1d7r6S8fkiZc0EUiHobjwyo?=
=?iso-8859-1?Q?VWcfAH7JGKcCESITsDfyHDSTkRusiwD17NQCczPsMXzIsoCbCf9NGnXFGZ?=
=?iso-8859-1?Q?K3Gx9WXbk3UBLIzUZR/tb91symuHsRQBKViUmvb8iZHhvb/L7XfkkFHOVK?=
=?iso-8859-1?Q?d+BX7yq3ZrvnwZ8HYPg4fKNTh1hgG6amNm0oOviBOzFxfXoPLG1kJD/9mo?=
=?iso-8859-1?Q?DGVNjwhE4JG9uWIOyMc0SLS8Lfl3gcAPExh4R0c7K/2Qi9eDgoxT2Th5Dq?=
=?iso-8859-1?Q?HCjvqdp+NhOV23n8N3KfsCoklWIi6cYMGvCkTD/y5tgmlE5PdIprhN29rH?=
=?iso-8859-1?Q?M1F/ek7yhYqe0NnllNDwdbsA/hRgA/Bn6Z3R27PLtBcwVbHF99YCa59zDg?=
=?iso-8859-1?Q?tpx4GgvMk+8zZCXnzWRHqh+Lqga1skKQtWQc9InjCweYOO3ESB1ayvLlCj?=
=?iso-8859-1?Q?fvJzfv5AlxqdifHw2mPRt9Vzh1nRkyQ+qid9Lak1hqlWG4/tyS+RmU9T53?=
=?iso-8859-1?Q?7r04A1Cz3s8aV7Y5b5ldRFckHb+tkMsikOeqoHYvsh89+HherWoiN3o6Ii?=
=?iso-8859-1?Q?2QSYaalOqVx8PzS+Ze327WtcQ9sjH3K4mwl3P61qWDjbOMxaprupRzBgUa?=
=?iso-8859-1?Q?ed5At+6xBuGMkndWGFnSAUVOeBYIzG33796CmVb4ZeUJfwh57jJe5bVnoZ?=
=?iso-8859-1?Q?JnZEVqE060oEFVKXg9rlP0Dhs6khxiDNVTBnGWXjVQ6xTRfbgGKZF368Ke?=
=?iso-8859-1?Q?aw5bgApqlxPD+31EuZ0oeTDz1uQNpbY1796ewpV/uIyQt60QJcxW1SCGZ7?=
=?iso-8859-1?Q?zwLwdsHyCsy6fOHs/lSTUOiDH2cUH2cCjhBMS5tSRG6WjoU7B25blXBEG8?=
=?iso-8859-1?Q?SNqpCtRd3knDLWmpbopbVxxljhH/VzR9Aqn7BhssYrVJF3iHnnC0tbCOlL?=
=?iso-8859-1?Q?gswx4dJaDllmb0OvM3TPNFOwzeyLflS+8NMe0OyMbEH6ZL4+EYdnQST2ZJ?=
=?iso-8859-1?Q?T1QtbgW9pna49mfxXHt1yEfAn74muKWNjyhvp2uaAYvzkDbirEDE37dMZU?=
=?iso-8859-1?Q?vg4mIb1TjiV6grU+NiUIuaMwO4tfvNDIiZE0ljj5+4uE+QoxciAdeTa5ij?=
=?iso-8859-1?Q?a1zA53RS6io4cRwuX339c+F8i9/SK9hWJ7fvP5P3a/UK7Xh9ABhV1+7VQf?=
=?iso-8859-1?Q?ThNFiJspCGsR50v8B8BWj5YQCi/qUxUDNcBsuvTDFiQbR3KayjmgrlpA1B?=
=?iso-8859-1?Q?kyIlu+5epv0Gzyfu09/3qe5qBk+9yaJlpHBFvT/TPa3jqnQP6VFFO7R+aY?=
=?iso-8859-1?Q?s6Bavi0iCCl21IIXgYBgrgIs4mdBvIsygRHWYBbCVFnE2j0mL4aSSLt3ao?=
=?iso-8859-1?Q?nHZE39s+Is8a/WwkIQ9Xa/SErtzRSQ2ClOdIfsLjroESZV4bi9rSU=3D?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: uoguelph.ca
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-Network-Message-Id: b53ed26f-ae04-4060-727c-08d8e8ecc627
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 02:31:30.6365 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: kwnvC7ROnPdKxrwvQ9TYwduUBbIbPTc3GSspDKXZ63nLfrFMP/+4o/5drui8J+OCOn32qPKcWg0dS4a5SpwXVA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3430
X-Rspamd-Queue-Id: 4F0Z0j5Cbwz4WGX
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=TZcAHbmw;
arc=pass (microsoft.com:s=arcselector9901:i=1);
dmarc=pass (policy=none) header.from=uoguelph.ca;
spf=temperror (mx1.freebsd.org: error in processing during lookup of
rmacklem@uoguelph.ca: DNS error) smtp.mailfrom=rmacklem@uoguelph.ca
X-Spamd-Result: default: False [-5.80 / 15.00];
NEURAL_HAM_MEDIUM(-1.00)[-1.000];
RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.51:from];
R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1];
FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[];
R_SPF_DNSFAIL(0.00)[temporary DNS error];
TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000];
MIME_GOOD(-0.10)[text/plain];
SPAMHAUS_ZRD(0.00)[40.107.67.51:from:127.0.2.255];
RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[];
DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2];
RCVD_IN_DNSWL_NONE(0.00)[40.107.67.51:from];
NEURAL_HAM_SHORT(-1.00)[-1.000];
DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none];
DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim];
FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+];
RCVD_TLS_LAST(0.00)[];
ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US];
ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
MAILMAN_DEST(0.00)[freebsd-fs];
RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.51:from]
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Mar 2021 02:31:39 -0000
Alan Somers wrote:=0A=
[stuff snipped]=0A=
>That all makes sense. So having a high request_space_used basically just =
means that >my storage is too slow.=0A=
Since it appears that you have plenty of nfsd threads, that would imply=0A=
that the bottleneck is probably the file system.=0A=
(You might want to check that there is cpu available. ie. Some amount of=0A=
idle cpu, just in case?)=0A=
[more stuff snipped]=0A=
>I've added an L2ARC since I sent that original email. We'll see how well =
it works. I >expect it to take 48 hours before results are apparent.=0A=
Others have tweaked something so that more of the arc is used for metadata.=
=0A=
[more stuff snipped]=0A=
=0A=
>Experimentally, I determined that 768 threads is sufficient. But your for=
mula would >suggest > 2500, which is a lot of threads! I'll keep it in min=
d if I ever reach 768, >though.=0A=
768 is probably plenty. My number would be an extreme upper bound.=0A=
I did it because the default of 8 per cpu is very low (assumes a machine mo=
stly=0A=
doing other things).=0A=
=0A=
Good luck with it, rick=0A=
=0A=
=0A=
rick=0A=
=0A=
-Alan=0A=
_______________________________________________=0A=
freebsd-fs@freebsd.org<mailto:freebsd-fs@freebsd.org> mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A=
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org<mailto=
:freebsd-fs-unsubscribe@freebsd.org>"=0A=
=0A=
From owner-freebsd-fs@freebsd.org Wed Mar 17 14:47:06 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6DFC756B611
for <freebsd-fs@mailman.nyi.freebsd.org>; Wed, 17 Mar 2021 14:47:06 +0000 (UTC)
(envelope-from tech-lists@zyxst.net)
Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com
[64.147.123.20])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 4F0tKP5Ymzz4SnV
for <freebsd-fs@freebsd.org>; Wed, 17 Mar 2021 14:47:05 +0000 (UTC)
(envelope-from tech-lists@zyxst.net)
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
by mailout.west.internal (Postfix) with ESMTP id E5C382458
for <freebsd-fs@freebsd.org>; Wed, 17 Mar 2021 10:47:01 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
by compute1.internal (MEProxy); Wed, 17 Mar 2021 10:47:02 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h=
date:from:to:subject:message-id:mime-version:content-type; s=
fm2; bh=bYuayMDbgYPH+pDH5rkq4QZkpAlVx7f4mpDuWsrt/xM=; b=Uw0LwOaK
wJLNUnPYrw/i5WnSgZOv5eImUUgjUm4TMTIHlavnvU95GqXVJRivvNUtuopTsD1q
86tYLxm+g1acBnoLOyEXS3+48HXIY4V5AraZ/3/0/b+tBKMhx49aHUQU4MH5eA1F
5047ny0J8nK8HO6MwpElmdvPGiifJ+8O54J9+xncdyXiC35Qv0wklnEZ40UYI0SI
tzH3Rls/08EuClZrh9sZI7UTlvpWSKZN2oiaFdclJabj5DPe4flCa11WsL9UnLTD
cnrBmUCWsVZLVfTvkkgm+Mm7MnSkzZ0Tln3IUSI1pvBxkwvDq5MR8/NCoqTHQX5E
S0rJoOi+EWJyrA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
messagingengine.com; h=content-type:date:from:message-id
:mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender
:x-me-sender:x-sasl-enc; s=fm2; bh=bYuayMDbgYPH+pDH5rkq4QZkpAlVx
7f4mpDuWsrt/xM=; b=Md23+iUzLg5S/nuchhbp+8zyeAyVF7oGfM6zYp+/KXcXC
kaFROZ6rQ6CoU5wi6gR62qNt70mg77Gzm4RueTd16FhRzMjBYy12H4V3p2qNkpO2
S9mppatq9AkxPpQlWItr/QNBa7rmaCzmPqYTUH4lFEQimmnxFdTeTdp2iyQ2eUHY
ixWaEnyPXY+MK80U1Lt5lqudIlMymBthlKVu4NoRiZB7+k3/nfeqI6zEzORLDghY
AGRQE7N+svuIjmj1dzIUkhXf1p54nXezataVlthUkD+4QYKnxb6usUDNBWH29KFY
LbKba832QdJ5Jp+YIdAgIA03MCH14NTLn2PH2unRw==
X-ME-Sender: <xms:ZRZSYEU28gkOjBNTSl48hLpne7C3ZkMcl8aqX-66oQESzRHF6cjjUg>
<xme:ZRZSYPnZrRmPgef8YJbr6gzbxVA6O80vLBGYsDcX6sWN4fkIcvDy7G9z29Pz614zG
ghEbxZcT1YTd4zhzA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudefgedgieelucetufdoteggodetrfdotf
fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkgggtugesghdtreertd
dtvdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseiihiig
shhtrdhnvghtqeenucggtffrrghtthgvrhhnpeevgffhffdtfeekleelhedtjeelvdfhvd
egieejveffgfduvdfhteegjeeujeeuieenucfkphepkedvrdejtddrledurddutddtnecu
vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthgvtghhqd
hlihhsthhsseiihiigshhtrdhnvght
X-ME-Proxy: <xmx:ZRZSYNaSUW76RtSyaVnuRMuVCo6jSzh-xggBzLNFw2lQXnapwWBNug>
<xmx:ZRZSYCPXDw27z-gjHv-N1M-ST3FdGxc33j0vcnA3trpnYi9w2qpECw>
<xmx:ZRZSYKZBjSZQwKyySVbkPZbHZMYdlmPXI_rT8yZ1DkNTluZPZtzyDg>
<xmx:ZRZSYB0w8cYOub3qxHfX2586cjbmxtDmj-OOIg0c9pUD4nr5IjRtEw>
Received: from ceres.zyxst.net (ceres.zyxst.net [82.70.91.100])
by mail.messagingengine.com (Postfix) with ESMTPA id 157C7240054
for <freebsd-fs@freebsd.org>; Wed, 17 Mar 2021 10:47:01 -0400 (EDT)
Date: Wed, 17 Mar 2021 14:46:59 +0000
From: tech-lists <tech-lists@zyxst.net>
To: freebsd-fs@freebsd.org
Subject: reference for zfs on freebsd stable/12 and stable/13 and main/14
Message-ID: <YFIWY0Gi6g5QZpFX@ceres.zyxst.net>
Mail-Followup-To: freebsd-fs@freebsd.org
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
protocol="application/pgp-signature"; boundary="mgc8tl8SHwXPqumN"
Content-Disposition: inline
X-Rspamd-Queue-Id: 4F0tKP5Ymzz4SnV
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=zyxst.net header.s=fm2 header.b=Uw0LwOaK;
dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Md23+iUz;
dmarc=none;
spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates
64.147.123.20 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net
X-Spamd-Result: default: False [-5.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[];
RWL_MAILSPIKE_VERYGOOD(0.00)[64.147.123.20:from];
R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2];
ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[];
TO_MATCH_ENVRCPT_ALL(0.00)[];
R_SPF_ALLOW(-0.20)[+ip4:64.147.123.20];
MIME_GOOD(-0.20)[multipart/signed,text/plain];
TO_DN_NONE(0.00)[];
PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000];
RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net];
DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+];
NEURAL_HAM_SHORT(-1.00)[-1.000];
NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SIGNED_PGP(-2.00)[];
FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~];
RCVD_TLS_LAST(0.00)[];
ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US];
MAILMAN_DEST(0.00)[freebsd-fs];
RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.20:from]
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Mar 2021 14:47:06 -0000
--mgc8tl8SHwXPqumN
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hi,
I know from reading around the lists that zfs is changing to OpenZFS.
What I'd like to know is, is there *recent* literature (website or pdf)=20
available for reference for zfs backup/incremental backup with zfs=20
send to a remote system in the following situations:
1. 12.2-p4 system sending to recent stable/12 system
2. 12.2-p4 ststem sending to recent stable/13 system
I realise some of this is in the handbook but the handbook is
out-of-date in many areas and references versions (of freebsd) that are=20
no longer supported.=20
thanks,
--=20
J.
--mgc8tl8SHwXPqumN
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAmBSFloACgkQs8o7QhFz
NAXkHg//fUTcKvngfyYdCSUk56VADt1rQ+rRRjLQBAjoP8+sERP5R7jr0GyXaWg7
5Z7J7c0+creYVAVJPxXh9ODlW/fW3ELhTvtBuYO/DK/7wtZh+hzSoi/kq5/g3wnH
Acm9vTU+8FxiV+JMbsuoXUejxoG5HE4hipFVnd+setf5n+Xhb8CEdC3j7q8Q63R1
SlRuW45P4xnh1QEKZStSsddQB6/VuI2/ZlRqKVjr3YmvBGbvErW/lF2cHXp7fugB
LVCj0DSftkYTMxwHkSRTCr9yOcqN6vbRKeF2Sr+dYq5UXBMVd8lWcZWDp7AUCFhO
1WNWRYDmkSCfrrrDOv3qMljrW9OBIemiGD97XCAK6subbj38F7mRpa7DzQ7IFmmv
UPh0qFT9ivwjlM0xjgqFukf+zRlnX5eIzO5s+rGCnBWZymu31GH5nDPXbWvIg610
NKI+EnIT/VU9gZXs15dwDgWKb/14+EA1TSAwo4/1PUxOBORpUzL8JqJa+8n07NKj
6/aF0prxR64sNz5gD6riDm0YtKFOjF9F6TG/xSnftezTs3tGrACMgs/sf2D0R+qu
3bYDpCSX2hj1pZrHulZ4UJQ2CtkbYyhDBax3tv6bci6Ruu0AwTJbs2MnE7Iegp11
zrEWfLL20OkarTPydvC0tneXUZxQlWsS7KPHHV5mYfqHWcnhXi0=
=MptM
-----END PGP SIGNATURE-----
--mgc8tl8SHwXPqumN--
From owner-freebsd-fs@freebsd.org Wed Mar 17 21:48:44 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A798579B6F
for <freebsd-fs@mailman.nyi.freebsd.org>; Wed, 17 Mar 2021 21:48:44 +0000 (UTC)
(envelope-from rmacklem@uoguelph.ca)
Received: from CAN01-TO1-obe.outbound.protection.outlook.com
(mail-eopbgr670082.outbound.protection.outlook.com [40.107.67.82])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.protection.outlook.com",
Issuer "DigiCert Cloud Services CA-1" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4F13gv0gqQz3Mrt;
Wed, 17 Mar 2021 21:48:42 +0000 (UTC)
(envelope-from rmacklem@uoguelph.ca)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=XSlJe0vq+wm4TA80eFJoz7zI89mYBhshe9zPKQKz9MOQbnC8ZJehM/+nr/+XXSvWNBq0PGD8WR6u2hSMHqGIe5JnLO2HG/dwvCnYTOc4hJXVXb9pq9vUAIsXFvOwuAOzzICUQ8fyntUU6H9hg9jTy0WUmPkLj8VOGNIlsqQ2x3uKiLE6H7JLCP/esSyAzcMo5CuM9OD/kftO98lk0ANjJMNP0DEnnz9ChnZx/V6NNl5D2vdcFjaVT5XZB16QmIsqTqgGMvO2xVOYCzVxtdccIWk52PZKi2OVx5JyPuZpVh0aDeELvpL+qaKBjsqgOtRg2LmAl2zWkwO4VxkGFGgffA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=DPEvm0/qKlhDJlzhuiDcziP+SZAPTJvTqWhnuAVz9SI=;
b=hyt39gLoSh4N+JSwCSzEAPQ4CXChBvk4o6XQWjsfglFC7aTYJLxTXPtrU+gRgwKJItDn4pXzU4tdH1atP1YtX1J0wMD4BiiPKYIKLUSNQIBU05cHAwFjT1bF/6we9KSXR9I/3m4Q9hVZdUrKfwCpj6zQO7MixMX/tVPokF1n4rC6UnFXqzE1TWPtTYzfy2UxC6oUqpQoHn4/fCdim8uK27h4jmCzoTKVYs756CM4iSs8YIH5ishNvsFkZ3KRrbfFvksit8k9LsCIVNITctL/8co+pfKTE7vAL/IKwCiWlGpz5r4/9cf01RomfuK2T14hl6ti1LwoHwPIb4Y+v1//XA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca;
dkim=pass header.d=uoguelph.ca; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=DPEvm0/qKlhDJlzhuiDcziP+SZAPTJvTqWhnuAVz9SI=;
b=ZpfgDTZQwiEtBEjmpmaDiZlWB9MHKYDGa6D5YxTdFP3ctbgrvA42tgnGhrEqq9T93t4t8g2UXzJk6PMflJReM27xtpmEe8mdX/0pxNRJL3i2wjbCq+5EwQPzjK6padTYojQUV+4ZGmdb3gCEkRkMzJDOAMGSn/tf2o3MtBGNig5tJlRfPEIaadYSxB62hknzWkveCfwx29gp0JeIiTX5n1LBo7fryqCyZR0Xmzn1oBguRVjq2gv9ulbGjDtiJJdL+LQV6zz3ZU99PpmB462tf+Yb0Jy+67X7fbxIrQ/8HQ1Dqe0ZsYL/Lv+y23ORPmwBzmOWUCvwA+JEToBUbYmdRQ==
Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
(2603:10b6:c00:19::29) by YQXPR01MB3432.CANPRD01.PROD.OUTLOOK.COM
(2603:10b6:c00:52::23) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar
2021 21:48:41 +0000
Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021
21:48:41 +0000
From: Rick Macklem <rmacklem@uoguelph.ca>
To: Alan Somers <asomers@freebsd.org>
CC: freebsd-fs <freebsd-fs@freebsd.org>
Subject: Re: Understanding the vfs.nfsd.request_space* sysctls
Thread-Topic: Understanding the vfs.nfsd.request_space* sysctls
Thread-Index: AQHXGom0SMOUlewkdkSbAUM6yshgqaqHSCtTgAAdewCAAVEfeg==
Date: Wed, 17 Mar 2021 21:48:41 +0000
Message-ID: <YQXPR0101MB0968373DFF89B4FDB6000E4DDD6A9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
References: <CAOtMX2geF7Tv7VYc9WtK23a+nDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com>
<YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>,
<CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ+Y3PG82pN0s2eVusRooRF_g@mail.gmail.com>
In-Reply-To: <CAOtMX2ih9cLq3-Mw9xw3T9ovBiZ+Y3PG82pN0s2eVusRooRF_g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 1c9e5862-1dc7-4773-db70-08d8e98e6e45
x-ms-traffictypediagnostic: YQXPR01MB3432:
x-microsoft-antispam-prvs: <YQXPR01MB34326F662B24D7515FC81A71DD6A9@YQXPR01MB3432.CANPRD01.PROD.OUTLOOK.COM>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: QgeRAAo+d1w0uz/+9hzSEtkpm/NjyyuW0aPpCfkeshZVe743SG/nN/eJGETzSsTKKflFq3gL4wXM/Ht4LQgFCiJNshm4Q60/S8eQN+5yL2+t9CJGqv/utVRmOqpFIgrhfDZN7FYjj/f8aiwHc2C/l0VK63z/GAtP/4QmZYB7WQVYB+PWHK5M7xJJnTXoI9Qdc4jHqvmKbnmKAF3HuHP0wqbRwF1tI9QVmmiH74QJhQz/RXgbjqu6l5MvCcUPtE2ZMCmuThPSQq/gruTlm2bSbFPI6vBY9rfr9LWnBXRHaS41gZfxxslbAEg9ojPp1dhJnTVBhL83tYOBK6Hvh+VP7qHfi/ObWS07kbD3TSyRb4qlCNTwk6o3et3Dawh5gTaZHwNMhehYZR6BZSE+rmGNTmHAS80dYDjRMDxErhY8q9whPzpjSuufuBIQIOnrtSEv3ulBGB4HX9rjKDfebWznT+jFm5/5dLMRhNHdF6LrUU8zr1u7Xq6GsiY427cUu083/6sCd8WLvNAlU74vjIkg8Rv40lkmAlFIpRjsQZV/9AaKpxXppHLxYpiHqnfBYbwmV2uQhZrq+mxY1747QQ+Q9JmKEatN/En+NKbtp6njC52dYrews4h5BFRDDA90KVbn0iB/jofDbyr/DEanZt+cZQ==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE;
SFS:(136003)(376002)(366004)(396003)(39860400002)(346002)(8936002)(2906002)(33656002)(9686003)(8676002)(86362001)(316002)(52536014)(76116006)(55016002)(186003)(478600001)(83380400001)(6916009)(71200400001)(786003)(6506007)(7696005)(66946007)(66476007)(66446008)(66556008)(64756008)(91956017)(5660300002)(4326008)(450100002)(966005);
DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?JdoC7+FN/ZSrf8SWKJn7fiLnownfmWMV1nNlgULrSEY9EcXM0kw+Q44IEa?=
=?iso-8859-1?Q?2eZPXgCNQbkKbhWyrDrreu41emQcpQZaSBo6oUrGmob89+XfrDGuhwMoTY?=
=?iso-8859-1?Q?lOzxze8I5GyOrm7VA3Xa6muqNRUPZVf/CWwOW/jIHfVep8ao6eJ/OGd1bR?=
=?iso-8859-1?Q?QL0ZayAWvKNMtzQcVTFf8URffJRnwHzj9ME6l69yMnblq8MFCgOX5z/I4b?=
=?iso-8859-1?Q?prvcU8UMMuACBqKV2c8ndkUeAJXz58g8JfUl3hgvMhaI/NBf92vuaZ39//?=
=?iso-8859-1?Q?clXGqslcfhsHH4nwwgYMY2WQt/8Ge8ElJ63pQh8B8z16dwDEpzEo9k626s?=
=?iso-8859-1?Q?JQ1tJhEm4XTa0vIAhCnE4ghzwt3ZLjt6WrBIU96BuCHi2a80WBr4Afkwxe?=
=?iso-8859-1?Q?mt64oFYd9DmreErgb4AsrHVoU7uN9ToS/TPSYIRlfNIsqX2LeElSZSk4EZ?=
=?iso-8859-1?Q?yZkj8zkAdFHsj06Jm/cELg3RFiaRLe1RCsZnOa9Fc1jYkUGO+SN8F0E265?=
=?iso-8859-1?Q?H0gDuS+VIlA/gDOGcWcbGXNsTFsy1HUPW/36jCw1X4xUdVqnWBKOt0L0eB?=
=?iso-8859-1?Q?7vJvGg2peUOdOKjiAZeKLpM4q7g8jL/cHLVhJLGUvZs8J3vB3FDm3z88bD?=
=?iso-8859-1?Q?VYw6Ea+7v9kwC3l7gdNq9wBqehUAzDH2OqaFjippVQVF2uczZ6r6QpnF6s?=
=?iso-8859-1?Q?cMmHqytm1GIWnjOfLln6lv3lXC8ZK201dnUjbnMYjRiR3nVuOxu5sJbGSU?=
=?iso-8859-1?Q?tWAAZkNp7a2xrqCHuk+z7ijGef72ulYToKKb7/tZOwJ3aYYv3zzxYDCFhD?=
=?iso-8859-1?Q?tqf20v1aeUsvqtc7+ixVLTOoXQyXll1d06VwHpYd74XeY6Ggw0+6p+qrJ6?=
=?iso-8859-1?Q?f2xoBgTRM6rFDACXI3UJNoR2j3DLqz0wtdK/DhzIoqov6oov1zKXwqKWgk?=
=?iso-8859-1?Q?ya1Dupbi2uv/+Z3F0qYHNx3AsjkzYu/OvG5TPjGs+ViVhf0NqJcr9iHs7Z?=
=?iso-8859-1?Q?XXd2AqUfK5CeKJbrwKFwvsc8q1eIIfc6D8qor+5JAHrKqNq6Dyagejq1g7?=
=?iso-8859-1?Q?r8jCYy24vVdgHHQz3tJ2DPqjav4o1hx/OBwhxiSVfcfgJj2NwKVhsvWwrF?=
=?iso-8859-1?Q?T2QT0MLfitvaHWj3DaNUKt8226QcFOJvuflQ7OMs8yCilohh3jkQGOMfhb?=
=?iso-8859-1?Q?GfekHnQrGUZOPsGrrTjRwC4dprzauSnoHgfgDhY0XWb0HlJk6B6VZtzrHt?=
=?iso-8859-1?Q?x9M7DAv4/I84TfJUddt2tLZVXdRvZ+8QBRI1pRrqlc131bHWHp/BKSsrly?=
=?iso-8859-1?Q?sSL+7AWwyoUE/qgiR0/FjO+ULY1dZInd0t96wEQ59LSfeVH8mpmAZfviXr?=
=?iso-8859-1?Q?HKNofDtF2M8ncQh4Tz1LtiTKySPQaCT1I7BC8sTcdXFxLWVdvrv7E=3D?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: uoguelph.ca
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-Network-Message-Id: 1c9e5862-1dc7-4773-db70-08d8e98e6e45
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 21:48:41.6975 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: +3Jl6bY+r8jMnnWOZVfc8VT/YEjRrXY0ot0S1UQV383wL76VZDq707Ppqe6b1hrOmClV/Qh39ETIX2UKNyubmg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3432
X-Rspamd-Queue-Id: 4F13gv0gqQz3Mrt
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=ZpfgDTZQ;
arc=pass (microsoft.com:s=arcselector9901:i=1);
dmarc=pass (policy=none) header.from=uoguelph.ca;
spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates
40.107.67.82 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca
X-Spamd-Result: default: False [-6.00 / 15.00];
NEURAL_HAM_MEDIUM(-1.00)[-1.000];
RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.82:from];
R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1];
FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[];
TO_MATCH_ENVRCPT_ALL(0.00)[];
R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16];
MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000];
SPAMHAUS_ZRD(0.00)[40.107.67.82:from:127.0.2.255];
RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[];
DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2];
RCVD_IN_DNSWL_NONE(0.00)[40.107.67.82:from];
NEURAL_HAM_SHORT(-1.00)[-1.000];
DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none];
DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim];
FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+];
RCVD_TLS_LAST(0.00)[];
ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US];
ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
MAILMAN_DEST(0.00)[freebsd-fs];
RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.82:from]
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Mar 2021 21:48:44 -0000
Alan Somers wrote:=0A=
>Rick Macklem wrote:=0A=
[stuff snipped]=0A=
>>Anytime used > high, it throttles, which means it leaves=0A=
>>the RPC messages on the socket receive queue.=0A=
>>This indicates the server is not keeping up with requests.=0A=
>>(ie Overloaded or ???)=0A=
>=0A=
>That all makes sense. So having a high request_space_used basically just =
means that >my storage is too slow.=0A=
One other thing you might want to look at is rcv interrupt moderation=0A=
in the net device driver.=0A=
--> This *might* result in received RPC requests being "batched", resulting=
in=0A=
a burst of RPCs that overloads the server. (Except for write requests=
, each=0A=
RPC is a small message typically in a single small packet.)=0A=
--> Disabling rcv interrupt moderation will result in more interrupts=
, but=0A=
then each RPC request will be handled right away. This might r=
esult=0A=
in more evenly spaced RPCs (in a temporal sense).=0A=
Also, delaying processing of RPC requests increases RPC RTT an=
d that=0A=
always has a negative impact on NFS performance.=0A=
=0A=
rick=0A=
=0A=
> And how can I either=0A=
>increase the available space or decrease the stuff that's using it?=0A=
Increasing vfs.nfsd.request_space_high avoids the throttling,=0A=
but it is hard to say that is a good idea, since there won't be=0A=
backpressure applied to the clients via TCP windows, etc.=0A=
=0A=
Fixing the server so that it isn't overloaded would be better,=0A=
I think?=0A=
--> I'm the last guy to take ZFS advice from, but I think there=0A=
is a tunable w.r.t. how much arc is used for metadata.=0A=
Getattrs will need cached (metadata) to reply quickly.=0A=
Lookups also depend on cached attributes for good=0A=
perf. as well.=0A=
=0A=
I've added an L2ARC since I sent that original email. We'll see how well i=
t works. I expect it to take 48 hours before results are apparent.=0A=
=0A=
--> Make sure you have lots of nfsd threads. They are=0A=
just kernel threads, so they don't use a lot of resources=0A=
and having too many is much better than not enough.=0A=
-->I can't remember what the upper limit is these days,=0A=
but I'd set it to that for a busy nfs server.=0A=
For NFSv4.1, each client can send a maximum of=0A=
session_slot_table_size concurrent RPCs. FreeBSD=0A=
uses a single 64slot table for each mount. I think=0A=
Linux uses a 32slot table, but supports trunking.=0A=
I don't know if the Linux client uses a separate=0A=
session table for each trunk or not?=0A=
--> Something like #clients * 32 (or 64) nfsd=0A=
threads running on the server.=0A=
=0A=
Experimentally, I determined that 768 threads is sufficient. But your form=
ula would suggest > 2500, which is a lot of threads! I'll keep it in mind =
if I ever reach 768, though.=0A=
=0A=
=0A=
rick=0A=
=0A=
-Alan=0A=
_______________________________________________=0A=
freebsd-fs@freebsd.org<mailto:freebsd-fs@freebsd.org> mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A=
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org<mailto=
:freebsd-fs-unsubscribe@freebsd.org>"=0A=
=0A=