From nobody Mon Aug 23 01:15:46 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4C94C179211C for ; Mon, 23 Aug 2021 01:15:59 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Received: from mail-lj1-x230.google.com (mail-lj1-x230.google.com [IPv6:2a00:1450:4864:20::230]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GtDp65z20z3qP3; Mon, 23 Aug 2021 01:15:58 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Received: by mail-lj1-x230.google.com with SMTP id h1so234476ljl.9; Sun, 22 Aug 2021 18:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lwV+G3nd36RqI0eLDa6HhiZ8RoLX8GMELZDbL3LM728=; b=n+XEgGmbr8TocUOadABAlbxc0gPE7acqxzX21knpKZ/iUPP23zIHf/6cgrd8L3npcg FJmVbqRa75nMdHW9XfbylWAxK2XIwmcBF1Zx88G55b1VcyfNXMcs1VQN+KTQaPYim8z8 AbEk/zDBblOX2cCzXkYh1q/DXedKjBUHmFy5y1aFyM1/1YHikky/4yMqZG2iFgmHZIIg HGTIcvf/L04vB44k6BjJtBPIdAmgdpRZIy48UmfGh2JGdtcxPJAdUFl+qOgxQoEt4/9u T0gHyRWHfCd5Va4yv5Xvuw1vAsd/7fMR1jZhdLBA6NnDJf4SkE2/AOYRKDHdwyNTlFb0 Hh4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lwV+G3nd36RqI0eLDa6HhiZ8RoLX8GMELZDbL3LM728=; b=d7iKBFUlu27Zv2RwrrbeSwPeGUfJb5gLrkfK0cZnVGQgVrb+RVxhJJHobRe2/c0jig b9lC4fKAkp1vXfvjaxbcSyJXQI+Q7IOC5Kz52h7Rgv8inAbqu4sUq/5+EyNMnNlXT2yT WEFH0toU54nft/Wz9TiVEpr6WA8PbCpyy4npjY1sYKEM092hd6w4fQ4ahz2FmTfDX7el UKLJpVc8f2pVbSNUMbMbUlK5eYhiBi3fbzlNJySnd55Eqd77PFb39MXX4zCNh/eKogMa I1xRNHgMGQf4QbbtpUKsQMwhgAvcAcS5ybwK3T0F9Waijql9f0+2ehRozB7Og5lD0YMG hIDQ== X-Gm-Message-State: AOAM533nLXcpKzmQ0M2D3vgBM4TyKRFAX2EQavYpaJ50rqAbNh4KfRGr SX71wg4fPiVmaFEY7+zHf4cAJUPUo55B+i6wUlG2PAaF X-Google-Smtp-Source: ABdhPJynujMaENADjkdO8y7j0i/RW4I1/8rymT2zAJ6SFKlC5y2FFDgPj+VzQXN/y6GavcIT11ydiZd0C6oRPPj1yqk= X-Received: by 2002:a2e:a499:: with SMTP id h25mr25488379lji.175.1629681357471; Sun, 22 Aug 2021 18:15:57 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <023225AD-2A97-47C5-9FE4-3ABF1BFD66F1@gmx.com> In-Reply-To: From: Chris Watson Date: Sun, 22 Aug 2021 20:15:46 -0500 Message-ID: Subject: Re: ZFS on high-latency devices To: George Michaelson Cc: Alan Somers , Ben RUBSON , freebsd-fs Content-Type: multipart/alternative; boundary="00000000000068c16a05ca2fc2ec" X-Rspamd-Queue-Id: 4GtDp65z20z3qP3 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: Y --00000000000068c16a05ca2fc2ec Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > > > This may be just adding weeds to the conversation but what does Panzura do here? Isn=E2=80=99t their hybrid cloud ZFS based product encrypted in the cloud a= nd in transit? I=E2=80=99m willing to bet someone working there could offer some valuable = insight here as I=E2=80=99m pretty sure they have fleshed this dragon out. Chris --00000000000068c16a05ca2fc2ec-- From nobody Mon Aug 23 17:54:41 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8BD1A1774719 for ; Mon, 23 Aug 2021 17:54:45 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GtfyY3ZH5z4dsp; Mon, 23 Aug 2021 17:54:45 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4859CF2C0; Mon, 23 Aug 2021 17:54:45 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.134.16] (unknown [94.19.243.255]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 932F8ECD1; Mon, 23 Aug 2021 20:54:41 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: ZFS on high-latency devices To: Alan Somers , George Michaelson Cc: Ben RUBSON , freebsd-fs References: <023225AD-2A97-47C5-9FE4-3ABF1BFD66F1@gmx.com> From: Lev Serebryakov Organization: FreeBSD Message-ID: <90ec9734-c4d8-b164-aec1-f95fcfda913b@FreeBSD.org> Date: Mon, 23 Aug 2021 20:54:41 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N On 23.08.2021 2:48, Alan Somers wrote: > mbuffer is not going to help the OP. He's trying to create a pool on top > of a networked block device. And if I understand correctly, he's > connecting over a WAN, not a LAN. ZFS will never achieve decent > performance in such a setup. It's designed as a local file system, and > assumes it can quickly read metadata off of the disks at any time. The > OP's best option is to go with "a": encrypt each dataset and send them with > "zfs send --raw". I don't know why he thinks that it would be "very > difficult". It's quite easy, if he doesn't care about old snapshots. Just: > > $ zfs create pool/new_dataset > $ cp -a pool/old_dataset/* pool/new_dataset/ Twice the space? And, realistically, triple the space to have enough free space before destroying pool/old_dataset, as ZFS feels bad on near-full pools. Sometimes it is "just", sometimes it is impossible! -- // Lev Serebryakov From nobody Tue Aug 24 11:54:45 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C2F74178E034 for ; Tue, 24 Aug 2021 11:54:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gv6wj4zXbz3vmH for ; Tue, 24 Aug 2021 11:54:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8CE4527CD7 for ; Tue, 24 Aug 2021 11:54:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 17OBsjtF020430 for ; Tue, 24 Aug 2021 11:54:45 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 17OBsj5s020429 for fs@FreeBSD.org; Tue, 24 Aug 2021 11:54:45 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 258022] [FUSES] Inode attributes are cached unnecessarily/for too long Date: Tue, 24 Aug 2021 11:54:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258022 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Aug 24 12:22:58 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9FC47177339E for ; Tue, 24 Aug 2021 12:22:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gv7YG3ywZz4YMl for ; Tue, 24 Aug 2021 12:22:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7129561B for ; Tue, 24 Aug 2021 12:22:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 17OCMwon037445 for ; Tue, 24 Aug 2021 12:22:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 17OCMwVE037444 for fs@FreeBSD.org; Tue, 24 Aug 2021 12:22:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 258022] [FUSEFS] Inode attributes are cached unnecessarily/for too long Date: Tue, 24 Aug 2021 12:22:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: chogata@moosefs.pro X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258022 Agata changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|[FUSES] Inode attributes |[FUSEFS] Inode attributes |are cached |are cached |unnecessarily/for too long |unnecessarily/for too long --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Aug 24 13:44:51 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 822B6179E368 for ; Tue, 24 Aug 2021 13:44:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gv9Ml3BM8z4wkg for ; Tue, 24 Aug 2021 13:44:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5071D16BB for ; Tue, 24 Aug 2021 13:44:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 17ODipbp073587 for ; Tue, 24 Aug 2021 13:44:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 17ODipEv073586 for fs@FreeBSD.org; Tue, 24 Aug 2021 13:44:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 258022] [FUSEFS] Inode attributes are cached unnecessarily/for too long Date: Tue, 24 Aug 2021 13:44:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258022 Alan Somers changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |asomers@FreeBSD.org --- Comment #1 from Alan Somers --- I'll look into this when I have some time. But could you please tell me wh= at FUSE protocol version MooseFS is mounting with? Earlier protocol versions = had very limited ability to specify cache retention times. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Aug 25 08:19:07 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 990D41777D10 for ; Wed, 25 Aug 2021 08:19:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gvf5R3sXkz3lbk for ; Wed, 25 Aug 2021 08:19:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 66F5E1919B for ; Wed, 25 Aug 2021 08:19:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 17P8J7Fu045590 for ; Wed, 25 Aug 2021 08:19:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 17P8J7wX045589 for fs@FreeBSD.org; Wed, 25 Aug 2021 08:19:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 258022] [FUSEFS] Inode attributes are cached unnecessarily/for too long Date: Wed, 25 Aug 2021 08:19:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: chogata@moosefs.pro X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258022 --- Comment #2 from Agata --- This is directly from the test machine I did this particular test on: compiled_with_fuse: 3.2 kernel_fuse_protocol: 7.28 I don't know about our users, but I assume they mount with the latest they = get with the system. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Aug 25 13:34:36 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 53C4217733B8 for ; Wed, 25 Aug 2021 13:34:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gvn5S1VJyz4hpT for ; Wed, 25 Aug 2021 13:34:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1843E1D437 for ; Wed, 25 Aug 2021 13:34:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 17PDYaZd002511 for ; Wed, 25 Aug 2021 13:34:36 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 17PDYaqH002510 for fs@FreeBSD.org; Wed, 25 Aug 2021 13:34:36 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 258022] [FUSEFS] Inode attributes are cached unnecessarily/for too long Date: Wed, 25 Aug 2021 13:34:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jSML4ThWwBID69YC@protonmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258022 --- Comment #3 from jSML4ThWwBID69YC@protonmail.com --- As a user, here's my settings.=20 compiled_with_fuse: 31.0 kernel_fuse_protocol: 7.28 PKG: fusefs-libs3-3.10.4 PKG: moosefs3-client-3.0.116 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Aug 26 23:29:29 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 44495178D1BE for ; Thu, 26 Aug 2021 23:30:04 +0000 (UTC) (envelope-from joe@via.net) Received: from smtp1.via.net (smtp1.via.net [157.22.3.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "via.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GwfFs6M66z3l1h for ; Thu, 26 Aug 2021 23:29:53 +0000 (UTC) (envelope-from joe@via.net) Received: from mail.via.net (mail.via.net [157.22.3.34]) by smtp1.via.net (8.15.2/8.14.1-VIANET) with ESMTPS id 17QNTdlG003453 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 26 Aug 2021 16:29:40 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at smtp1.via.net Received: from smtpclient.apple ([209.81.2.65]) by mail.via.net (8.15.2/8.14.1-VIANET) with ESMTP id 17QNTdl1004162 for ; Thu, 26 Aug 2021 16:29:39 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at mail.via.net From: joe mcguckin Content-Type: multipart/alternative; boundary="Apple-Mail=_18E230CF-FD8C-43B5-B274-21F14CC62ACF" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Backups to disk Message-Id: <13B8BFED-E1E0-4912-BD9E-6581C308FF25@via.net> Date: Thu, 26 Aug 2021 16:29:29 -0700 To: freebsd-fs X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (smtp1.via.net [157.22.3.5]); Thu, 26 Aug 2021 16:29:42 -0700 (PDT) X-Rspamd-Queue-Id: 4GwfFs6M66z3l1h X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=temperror reason="query timed out" header.from=via.net (policy=temperror); spf=temperror (mx1.freebsd.org: error in processing during lookup of joe@via.net: DNS error) smtp.mailfrom=joe@via.net X-Spamd-Result: default: False [-2.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[joe]; FROM_HAS_DN(0.00)[]; R_SPF_DNSFAIL(0.00)[temporary DNS error]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DMARC_DNSFAIL(0.00)[via.net : query timed out]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[157.22.3.5:from] X-ThisMailContainsUnwantedMimeParts: Y --Apple-Mail=_18E230CF-FD8C-43B5-B274-21F14CC62ACF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I created a second zfs filesystem to be used as backups. I made a = snapshot and tried zfs send archive@08252021 | zfs recv = /backup/08252021. This fails with Freebsd complaining that zfs recv got a signal and = aborted. Any thoughts? Thanks, Joe Joe McGuckin ViaNet Communications joe@via.net 650-207-0372 cell 650-213-1302 office 650-969-2124 fax --Apple-Mail=_18E230CF-FD8C-43B5-B274-21F14CC62ACF-- From nobody Fri Aug 27 01:04:06 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BD58C178E857 for ; Fri, 27 Aug 2021 01:04:20 +0000 (UTC) (envelope-from ggm@algebras.org) Received: from mail-lj1-x232.google.com (mail-lj1-x232.google.com [IPv6:2a00:1450:4864:20::232]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GwhLr44rQz4gJP for ; Fri, 27 Aug 2021 01:04:20 +0000 (UTC) (envelope-from ggm@algebras.org) Received: by mail-lj1-x232.google.com with SMTP id q21so8461071ljj.6 for ; Thu, 26 Aug 2021 18:04:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=algebras-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Slq42UFNJHHSdZa0ZKfD8qdfCFtFG+QKcuSKAz7TKTA=; b=GvRVQopIICJEWrhnbEnj6G7PQphfKr3WarWdcUSm0PljLmLAFXxRzMVM1TD80xXxOn mkqJXIVIpsWBS+ElEtFqmRDbSjYc+iPoAyDDpcaqVGMs5BQjCjG5OOOUgweMZFH2dh9l 5kNVAF0HVrnL2l6/fegxcNBt4O1vXnEEBRRYK54p/0E5EhyjFPtZ8zuBJvjWsJfwW/Iv Jd46MMMcyOhQtwCGWG2zz9mxg/uRk83TgE9lvGl1WERhE52nZ3J1N5QTnKnjUWVSlF2i RqDJ3IrQ1q3shZ8ZHvuejoMUgRgLNJ1+a/g7n0m0G5aeqqRtu6P0MNqSfz85U0XNI/ye GNDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Slq42UFNJHHSdZa0ZKfD8qdfCFtFG+QKcuSKAz7TKTA=; b=LAn25yiulF3zEc1eD/sy//d/VWUJ+I0CTUEN3KIkFJHwNE+rv/TTPy6v30/hSfIbqF Z39rw/j5PvtCEmW8/DAwmkeny8CipZLDWdJSsAmiOcfJCprw/lpZyPFvbfQo8+ozRQ7s WWKfjpoiRItTUnmxKJqmCJAZ1DvaKqQ7gARHXJZJRZMKxm/Q7jJDeiUrWV38Wqkhoc/B VV+ZWxXW06CotPJb5an/h7WlgtOms3wnPDv9ixpgWSLaokKi1DhLVelVzjH9gAk5FpyB 9Lvxm6v6Tr8QgmDG0kKTlFSq6gI55A83uWPwMNE7hjHMkCI1ARj1P6lBeB1dTk8Dd56z sEwg== X-Gm-Message-State: AOAM5328VrDuF8d1DZ5b0uN8l66qC2lOuacbueOjJ8NdcjkO9rCqAQEa 6DQZRIj2Omy6ELJETokpgSK9315qaYWnZbEOxxrNtg== X-Google-Smtp-Source: ABdhPJytTXduKzT+W2lO741MWeReTmho5msKcxuoMeGXuSaPUXqvtDWSHfi9WylVSK+W7Lm4hUN6l26ipRfiuGvyqbs= X-Received: by 2002:a05:651c:11c7:: with SMTP id z7mr5481817ljo.288.1630026258190; Thu, 26 Aug 2021 18:04:18 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <13B8BFED-E1E0-4912-BD9E-6581C308FF25@via.net> In-Reply-To: <13B8BFED-E1E0-4912-BD9E-6581C308FF25@via.net> From: George Michaelson Date: Fri, 27 Aug 2021 11:04:06 +1000 Message-ID: Subject: Re: Backups to disk To: joe mcguckin Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4GwhLr44rQz4gJP X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N This is how I do it (this example is using increments) zfs send -R -i tank@2021-05-16 tank@2021-07-28 | zfs recv -s -F -v backup On Fri, Aug 27, 2021 at 9:30 AM joe mcguckin wrote: > > I created a second zfs filesystem to be used as backups. I made a snapshot and tried zfs send archive@08252021 | zfs recv /backup/08252021. > > This fails with Freebsd complaining that zfs recv got a signal and aborted. > > Any thoughts? > > Thanks, > > Joe > > > Joe McGuckin > ViaNet Communications > > joe@via.net > 650-207-0372 cell > 650-213-1302 office > 650-969-2124 fax > > > From nobody Fri Aug 27 08:08:45 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AE14E17942CD for ; Fri, 27 Aug 2021 08:08:54 +0000 (UTC) (envelope-from matt@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gwsmk3N5Tz4cWd for ; Fri, 27 Aug 2021 08:08:54 +0000 (UTC) (envelope-from matt@userve.net) Received: from dc.ad.unitron.net (owa.usd-group.com [217.196.1.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id 1D3B8238450; Fri, 27 Aug 2021 09:08:46 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1630051726; bh=gN3UqzYM2gVZyl5ypvqxazVfw1BdsnGSIHvXC7D2nXo=; h=From:To:Subject:Date:References:In-Reply-To; b=S8Bi1k7VY5E4JG0w3pMBGt39lB4V9RptLucNpbtNRhV+PUdh3bBWzZqdt2BX21z/X rgPoME8J/n+eo2tzFLjlDVQM02EnNGJO7d/kKOCASETvBbyFKkTIc9wYvquCHoR2rV +ot/gs/n346YdSRcmk9BGCJVHtCLOI7WGupF97M0= Received: from dc.ad.unitron.net (192.168.0.100) by dc.ad.unitron.net (192.168.0.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.5; Fri, 27 Aug 2021 09:08:45 +0100 Received: from dc.ad.unitron.net ([fe80::5f3:fd35:db1:9809]) by dc.ad.unitron.net ([fe80::5f3:fd35:db1:9809%6]) with mapi id 15.02.0858.002; Fri, 27 Aug 2021 09:08:45 +0100 From: Matt Churchyard To: joe mcguckin , freebsd-fs Subject: RE: Backups to disk Thread-Topic: Backups to disk Thread-Index: AQHXmtJXMI4mGsgSYEOUlrpwalSE6KuG/NVw Date: Fri, 27 Aug 2021 08:08:45 +0000 Message-ID: References: <13B8BFED-E1E0-4912-BD9E-6581C308FF25@via.net> In-Reply-To: <13B8BFED-E1E0-4912-BD9E-6581C308FF25@via.net> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.8] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4Gwsmk3N5Tz4cWd X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N Do you mean you created a second zfs pool? A pool can contain multiple file= systems.=20 Judging by the command you give it looks like you have two pools called "ar= chive" and "backup", which by default will contain a single filesystem (dat= aset) with the same name, mounted on /{name}. If that's the exact command you used then the recv argument is wrong. It sh= ould be the dataset name, not a path. (i.e. drop the first slash if you wan= t the backup stored in a dataset called 08252021 on the backup pool) Personally, assuming you are backing up the primary dataset on the "archive= " pool, I would just do something like the following - zfs snapshot archive@08252021 zfs send archive@08252021 | zfs recv backup/archive Then the next day (as long as you still have the 08252021 snapshot) - zfs snapshot backup@08262021 zfs send -i 08252021 archive@08262021 | zfs recv backup/archive This will send only the changes, and you can always look through the snapsh= ots on the backup pool to see the list of snapshots and dates. Creating a b= ackup dataset using the date doesn't make a lot of sense, to me at least. A= fter this you can delete the 08252021 snapshot on the archive pool unless y= ou have any specific reason to want to keep it. If the backup pool was on a= separate system then it often makes sense to keep some snapshots so you ca= n easily recover files on the live system without having to pull them from = backup. If the backup pool is in the same system then there's not really an= y point. Matt -----Original Message----- From: owner-freebsd-fs@freebsd.org On Behalf= Of joe mcguckin Sent: 27 August 2021 00:29 To: freebsd-fs Subject: Backups to disk I created a second zfs filesystem to be used as backups. I made a snapshot = and tried zfs send archive@08252021 | zfs recv /backup/08252021. This fails with Freebsd complaining that zfs recv got a signal and aborted. Any thoughts? Thanks, Joe Joe McGuckin ViaNet Communications joe@via.net 650-207-0372 cell 650-213-1302 office 650-969-2124 fax From nobody Fri Aug 27 09:05:59 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D1DA5177697A for ; Fri, 27 Aug 2021 09:06:08 +0000 (UTC) (envelope-from SRS0=bVb0=NS=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gwv2l5BHfz4v2W for ; Fri, 27 Aug 2021 09:06:07 +0000 (UTC) (envelope-from SRS0=bVb0=NS=klop.ws=ronald-lists@realworks.nl) Date: Fri, 27 Aug 2021 11:05:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1630055159; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=cqHI5/htJvbNvZtW3egUhg7TzfRd7Fw+acS5ae876b4=; b=Wcp2guJndJ3RwQcPPz4UkGikVqixl8YCWxT/81nWTXeprrK7BkY83a3pI/zwQfGUTOKtgK ObILq9QNZKG+vzoRAACz4wqfCNQk4xpD153DgeaWsDJHe6oe7GT8iG4MFQZfkLRngLIEdY OZkSYn0Nhqg7ISil9xRosmuvbbZhexfOXACjHyDyBXB3UFhrLkjwc+0Ql6m8H/51G067Qj 8abCuBN0YvxSyNO/kPXu81kVIPZJ2kDOuHOU5+NtclFgelSlhsTu/kVyegJ9+7ETT63LIo c5zIzD/Blz6cty2JhxITWckB2luO7xaATyTk8xxUTcmq9at9DP9YqDdRRVTKrg== From: Ronald Klop To: freebsd-fs Message-ID: <240616455.10.1630055159842@localhost> In-Reply-To: <13B8BFED-E1E0-4912-BD9E-6581C308FF25@via.net> References: <13B8BFED-E1E0-4912-BD9E-6581C308FF25@via.net> Subject: Re: Backups to disk List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_9_402275875.1630055159833" X-Mailer: Realworks (574.746.c373083) Importance: Normal X-Priority: 3 (Normal) X-Rspamd-Queue-Id: 4Gwv2l5BHfz4v2W X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=rw2 header.b=Wcp2guJn; dmarc=pass (policy=none) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of "SRS0=bVb0=NS=klop.ws=ronald-lists@realworks.nl" designates 194.109.157.24 as permitted sender) smtp.mailfrom="SRS0=bVb0=NS=klop.ws=ronald-lists@realworks.nl" X-Spamd-Result: default: False [-3.20 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[klop.ws:s=rw2]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:194.109.157.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_SHORT(-1.00)[-0.998]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[klop.ws:+]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,none]; HAS_X_PRIO_THREE(0.00)[3]; RCVD_IN_DNSWL_NONE(0.00)[194.109.157.24:from]; FORGED_SENDER(0.30)[ronald-lists@klop.ws,SRS0=bVb0=NS=klop.ws=ronald-lists@realworks.nl]; RCVD_COUNT_ZERO(0.00)[0]; RWL_MAILSPIKE_POSSIBLE(0.00)[194.109.157.24:from]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]; FROM_NEQ_ENVFROM(0.00)[ronald-lists@klop.ws,SRS0=bVb0=NS=klop.ws=ronald-lists@realworks.nl] X-ThisMailContainsUnwantedMimeParts: Y ------=_Part_9_402275875.1630055159833 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Hi, What you are trying to do does not sound weird. So the error might be in the details. Can you copy-paste the command and the output? Or the mention of the signal in the logs somewhere. It can be quite important what signal it gets. Some output of "zfs list" and "zpool status" might also help. Regards, Ronald. Van: joe mcguckin Datum: vrijdag, 27 augustus 2021 01:29 Aan: freebsd-fs Onderwerp: Backups to disk > > I created a second zfs filesystem to be used as backups. I made a snapshot and tried zfs send archive@08252021 | zfs recv /backup/08252021. > > This fails with Freebsd complaining that zfs recv got a signal and aborted. > > Any thoughts? > > Thanks, > > Joe > > > Joe McGuckin > ViaNet Communications > > joe@via.net > 650-207-0372 cell > 650-213-1302 office > 650-969-2124 fax > > > > > > ------=_Part_9_402275875.1630055159833-- From nobody Fri Aug 27 13:48:38 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 314A41775608 for ; Fri, 27 Aug 2021 13:48:48 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "DFN-Verein Global Issuing CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gx1Jv0s48z4b91 for ; Fri, 27 Aug 2021 13:48:46 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from [192.168.111.102] (p4fd978e7.dip0.t-ipconnect.de [79.217.120.231]) (authenticated bits=0) by unimail.uni-dortmund.de (8.17.1/8.17.1) with ESMTPSA id 17RDmdTT004507 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Fri, 27 Aug 2021 15:48:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1630072119; bh=75DaACqcNNYaVnvlddk0zWV6VzVRUrzyaIdBx1kvTyc=; h=From:To:Cc:Subject:Date; b=igLWZHJW9C8mzoPcSfmt2gSrV66Zi3KM/0rn7soZqjc4C4yxYiGo02F3WiLWS4uj5 WIRpMGM83HxttcyfmDKSoyv3eBnq9e3p6/bHfmraqqlevwV5DJpEkIP3yCXNkWDypo QZoBq55J5mY+6NSVa1zITWm+Vy5SGB/OarPOYUE4= From: Alexander Lochmann To: freebsd-fs , Konstantin Belousov Cc: Horst Schirmeier Subject: Various unprotected accesses to buf and vnode Message-ID: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> Date: Fri, 27 Aug 2021 15:48:38 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4Gx1Jv0s48z4b91 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tu-dortmund.de header.s=unimail header.b=igLWZHJW; dmarc=none; spf=pass (mx1.freebsd.org: domain of alexander.lochmann@tu-dortmund.de designates 129.217.128.51 as permitted sender) smtp.mailfrom=alexander.lochmann@tu-dortmund.de X-Spamd-Result: default: False [-5.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[129.217.128.51:from]; R_SPF_ALLOW(-0.20)[+ip4:129.217.128.0/24]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[129.217.128.51:from]; DKIM_TRACE(0.00)[tu-dortmund.de:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[freebsd.org,gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:129.217.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[79.217.120.231:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[tu-dortmund.de:s=unimail]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tu-dortmund.de]; DWL_DNSWL_LOW(-1.00)[tu-dortmund.de:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hi folks, I'm still analyzing our LockDoc (lock analysis) data for FreeBSD. I came across accesses that do not adhere to the locking documentation. I cannot tell whether these accesses are made deliberately without locks or not. I listed them below. Can you please shed some light on those cases? Thx and regards, Alex - Write to buf.b_error without buf.b_lock: https://github.com/freebsd/freebsd-src/blob/main/sys/kern/vfs_vnops.c#L2846 - Read of buf.b_blkno in cluster_write(): According to the documentation b_lock is needed. Is b_blkno maybe a read-only element of struct buf? - Read of buf.b_flags, buf.b_xflags and buf.b_vp: https://github.com/freebsd/freebsd-src/blob/main/sys/kern/vfs_subr.c#L2351 Are those reads innocent races? According to our data, buf.b_lock is not acquired. - Write to vnode.v_bufobj.bo_object: https://github.com/freebsd/freebsd-src/blob/main/sys/vm/vnode_pager.c#L291 According to the documentation, '[...] the vnode lock which embeds the bufobj' is needed. However, interlock is taken in line 276. Is the interlock equivalent to the vnode lock? (I assume 'the vnode lock' refers to vnode.v_lock.) - Is buf.b_bufobj a read-only element? -- Technische Universität Dortmund Alexander Lochmann PGP key: 0xBC3EF6FD Otto-Hahn-Str. 16 phone: +49.231.7556141 D-44227 Dortmund fax: +49.231.7556116 http://ess.cs.tu-dortmund.de/Staff/al From nobody Fri Aug 27 18:40:01 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 824921773B5D for ; Fri, 27 Aug 2021 18:40:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gx7n708LWz3vvS for ; Fri, 27 Aug 2021 18:40:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 17RIe23Z018668 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 27 Aug 2021 21:40:05 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 17RIe23Z018668 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 17RIe1h9018658; Fri, 27 Aug 2021 21:40:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 27 Aug 2021 21:40:01 +0300 From: Konstantin Belousov To: Alexander Lochmann Cc: freebsd-fs , Horst Schirmeier Subject: Re: Various unprotected accesses to buf and vnode Message-ID: References: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.5 X-Spam-Checker-Version: SpamAssassin 3.4.5 (2021-03-20) on tom.home X-Rspamd-Queue-Id: 4Gx7n708LWz3vvS X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Fri, Aug 27, 2021 at 03:48:38PM +0200, Alexander Lochmann wrote: > Hi folks, > > I'm still analyzing our LockDoc (lock analysis) data for FreeBSD. I came > across accesses that do not adhere to the locking documentation. I cannot > tell whether these accesses are made deliberately without locks or not. > I listed them below. > > Can you please shed some light on those cases? > > Thx and regards, > Alex > > - Write to buf.b_error without buf.b_lock: > https://github.com/freebsd/freebsd-src/blob/main/sys/kern/vfs_vnops.c#L2846 I believe this is fine. If the buffer is still on the delayed write queue, then nobody can start the write, and buffer cannot leave the queue because we locked the bo lock. It might be slightly formally better to move clearing of b_error after we obtained the buffer lock, but I do not want to change this code just for formality. It might be that the clearing can be avoided at all, in fact. > > - Read of buf.b_blkno in cluster_write(): > According to the documentation b_lock is needed. > Is b_blkno maybe a read-only element of struct buf? No, b_blkno is not read-only, it is the disk block number for the block, as opposed to b_lbklno which is logical file block number. The buffer is instantiated with b_blkno == b_lblkno, and when the buffer is mapped to the real disk block, b_blkno is updated. Could you show me the backtrace of the situation where cluster_write() is called with unlocked buffer? > > - Read of buf.b_flags, buf.b_xflags and buf.b_vp: > https://github.com/freebsd/freebsd-src/blob/main/sys/kern/vfs_subr.c#L2351 > Are those reads innocent races? > According to our data, buf.b_lock is not acquired. These are fine. We check that nbp buffer is still on the clean/dirty queue of our vnode, and we own the buffer object lock. Since buffers are moved from the queues under the bo lock, the operation is safe. You may think about it in the following way: - the update of the flags word require owning corresponding lock to not loose other updates - but changes of the flags that are tested in the referenced lines are also protected by the buffer object lock > > - Write to vnode.v_bufobj.bo_object: > https://github.com/freebsd/freebsd-src/blob/main/sys/vm/vnode_pager.c#L291 > According to the documentation, '[...] the vnode lock which embeds the > bufobj' is needed. However, interlock is taken in line 276. > Is the interlock equivalent to the vnode lock? > (I assume 'the vnode lock' refers to vnode.v_lock.) vnode_pager_alloc() must be called with the vnode exclusively locked. This is asserted at the beginning of the function. > > - Is buf.b_bufobj a read-only element? You should scope the question. While buffer is owned by a vnode, b_bufobj is constant. Since buffers are type-stable, they migrate between vnodes as cache finds it required to reclaim and reuse. There, b_bufobj is changed. From nobody Sat Aug 28 02:03:05 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 91DE6178476B for ; Sat, 28 Aug 2021 02:03:32 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "vtr.rulingia.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GxKcg0vBpz4TQV; Sat, 28 Aug 2021 02:03:30 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (2001-44b8-31fc-0d00-11e8-f292-ae6b-537a.static.ipv6.internode.on.net [IPv6:2001:44b8:31fc:d00:11e8:f292:ae6b:537a]) by vtr.rulingia.com (8.16.1/8.15.2) with ESMTPS id 17S23BX5084633 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 28 Aug 2021 12:03:17 +1000 (AEST) (envelope-from peter@rulingia.com) DKIM-Filter: OpenDKIM Filter v2.10.3 vtr.rulingia.com 17S23BX5084633 X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.16.1/8.16.1) with ESMTPS id 17S235TP090583 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 28 Aug 2021 12:03:06 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.16.1/8.16.1/Submit) id 17S235vQ090582; Sat, 28 Aug 2021 12:03:05 +1000 (AEST) (envelope-from peter) Date: Sat, 28 Aug 2021 12:03:05 +1000 From: Peter Jeremy To: Alan Somers Cc: George Michaelson , Ben RUBSON , freebsd-fs Subject: Re: ZFS on high-latency devices Message-ID: References: <023225AD-2A97-47C5-9FE4-3ABF1BFD66F1@gmx.com> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="R5DOk3Sae8nJZlGU" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp X-Rspamd-Queue-Id: 4GxKcg0vBpz4TQV X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=quarantine) header.from=rulingia.com; spf=pass (mx1.freebsd.org: domain of peter@rulingia.com designates 2001:19f0:5801:ebe:5400:1ff:fe53:30fd as permitted sender) smtp.mailfrom=peter@rulingia.com X-Spamd-Result: default: False [-5.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[peter]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MID_RHS_MATCH_FROMTLD(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[rulingia.com,quarantine]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; FREEMAIL_CC(0.00)[algebras.org,gmx.com,freebsd.org]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --R5DOk3Sae8nJZlGU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2021-Aug-22 17:48:13 -0600, Alan Somers wrote: >mbuffer is not going to help the OP. I agree that mbuffer won't help. I already use something equivalent to remove the read latency on the send side. >And if I understand correctly, he's >connecting over a WAN, not a LAN. ZFS will never achieve decent >performance in such a setup. It's designed as a local file system, and >assumes it can quickly read metadata off of the disks at any time. Yes. But, at least with a relatively empty destination, zfs actually does almost no reads whilst doing a recv. As far as I can tell, the problem is that zfs does a complete flush of all data and metadata at snapshot boundaries. This is painful even with local filesystems (it typically takes >1s to recv an empty snapshot with local disks). >The >OP's best option is to go with "a": encrypt each dataset and send them with >"zfs send --raw". I don't know why he thinks that it would be "very >difficult". It's quite easy, if he doesn't care about old snapshots. Jus= t: I agree that "zfs send --raw" is the best solution to network RTT and I agree that migrating to ZFS native encryption is quite easy if you don't care about any ZFS features. However, I do care about old snapshots - migrating to ZFS native encryption is a non-starter if it involves throwing away all my old snapshots and clones. I have also been working on migrating to native encryption. I know how to migrate snapshots and think there a way to migrate clones (but I need to validate it). The remaining definite blocker is working out how to migrate the pool root filesystem (including snapshots). --=20 Peter Jeremy --R5DOk3Sae8nJZlGU Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAmEpmVNfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzRr7hAAmb3w4XGMgHuPi97RlWZcYahErnCaYB+ROGs2pJLRUVJQyTDGxBQqVVJ8 AEiNYtMwdCqtj9WI42klYuJQB5tO4IhByKulcDs8oqzeUSJW2IVeqRiMfqxJMyuk 74XHBLiDCZyy6+U4d/Bh6YIWQZpXsZXNUFJ+GcKy/1Y0ciyPAjeugRXzxpj5+Wg3 PaXC2zOnlipWSbHI9966JsKn/2yIXIsPQadfIhBy0KsdlXGdRq2mGVkgmaXpLBJD nxdN+fY/6ighFRuc4orh2RY4HOIpRPnWGYpG4MBAjTB2CskrgpSRxms2vYsoOaYq hJOe6t5BjWehh+cC3aq86Z9M15ceaogIk+D9D6okOOsOUBJfCfDUoNmlzAhZAZpp YEfp1tiToPKvBtccWAsIs8qoxp9aaChuwl6qxpELIg6e/U/FdpWV2R4xJYuzUIW5 CW8ArFLm28yRYSNvfNiylJQrwK9dZSUSWTGD4mUlU4Nt3/Sil5nAsvdZ1i530lNZ PUWuxlqO47sa2bUamvP/Tua6WOxhF0boNMaX6d4iPuf/X+S8yhOBbl/uvYAFD7Di ZKKbdA8yjzf46mLVeZy0UFytkf61InDDlkGIiXaQA+RNCqu4Tm8mGI6vEh0C3EMt SwcmG+ooxWJTfdIF1lTDHcGrHho8zklbBgQJWWD0e+NNagEMcHg= =hLr9 -----END PGP SIGNATURE----- --R5DOk3Sae8nJZlGU-- From nobody Sat Aug 28 03:37:47 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1B3AD1788FAD for ; Sat, 28 Aug 2021 03:38:06 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "vtr.rulingia.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GxMjn0Ghrz4wZ5 for ; Sat, 28 Aug 2021 03:38:04 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (2001-44b8-31fc-0d00-b87b-26bc-93d4-d5e3.static.ipv6.internode.on.net [IPv6:2001:44b8:31fc:d00:b87b:26bc:93d4:d5e3]) by vtr.rulingia.com (8.16.1/8.15.2) with ESMTPS id 17S3btPY084958 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 28 Aug 2021 13:38:00 +1000 (AEST) (envelope-from peter@rulingia.com) DKIM-Filter: OpenDKIM Filter v2.10.3 vtr.rulingia.com 17S3btPY084958 X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.16.1/8.16.1) with ESMTPS id 17S3bl4j093736 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 28 Aug 2021 13:37:47 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.16.1/8.16.1/Submit) id 17S3bll7093735; Sat, 28 Aug 2021 13:37:47 +1000 (AEST) (envelope-from peter) Date: Sat, 28 Aug 2021 13:37:47 +1000 From: Peter Jeremy To: Johannes Totz Cc: freebsd-fs@freebsd.org Subject: Re: ZFS on high-latency devices Message-ID: References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="sc46ZKMZH17jPmMr" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp X-Rspamd-Queue-Id: 4GxMjn0Ghrz4wZ5 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=quarantine) header.from=rulingia.com; spf=pass (mx1.freebsd.org: domain of peter@rulingia.com designates 2001:19f0:5801:ebe:5400:1ff:fe53:30fd as permitted sender) smtp.mailfrom=peter@rulingia.com X-Spamd-Result: default: False [-5.89 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[peter]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.993]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[rulingia.com,quarantine]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --sc46ZKMZH17jPmMr Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2021-Aug-20 00:16:28 +0100, Johannes Totz wrote: >Do you have geli included in those perf tests? Any difference if you=20 >leave it out? Yes, I mentioned geli (and I also have IPSEC, which I forgot to mention). I haven't tried taking them out but dd(1) tests suggest they aren't a problem. >What's making the throughout slow? zfs issuing a bunch of small writes=20 >and then trying to read something (unrelated)? Is there just not enough=20 >data to be written to saturate the link? At least from eyeballing gstat, there are basically no reads involved in the zfs recv. The problem seems to be that writes aren't evenly spread across all the vdevs, combined with very long delays associated with flushing snapshots. I have considered instrumenting ggate{c,d} to see if I can identify any issues. >Totally random thought: there used to be a vdev cache (not sure if=20 >that's still around) that would inflate read requests to hopefully drag=20 >in more data that might be useful soon. ZFS includes that functionality itself. >Have you tried hastd? I haven't but hastd also uses GEOM_GATE so I wouldn't expect significantly different behaviour. --=20 Peter Jeremy --sc46ZKMZH17jPmMr Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAmEpr4VfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzSVew//fZTheVP8ILPkpNBjJN7E5qEn3UPZFydrf0Og2oF/vHf9KYcWgM1f9w3m c27H31kI1I0025/XXBE8fAv4XsjvhfaNI3bIzq4M9fNMMErPkGAqTd/396GJIoNW LYcqpjOGGDWmZWtB1WwVOwLYsPPjOGpbt8cQ6YDf1USGWN0U9KmBzfBryZvGluaH CVDQlxXQSr+mWfKE0dGhgfYPK5pwxv0WJRPTMLOnnyf2BeoLl5OxCfdTaO9mpB4v 5xoJCfHKh3udX9MJXtOaKboSUBw9t5pQhcluqPwoYoQk2c0kyawxUITfBGZqejGU 9BU3yD/l6m+8Mln7P8vE9oPhYnpE7ikZ3tW9YT0dmlYFjrjG0gQKysWc6G2RFJ7Q 2iRbOghT/Cc5wu59sNj/enfsCZ3E6apk7RhDXEzd8bNgwadD+vC3SF58lNyz0ItM zRxvjR/1BGMeIFG/ksMDmPLnvLhCD6P/O+OJ2oYWiu2FYROFyUBkBPT7+OS9qgJg 59RLb8iT4ITixf3XF8mHHXfyl+OMaJSh/dSkOnRNPep84CiZdA1ncpdxtH/dDuOr tpg4S1qs1AzE1zJB2+rpmc+YpJ9ueRZkqu2btcrL6twrUfTa0PWBu/yOhCsbq7oD B43grMzLA/CJS7A4v9SRbsdqUsN9B7utXqKYgjuFFad2BkmRAXg= =IEmC -----END PGP SIGNATURE----- --sc46ZKMZH17jPmMr-- From nobody Sat Aug 28 18:53:19 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DF95A17ABEBC for ; Sat, 28 Aug 2021 18:53:28 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "DFN-Verein Global Issuing CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gxm1z6Pt7z3s4t for ; Sat, 28 Aug 2021 18:53:27 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from [192.168.111.102] (p2e513846.dip0.t-ipconnect.de [46.81.56.70]) (authenticated bits=0) by unimail.uni-dortmund.de (8.17.1/8.17.1) with ESMTPSA id 17SIrJJG016985 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Sat, 28 Aug 2021 20:53:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1630176800; bh=wviWJVRCqBclXtBTm3AguuZKWx2IbiYbNuTrfrYVuW0=; h=To:Cc:References:From:Subject:Date:In-Reply-To; b=fjFBKA7CK1gHUal7b9yhgwT/6EGXeHbMXAy3vY+Tx63IVDDRUBharKuglO6evMI4n 5AzKCWGVSAGb/PuVQFBiVBgJ5dc/maU68L6FGTQ/e4A82w8PthYCDHOPvpZCKnK1MN 2HqraW5NQapd2XX/aJVGaZ5PC2MaaKicS1lSsAFk= To: Konstantin Belousov Cc: freebsd-fs , Horst Schirmeier References: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> From: Alexander Lochmann Subject: Re: Various unprotected accesses to buf and vnode Message-ID: <380bdcc8-bede-2a64-8e5e-031552231d82@tu-dortmund.de> Date: Sat, 28 Aug 2021 20:53:19 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: de-DE-1901 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4Gxm1z6Pt7z3s4t X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tu-dortmund.de header.s=unimail header.b=fjFBKA7C; dmarc=none; spf=pass (mx1.freebsd.org: domain of alexander.lochmann@tu-dortmund.de designates 129.217.128.51 as permitted sender) smtp.mailfrom=alexander.lochmann@tu-dortmund.de X-Spamd-Result: default: False [-5.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[129.217.128.51:from]; R_SPF_ALLOW(-0.20)[+ip4:129.217.128.0/24]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[129.217.128.51:from]; DKIM_TRACE(0.00)[tu-dortmund.de:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:129.217.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[46.81.56.70:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[tu-dortmund.de:s=unimail]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tu-dortmund.de]; DWL_DNSWL_LOW(-1.00)[tu-dortmund.de:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 27.08.21 20:40, Konstantin Belousov wrote: >> - Read of buf.b_blkno in cluster_write(): >> According to the documentation b_lock is needed. >> Is b_blkno maybe a read-only element of struct buf? > No, b_blkno is not read-only, it is the disk block number for the block, > as opposed to b_lbklno which is logical file block number. The buffer > is instantiated with b_blkno == b_lblkno, and when the buffer is mapped > to the real disk block, b_blkno is updated. > > Could you show me the backtrace of the situation where cluster_write() > is called with unlocked buffer? > >> If you don't mind, you can find them here: https://thasos.cs.tu-dortmund.de/bugs/ctx-buf-b_blkno-r-cex.html (Pls ignore the line 'Hypothesis 1: ....'.) >> - Write to vnode.v_bufobj.bo_object: >> https://github.com/freebsd/freebsd-src/blob/main/sys/vm/vnode_pager.c#L291 >> According to the documentation, '[...] the vnode lock which embeds the >> bufobj' is needed. However, interlock is taken in line 276. >> Is the interlock equivalent to the vnode lock? >> (I assume 'the vnode lock' refers to vnode.v_lock.) > vnode_pager_alloc() must be called with the vnode exclusively locked. > This is asserted at the beginning of the function. > Yeah, I see that check: ASSERT_VOP_LOCKED(vp, "vnode_pager_alloc");. However, our data says otherwise: According to our trace, the shared lock is taken. You may have a look at https://thasos.cs.tu-dortmund.de/bugs/ctx-vnode-v_bufobj.bo_object-w-cex.html. 'EMBSAME(vnode.v_lock[r])' refers to the shared vnode lock. A click on each lock description, e.g., EMBSAME(vnode.v_lock[r]), leads to the code where it was taken. (Pls ignore the line 'Hypothesis 1: ....'.) >> >> - Is buf.b_bufobj a read-only element? > You should scope the question. > > While buffer is owned by a vnode, b_bufobj is constant. Since buffers are > type-stable, they migrate between vnodes as cache finds it required to > reclaim and reuse. There, b_bufobj is changed. > I'm referring to getdirtybuf(): msleep(&bp->b_xflags, BO_LOCKPTR(bp->b_bufobj),PRIBIO | PDROP, "getbuf", 0); -- Technische Universität Dortmund Alexander Lochmann PGP key: 0xBC3EF6FD Otto-Hahn-Str. 16 phone: +49.231.7556141 D-44227 Dortmund fax: +49.231.7556116 http://ess.cs.tu-dortmund.de/Staff/al From nobody Sat Aug 28 20:50:03 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1BB35178DC10 for ; Sat, 28 Aug 2021 20:50:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gxpch6C1yz4txb for ; Sat, 28 Aug 2021 20:50:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 17SKo3Ix010751 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 28 Aug 2021 23:50:06 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 17SKo3Ix010751 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 17SKo3vY010748; Sat, 28 Aug 2021 23:50:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 28 Aug 2021 23:50:03 +0300 From: Konstantin Belousov To: Alexander Lochmann Cc: freebsd-fs , Horst Schirmeier Subject: Re: Various unprotected accesses to buf and vnode Message-ID: References: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> <380bdcc8-bede-2a64-8e5e-031552231d82@tu-dortmund.de> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <380bdcc8-bede-2a64-8e5e-031552231d82@tu-dortmund.de> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.5 X-Spam-Checker-Version: SpamAssassin 3.4.5 (2021-03-20) on tom.home X-Rspamd-Queue-Id: 4Gxpch6C1yz4txb X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Sat, Aug 28, 2021 at 08:53:19PM +0200, Alexander Lochmann wrote: > On 27.08.21 20:40, Konstantin Belousov wrote: > > > - Read of buf.b_blkno in cluster_write(): > > > According to the documentation b_lock is needed. > > > Is b_blkno maybe a read-only element of struct buf? > > No, b_blkno is not read-only, it is the disk block number for the block, > > as opposed to b_lbklno which is logical file block number. The buffer > > is instantiated with b_blkno == b_lblkno, and when the buffer is mapped > > to the real disk block, b_blkno is updated. > > > > Could you show me the backtrace of the situation where cluster_write() > > is called with unlocked buffer? > > > > > > If you don't mind, you can find them here: > https://thasos.cs.tu-dortmund.de/bugs/ctx-buf-b_blkno-r-cex.html I see some banner "Counterexamples", two buttons, and then a blank space. Both under Firefox and Chrome under FreeBSD. I went as far as to ask and see what happens on Chrome under Windows, it looks the same. > (Pls ignore the line 'Hypothesis 1: ....'.) > > > - Write to vnode.v_bufobj.bo_object: > > > https://github.com/freebsd/freebsd-src/blob/main/sys/vm/vnode_pager.c#L291 > > > According to the documentation, '[...] the vnode lock which embeds the > > > bufobj' is needed. However, interlock is taken in line 276. > > > Is the interlock equivalent to the vnode lock? > > > (I assume 'the vnode lock' refers to vnode.v_lock.) > > vnode_pager_alloc() must be called with the vnode exclusively locked. > > This is asserted at the beginning of the function. > > > Yeah, I see that check: ASSERT_VOP_LOCKED(vp, "vnode_pager_alloc");. > However, our data says otherwise: According to our trace, the shared lock is > taken. > You may have a look at https://thasos.cs.tu-dortmund.de/bugs/ctx-vnode-v_bufobj.bo_object-w-cex.html. > 'EMBSAME(vnode.v_lock[r])' refers to the shared vnode lock. > A click on each lock description, e.g., EMBSAME(vnode.v_lock[r]), leads to > the code where it was taken. > (Pls ignore the line 'Hypothesis 1: ....'.) > > > > > > - Is buf.b_bufobj a read-only element? > > You should scope the question. > > > > While buffer is owned by a vnode, b_bufobj is constant. Since buffers are > > type-stable, they migrate between vnodes as cache finds it required to > > reclaim and reuse. There, b_bufobj is changed. > > > I'm referring to getdirtybuf(): msleep(&bp->b_xflags, > BO_LOCKPTR(bp->b_bufobj),PRIBIO | PDROP, "getbuf", 0); And? The msleep() just uses the address as the sleep chain id. The getdirtybuf() function fails if it really slept, thus dropping bo lock and allowing the buffer identity to change (Really not identity, because vnode is locked, but allowing the buffer to be written out and moved to clean queue). From nobody Sat Aug 28 21:16:15 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AFCBD17A14B8 for ; Sat, 28 Aug 2021 21:16:17 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "DFN-Verein Global Issuing CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GxqBn3cNWz3J1F for ; Sat, 28 Aug 2021 21:16:17 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from [192.168.111.102] (p2e513846.dip0.t-ipconnect.de [46.81.56.70]) (authenticated bits=0) by unimail.uni-dortmund.de (8.17.1/8.17.1) with ESMTPSA id 17SLGFFQ028071 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Sat, 28 Aug 2021 23:16:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1630185376; bh=hyxjyZaLfeESpLqYxFiz4i6Jzyav2MuR9Q0MdW2JNIU=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=nc7l796vqT7MBWS+UqtbhsMhBZfFBr81LK3L/VQcsONXWHqDKwqZ+Pzf0KInRp5LN 76eLAgPw2TXSxU72UpWpMayJLrhLYQYR6r4uomfHHWpuW1nNdLrmMe3VX+BW64M3Nn kCdRldMy5Sbw7Kx4uwAAkJW3Ja8+NLgehfL1H9fI= Subject: Re: Various unprotected accesses to buf and vnode To: Konstantin Belousov Cc: freebsd-fs , Horst Schirmeier References: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> <380bdcc8-bede-2a64-8e5e-031552231d82@tu-dortmund.de> From: Alexander Lochmann Message-ID: <46649402-d28a-6f81-f0a8-39180b681f4c@tu-dortmund.de> Date: Sat, 28 Aug 2021 23:16:15 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: de-DE-1901 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4GxqBn3cNWz3J1F X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 28.08.21 22:50, Konstantin Belousov wrote: > On Sat, Aug 28, 2021 at 08:53:19PM +0200, Alexander Lochmann wrote: >> On 27.08.21 20:40, Konstantin Belousov wrote: >>>> - Read of buf.b_blkno in cluster_write(): >>>> According to the documentation b_lock is needed. >>>> Is b_blkno maybe a read-only element of struct buf? >>> No, b_blkno is not read-only, it is the disk block number for the block, >>> as opposed to b_lbklno which is logical file block number. The buffer >>> is instantiated with b_blkno == b_lblkno, and when the buffer is mapped >>> to the real disk block, b_blkno is updated. >>> >>> Could you show me the backtrace of the situation where cluster_write() >>> is called with unlocked buffer? >>> >>>> >> If you don't mind, you can find them here: >> https://thasos.cs.tu-dortmund.de/bugs/ctx-buf-b_blkno-r-cex.html > I see some banner "Counterexamples", two buttons, and then a blank space. > Both under Firefox and Chrome under FreeBSD. > > I went as far as to ask and see what happens on Chrome under Windows, it > looks the same. Oh, I'm sorry. Pls click on "Show Member List", and then select an entry. For both cases, there should be only one entry. (JavaScript must be enabled for this to work.) > >> (Pls ignore the line 'Hypothesis 1: ....'.) >>>> - Write to vnode.v_bufobj.bo_object: >>>> https://github.com/freebsd/freebsd-src/blob/main/sys/vm/vnode_pager.c#L291 >>>> According to the documentation, '[...] the vnode lock which embeds the >>>> bufobj' is needed. However, interlock is taken in line 276. >>>> Is the interlock equivalent to the vnode lock? >>>> (I assume 'the vnode lock' refers to vnode.v_lock.) >>> vnode_pager_alloc() must be called with the vnode exclusively locked. >>> This is asserted at the beginning of the function. >>> >> Yeah, I see that check: ASSERT_VOP_LOCKED(vp, "vnode_pager_alloc");. >> However, our data says otherwise: According to our trace, the shared lock is >> taken. >> You may have a look at https://thasos.cs.tu-dortmund.de/bugs/ctx-vnode-v_bufobj.bo_object-w-cex.html. >> 'EMBSAME(vnode.v_lock[r])' refers to the shared vnode lock. >> A click on each lock description, e.g., EMBSAME(vnode.v_lock[r]), leads to >> the code where it was taken. >> (Pls ignore the line 'Hypothesis 1: ....'.) >>>> >>>> - Is buf.b_bufobj a read-only element? >>> You should scope the question. >>> >>> While buffer is owned by a vnode, b_bufobj is constant. Since buffers are >>> type-stable, they migrate between vnodes as cache finds it required to >>> reclaim and reuse. There, b_bufobj is changed. >>> >> I'm referring to getdirtybuf(): msleep(&bp->b_xflags, >> BO_LOCKPTR(bp->b_bufobj),PRIBIO | PDROP, "getbuf", 0); > And? The msleep() just uses the address as the sleep chain id. > The getdirtybuf() function fails if it really slept, thus dropping bo lock > and allowing the buffer identity to change (Really not identity, because > vnode is locked, but allowing the buffer to be written out and moved to > clean queue). > -- Technische Universität Dortmund Alexander Lochmann PGP key: 0xBC3EF6FD Otto-Hahn-Str. 16 phone: +49.231.7556141 D-44227 Dortmund fax: +49.231.7556116 http://ess.cs.tu-dortmund.de/Staff/al From nobody Sat Aug 28 22:29:44 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1A4CE17A0492 for ; Sat, 28 Aug 2021 22:29:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gxrqj5Y2Wz3tCg for ; Sat, 28 Aug 2021 22:29:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 17SMTiqa035030 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 29 Aug 2021 01:29:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 17SMTiqa035030 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 17SMTiat035029; Sun, 29 Aug 2021 01:29:44 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 29 Aug 2021 01:29:44 +0300 From: Konstantin Belousov To: Alexander Lochmann Cc: freebsd-fs , Horst Schirmeier Subject: Re: Various unprotected accesses to buf and vnode Message-ID: References: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> <380bdcc8-bede-2a64-8e5e-031552231d82@tu-dortmund.de> <46649402-d28a-6f81-f0a8-39180b681f4c@tu-dortmund.de> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46649402-d28a-6f81-f0a8-39180b681f4c@tu-dortmund.de> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.5 X-Spam-Checker-Version: SpamAssassin 3.4.5 (2021-03-20) on tom.home X-Rspamd-Queue-Id: 4Gxrqj5Y2Wz3tCg X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Sat, Aug 28, 2021 at 11:16:15PM +0200, Alexander Lochmann wrote: > > > On 28.08.21 22:50, Konstantin Belousov wrote: > > On Sat, Aug 28, 2021 at 08:53:19PM +0200, Alexander Lochmann wrote: > > > On 27.08.21 20:40, Konstantin Belousov wrote: > > > > > - Read of buf.b_blkno in cluster_write(): > > > > > According to the documentation b_lock is needed. > > > > > Is b_blkno maybe a read-only element of struct buf? > > > > No, b_blkno is not read-only, it is the disk block number for the block, > > > > as opposed to b_lbklno which is logical file block number. The buffer > > > > is instantiated with b_blkno == b_lblkno, and when the buffer is mapped > > > > to the real disk block, b_blkno is updated. > > > > > > > > Could you show me the backtrace of the situation where cluster_write() > > > > is called with unlocked buffer? > > > > > > > > > > > > If you don't mind, you can find them here: > > > https://thasos.cs.tu-dortmund.de/bugs/ctx-buf-b_blkno-r-cex.html > > I see some banner "Counterexamples", two buttons, and then a blank space. > > Both under Firefox and Chrome under FreeBSD. > > > > I went as far as to ask and see what happens on Chrome under Windows, it > > looks the same. > Oh, I'm sorry. > Pls click on "Show Member List", and then select an entry. For both cases, > there should be only one entry. > (JavaScript must be enabled for this to work.) Ok, I see some call sequences (?), but again all of them are ffs_write() (one is ext2_write) calling into cluster_write(). There the buffer lock is owned. Show me the specific call sequence where it is not. > > > > > (Pls ignore the line 'Hypothesis 1: ....'.) > > > > > - Write to vnode.v_bufobj.bo_object: > > > > > https://github.com/freebsd/freebsd-src/blob/main/sys/vm/vnode_pager.c#L291 > > > > > According to the documentation, '[...] the vnode lock which embeds the > > > > > bufobj' is needed. However, interlock is taken in line 276. > > > > > Is the interlock equivalent to the vnode lock? > > > > > (I assume 'the vnode lock' refers to vnode.v_lock.) > > > > vnode_pager_alloc() must be called with the vnode exclusively locked. > > > > This is asserted at the beginning of the function. > > > > > > > Yeah, I see that check: ASSERT_VOP_LOCKED(vp, "vnode_pager_alloc");. > > > However, our data says otherwise: According to our trace, the shared lock is > > > taken. > > > You may have a look at https://thasos.cs.tu-dortmund.de/bugs/ctx-vnode-v_bufobj.bo_object-w-cex.html. > > > 'EMBSAME(vnode.v_lock[r])' refers to the shared vnode lock. > > > A click on each lock description, e.g., EMBSAME(vnode.v_lock[r]), leads to > > > the code where it was taken. > > > (Pls ignore the line 'Hypothesis 1: ....'.) Ah, yes, the calls from lookup and open would be with the shared lock. Still, we lock the vnode interlock to avoid double-allocating the v_object object, so it is fine. Some mode of the vnode lock is required nonetheless, because otherwise we might miss reclaim which guarantees that v_object is freed. > > > > > > > > > > - Is buf.b_bufobj a read-only element? > > > > You should scope the question. > > > > > > > > While buffer is owned by a vnode, b_bufobj is constant. Since buffers are > > > > type-stable, they migrate between vnodes as cache finds it required to > > > > reclaim and reuse. There, b_bufobj is changed. > > > > > > > I'm referring to getdirtybuf(): msleep(&bp->b_xflags, > > > BO_LOCKPTR(bp->b_bufobj),PRIBIO | PDROP, "getbuf", 0); > > And? The msleep() just uses the address as the sleep chain id. > > The getdirtybuf() function fails if it really slept, thus dropping bo lock > > and allowing the buffer identity to change (Really not identity, because > > vnode is locked, but allowing the buffer to be written out and moved to > > clean queue). From nobody Sun Aug 29 21:00:39 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2C5361797EB6 for ; Sun, 29 Aug 2021 21:00:40 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GyQpH4wXyz4W4S for ; Sun, 29 Aug 2021 21:00:39 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6C3121174C for ; Sun, 29 Aug 2021 21:00:39 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 17TL0dYj094683 for ; Sun, 29 Aug 2021 21:00:39 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 17TL0dqc094682 for fs@FreeBSD.org; Sun, 29 Aug 2021 21:00:39 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202108292100.17TL0dqc094682@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 29 Aug 2021 21:00:39 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16302708395.A42fd.93640" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: Y --16302708395.A42fd.93640 Date: Sun, 29 Aug 2021 21:00:39 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 211491 | System hangs after "Uptime" on reboot with ZFS Open | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 240831 | zfs: Panic during snapshot on 12.1-STABLE r352648 Open | 243973 | [zfs] rollback segmentation fault Open | 244656 | zfs: resilver doesn't provide enough information Open | 244692 | gjournal: Does not support TRIM Open | 244899 | [PATCH] zfs: xattr on a symlink target > 136 caus Open | 251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W 9 problems total for which you should take action. --16302708395.A42fd.93640--