From nobody Sun Dec 24 06:04:21 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SyVqB36Ztz546dm for ; Sun, 24 Dec 2023 06:04:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SyVq91RgCz4PSS for ; Sun, 24 Dec 2023 06:04:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703397861; a=rsa-sha256; cv=none; b=xUPgVSpCQUJkwHsNJ7ctKWTClMxiG6xXZHiIMYhjRZeYr9pe2DkXgVmif0xQJ2IwhfIWJX JOkqvJG/D+IvpnmJCcowbCbdHUd+oibwKbnfxaVreKDXAl865B/771eHU4MHFJXMPetczp fMkspzyQCQ8THKZ7wPXWUSWN755Q2o6AKOiZUY92lOfIRPF24aKw1NQHIxvqeBkOD57uws CcNmN2B+a6IBKVtgXVAOvZnXIiA3d7KaND9YDl9V9NimexBiTpOpo783sUVm57JL513Rgn E2EJJE+LMhrDL69tXR5YSJ0SyCwhwGY0g+dfhWWkrZrBZpAb+jMlpcSuNk7lqA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703397861; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HoqvpQgVux7rV5OJbisJEn+gt2MXMx7BSl6k757ZHE8=; b=Veryym2BLwog+1UkwUEYGRq1DtcoZUUzUz8JAoULxcFVwMPlwFEQ+9GDcGIw27i/kHbcqU BNVSUj+ZX1TmCwy06C8q+12LSVseU9gfS6pBuMD73bpCeKkWZI/bEm4pBd0kkNpEY6rvfi 6fn+7hhIs+4qwD+L5uztEBDmSkmaTsIfYvkr8WcJHk7IzI+opbzcnhh98rK7H875urCJf7 A7Nvr+aV7HD3+A7UYcNduljhG9f5zIsdx6TfoOSavkc8U3pb0wTMgUcrig/gGLq/duMfKR J4GQWMiheOXrqiX6V3zAC9Jjf+ajhiL/IpB7vD/0G1JDZ0xvk5BfOoTB209Bjw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SyVq85Tvfz2K5 for ; Sun, 24 Dec 2023 06:04:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BO64KZb032173 for ; Sun, 24 Dec 2023 06:04:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BO64KAI032172 for fs@FreeBSD.org; Sun, 24 Dec 2023 06:04:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275905] nfs client: mount becomes unresponsive Date: Sun, 24 Dec 2023 06:04:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 11:16:24 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sydlr0qKKz54TFD for ; Sun, 24 Dec 2023 11:16:56 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sydlq2bKXz3Qvj for ; Sun, 24 Dec 2023 11:16:55 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=rK1zeYfL; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=TqgCbCAu; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.25 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 8ED0F5C0126 for ; Sun, 24 Dec 2023 06:16:54 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Sun, 24 Dec 2023 06:16:54 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1703416614; x=1703503014; bh=3EnJSCx2Vg ZUQh12yh+2aIXkq0bvwB1q7F6lrtx2rKs=; b=rK1zeYfLlBrvLJ5Sojaw3bw2Vs wvmPsZcD2bUo/uFHVvwHV3OqRgXNPj1hBOu13npJBObho+9dVJgGXKFt9inFHFMh wgjED8CFg19QohAe3pa0oZHlTDcI9pqpuml8v5m220dXQ6ekv/lWuNbtpwLMd6xv RjAjSCTyfFxJEb+e7k8T3MY/bt9p9fefz2U6QncXSuEI73A13aSiFE8x6hC6W5Es X5YQXwFfCqN7GbYvaVr/ywn1JhB/GHh8IG90MHl7qT89FlDzTz+zYQuvP2iiod2y oeV+F246nS7xtp9KpTqcuJsPSOxtD9W0n3cN3mm4VbP17tF5qZHJa2PaUQLw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1703416614; x=1703503014; bh=3EnJSCx2VgZUQh12yh+2aIXkq0bv wB1q7F6lrtx2rKs=; b=TqgCbCAuAmagsgDvES5WTfTAFNBdwbesRlbAd0O+DZS6 puVZgV7lxi/H4O6F6r2x1bOtTX/DnnBa1enR3ht0rGkCnYm0i0tij+OC/FAPO7TI fajzCIKRrpltr2W41NSnq6Z4H+zDaMTGYZGh07kGs9Hn88pd8WwOV2B3NY4Fr9SQ 7E1DBRhvAo/TeIxO9+dmsY/HIq5KufjS7SPwNmasyKZZIII3QM7s4gNVSLpb0EDs LUi5YYJlTZ4DtclZaZM25nmJr7rtEVLXEf0LSWPBpVVzJUrbKCcnSeXl93Nn5Rw7 MaPuGxFeVAcYyJ4EZenJkSygIr6XxnAS/4S6/fNmRA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvddvuddgvdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeeitedvueehtdehtddvhfeuhfevhedvieelvdeiffehveelheegfedule ejudekvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm pehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 5757C2A2008B; Sun, 24 Dec 2023 06:16:54 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-1364-ga51d5fd3b7-fm-20231219.001-ga51d5fd3 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Message-Id: <0ac6ceb8-cdf4-49f7-99c3-026a56527fd3@app.fastmail.com> In-Reply-To: References: <2dcc21bb-f446-41fa-871b-2c37cdf9afe0@app.fastmail.com> Date: Sun, 24 Dec 2023 11:16:24 +0000 From: void To: freebsd-fs Subject: Re: measuring swap partition speed Content-Type: text/plain X-Spamd-Result: default: False [-4.13 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.943]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.25:from]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.25:from]; XM_UA_NO_VERSION(0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[f-m.fm]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4Sydlq2bKXz3Qvj X-Spamd-Bar: ---- On Sat, 23 Dec 2023, at 21:19, Warner Losh wrote: > 5s+ of latency on the average, max latency of 12s! Woof. No wonder you > hate life. LOL! Thing is, it's not all the time. But when it happens, it tends to stay happening, unless something like swapflush runs or if I manually swapoff then on. From nobody Sun Dec 24 11:50:07 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SyfVB5TmQz54W8R for ; Sun, 24 Dec 2023 11:50:10 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SyfV96sv7z3VWW for ; Sun, 24 Dec 2023 11:50:09 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=L5p2SzWr; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=vsTolTrm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.25 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id D3C6A5C0101 for ; Sun, 24 Dec 2023 06:50:09 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Sun, 24 Dec 2023 06:50:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1703418609; x=1703505009; bh=bQzFfhdAIi 5zqqif0tJGJWpFEd9zKW9YS0bAXseKdb0=; b=L5p2SzWrHaxy8T6lPw8xY7THcy oV0DJGw9l2XWPKgq7BF4raSRu3NF84Vo1kqJ3Wkb48g+/QN5ntIE8yBswTkqab8V soAI7NoIWL5EbBg4T4VapbaXVsU4I5hp7UVgICvtFlKj9WHjZBBjhqJjFmyV17si 1qKdW732XbXGFubWv7Nd2Z8e980OC4W1v9LmW26hrZBPXbI2qD5zfaUau6MSDNuU e2F5YTe/Tjt+oBeYo8vKA/cIJJ5i4MdIhmcFb/qWB+7a7X7dqU0tWlOnlRDvV9/Q h8/wY21NQ4g68UfiFXqu0b/q64hPeTKmQvM0YfLckWrGEfQlY5fck7u5z00w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1703418609; x=1703505009; bh=bQzFfhdAIi5zqqif0tJGJWpFEd9z KW9YS0bAXseKdb0=; b=vsTolTrmFALO+/Cx9HGOjgEGmUZabI5b0Jnv1hKo0wnO +55dN/zqApCFTQj4oZFuoYSPFWfSNUM0Ua1Pxy1J19b6uhgWf/RAuPZ2gQ7s1L5O mQWZfZR0VyRzFNwfTkXxFLt74aryh4qp/M+0aQ/8kQ19RnC6Cca3M2dRkz5Csnhv STnYU4HfIcszA8v6QmFaA3osV11DOqJQzkpnA7hxZxlS3omrcz+lHD6yKlxa1P9N dMN5uOTOG66biFGIvBthsustmJ1Km+2xUNBk1z7x/3tQc2gFCssIH/PdTAMtLdze s6RvjJimQu3Yg0a1ICTbXu6ekmncdxZ55HKz++v+tw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvddvuddgfeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepkeeluddvlefhieelfefggffhffektdehleelgfdugfdvgeekjeejuddthe ehgfeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sun, 24 Dec 2023 06:50:09 -0500 (EST) Date: Sun, 24 Dec 2023 11:50:07 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: measuring swap partition speed Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-3.68 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.978]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25:c]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.25:from]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.25:from]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SyfV96sv7z3VWW X-Spamd-Bar: --- On Sat, Dec 23, 2023 at 11:55:43AM -0800, Mark Millard wrote: >That last indicates another potentially significant difference >from my example benchmarks done for comparison to yours: > >GEOM_ELI: Device da0p3.eli created. >GEOM_ELI: Encryption: AES-XTS 256 >GEOM_ELI: Crypto: software > >That likely contributes to slowing down I/O for da0p3 and >possibly to leaving less time for I/O to other partitions >when da0p3 is in use. I think *some* slowdown is reasonable. What's happening here, though, amounts to blocking. 300-1000Kb/s against 16MB. zfs filesystem reads/writes are slower to an extent but it's not by a massive amount, like it is with swap, (and the swap is unencrypted) >If you had the resources to test avoiding the encrypted >partition for your type of media, that might prove >interesting. I think i did early on but I can do so again. I'll delete some swap partitions and format as ufs. >I've never had occasion to want to use any encrypted >partitions for my FreeBSD activity. A couple of reasons for this particular instance: 1. the hardware is portable and visible and would be easily stolen by someone of a mind to 2. the system performs authentication of which some parts are in clear txt >I'm unsure of the GEOM_ELI processing-time contributions >vs. "gstat -spod" reporting. yeah I've long wanted to measure the performance penalty for this, given that theres no aesni or equivalent capability available in the hardware. -- From nobody Sun Dec 24 12:28:05 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SygL13chBz54YfQ for ; Sun, 24 Dec 2023 12:28:09 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SygL05sBhz3XPd for ; Sun, 24 Dec 2023 12:28:08 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=nZW0LapO; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=ICbr3qjj; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.25 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 1A8B55C010C for ; Sun, 24 Dec 2023 07:28:08 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Sun, 24 Dec 2023 07:28:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1703420888; x=1703507288; bh=HEb6vdONQB y+wmgvCyu3lhx48tw7u7K28ou3l0PrzWM=; b=nZW0LapOdmCMz/7uwkbgsVOAFy eLcB3pECHTrlMiPWtojnkgWoo9xR7vkrC9qyiT4mcQy/TCdvy7xOroXJdoLvcUBc wD/rT41tsj4cw8olWTPoTYdbbLAVaUbrxTGhsmUVyl/sOh1VGmp0aBKdjg3qeeic 5waURmksAjwNCQdYAWpmF2t2ns+L36pRGqO3t4U4lIBUrTt2pknNR5n8dB7TpzTy yO3XJZhzzt0UkGw0QCGbL/j+RM+zJSLbeOSM9FpOi6kaRh9seGKa9ENggDQ+UzPP RcQ46KIQq9LlSUwBUvSDYjCkPARDCoHvX+XMQFPhCVU75Ng1o6faBfhkfGuw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1703420888; x=1703507288; bh=HEb6vdONQBy+wmgvCyu3lhx48tw7 u7K28ou3l0PrzWM=; b=ICbr3qjj9KuBB/j5MqRcjgFgQwSZlb8Zv/KWPGxKNEsR 1pGF+2B8nHjpRJyCskjnrvDTq1jwrHxlWi7NNE8swMaIieoL5Dv1Cd3ZA3uVoBqO Cnk7Ws1PlDZ1YO+/FhPr/4AiW8sLrDtqoYlI6leEpIjurppDeTLtxtM0M2/fy3Bx LxKzDGidKrriHK8T/wVMRYur5ajX01G03Nwtb6ramN6dcZ9mItcc8xLmDRVPDRE0 RcziYIB+d9+m/JggcFZ0a5noCq0KYVJHlzUpjw4r0Xr3RxEwIl8+IsE0x7VyCkx1 kDXRqKBBy2gTZKIZRhkvfJURxzxIrMpxYFN5YaEZew== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvddvuddggedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnheptdfhheeuteejudffkefhtdfhfeekgedtvdeiteevgfejtdfgfeffhfeuie eltdeinecuffhomhgrihhnpehfrhgvvggsshgurdhorhhgnecuvehluhhsthgvrhfuihii vgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sun, 24 Dec 2023 07:28:07 -0500 (EST) Date: Sun, 24 Dec 2023 12:28:05 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: measuring swap partition speed Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-3.70 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.25:from]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.25:from]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SygL05sBhz3XPd X-Spamd-Bar: --- On Sat, Dec 23, 2023 at 11:55:43AM -0800, Mark Millard wrote: >If you had the resources to test avoiding the encrypted >partition for your type of media, that might prove >interesting. The problem remains even with UFS. It remains after any non-swap i/o load stops, even when swap is not otherwise in use. I took /dev/da0p5, formatted it with newfs -U and put it at /mnt. then root@pi4:/mnt# date && git clone https://git.freebsd.org/ports.git Sun Dec 24 12:06:50 GMT 2023 Cloning into 'ports'... remote: Enumerating objects: 6044522, done. remote: Counting objects: 100% (941/941), done. remote: Compressing objects: 100% (125/125), done. remote: Total 6044522 (delta 923), reused 821 (delta 816), pack-reused 6043581 Receiving objects: 100% (6044522/6044522), 1.16 GiB | 6.94 MiB/s, done. ^Csolving deltas: 2% (101172/3649394) date && dd if=/dev/urandom of=/dev/da0p6 bs=8k conv=sync status=progress Sun Dec 24 12:06:52 GMT 2023 ^C203841536 bytes (204 MB, 194 MiB) transferred 205.012s, 994 kB/s 25009+0 records in 25008+0 records out 204865536 bytes transferred in 205.914885 secs (994904 bytes/sec) cancelled both processes, checked swap # date && swapinfo -h Sun Dec 24 12:11:56 GMT 2023 Device Size Used Avail Capacity /dev/da0p2 2.0G 0B 2.0G 0% /dev/da0p4 2.0G 0B 2.0G 0% /dev/da0p6 2.0G 0B 2.0G 0% /dev/da0p8 2.0G 0B 2.0G 0% Total 8.0G 0B 8.0G 0% while /dev/da0p6 is available as swap to the system # date && dd if=/dev/urandom of=/dev/da0p6 bs=8k conv=sync status=progress Sun Dec 24 12:10:38 GMT 2023 ^C78790656 bytes (79 MB, 75 MiB) transferred 96.008s, 821 kB/s 9767+0 records in 9766+0 records out 80003072 bytes transferred in 96.986518 secs (824889 bytes/sec) # swapoff /dev/da0p6 # date && dd if=/dev/urandom of=/dev/da0p6 bs=8k conv=sync status=progress Sun Dec 24 12:21:28 GMT 2023 dd: /dev/da0p6: end of device2031 MiB) transferred 105.007s, 20 MB/s 262145+0 records in 262144+0 records out 2147483648 bytes transferred in 105.837991 secs (20290291 bytes/sec) -- From nobody Sun Dec 24 12:38:43 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SygZD1bZBz54Z6D for ; Sun, 24 Dec 2023 12:38:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SygZD0WnSz3YNB for ; Sun, 24 Dec 2023 12:38:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703421524; a=rsa-sha256; cv=none; b=D53uPGNY8oWxa7Q0CyJALqKeX2LzvUWUbj5QaFOIUk4+IclInm6AqnAotiW64NQTzjmaXf fwQYdRdCYRSSJFNrWpfJAFiSttPlzjT7rij9h/lpcPqV8+jQg+80aOCYekqBmf+ZgkSXxU pFKlnHzoKimsBTtB9m1/ezSxu2WkS/W4SUvRUsstARq3lRGbNtbezKzF6qcAhNj6d11zvJ sHubVrdNhO7kJgO3jlJ1RE9WOfTr3BrAmcanSUJlGJIbG2PQlF36XnnxIIj6NCyXR8ytjW B71lcKvetd0VNVz6K31Wn+Og7cQSoFitz6UDk49KxRE2od4EjghboY3Mt4VdsQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703421524; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yFoYeTDXyTgKxF3XmfYemKizQyneTQJlOFjE+o2WmNY=; b=WJ/5PvCiNFr+FkTrHTNJ5WM5ah0H//YGSpRuufMKDweDUZPQo35jpcdmaSZ38uEgz21/O/ qlu13rAqj5g7J6M7gtfMimKGOTvJszq2svl23BgiATE9yc1uyO0AtA7XP14ZPdT8JSnnLG FJfID0sl5wT/7qHz6TinTH6y/+Q7BiHoZdTERRPTs4+9jZ9RIQmdVAMxQy/p1JkfkorKQQ QRstY/Om9aXkzUfj8Ttmcds2X19dw1YRi4yf5q2ZTmGN0GMuS1/sdVZqKVM4ZxyK/zafGt LFfgE6SovI6XK61FFPC8UJz8QDTWgbLkIootOLGAwZTMSovb5sda8a78oCN0QA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SygZC6bmmzDKk for ; Sun, 24 Dec 2023 12:38:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BOCch3u018708 for ; Sun, 24 Dec 2023 12:38:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BOCchSO018707 for fs@FreeBSD.org; Sun, 24 Dec 2023 12:38:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275597] Samba: smbd sometimes aborts by PANIC when 'vfs objects = cap' Date: Sun, 24 Dec 2023 12:38:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: uratan@miomio.jp X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275597 --- Comment #7 from uratan --- (In reply to Kirk McKusick from comment #6) It is OK either to close this bug report here or not because I have provided enough information for: [a] someone who may have the same problem as me. [b] someone who may try to reproduce the problem. (My role ends here, I think) - * - * - (fully off topic) By the way, you !!! Kirk McKusick san ??? One of the author of "The Design and Implementation of the 4.4BSD Operating System" ? I have the book hard-covered English edition ! (not second-hand) (but not yet read it, saving it for my fun in old age/post retirement ;-) May I use my second name as "A man commented on by Kirk McKusick" ? And..., if you know something, tell me that my 8 years ago report about setting of disk Command Queuing and CAM sort_io_queue: https://forums.freebsd.org/threads/bad-response-while-reading-long-long-fil= es.54768/ was helpful/meaningful or not..., if if if you know or someone please. (I confirmed the problem is not appeared on 14.0R) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 14:25:36 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SyjxY4zHwz54gkY for ; Sun, 24 Dec 2023 14:25:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SyjxY06GDz4HRt for ; Sun, 24 Dec 2023 14:25:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703427937; a=rsa-sha256; cv=none; b=iqKLQSK7hllDv+2Jr23tnZ0RPL59c5m8Z5F3UKAWUt2l1/FI8mq61mQuK8gtaghYhRRO3n D58eNI6j/YUqDkUOJv9eRtxW4omy6tHOpOGE9If5/bciZmrqocLQ+TGvmQqoTrC/Ln6Yxj HnuDcWSZBCR/SP/1oWoih0BkaKjroJfwfqJl5kS0nBVDgwkNT4dWrAwV/7M/qlv9wqUzhJ TOxmGJZrIf8AP2D+foS6NpVplA0FnaGKUeL7RdPVstQ5PFH1b8FrfwCyG40cw5MYeZpPcK qLXksLQgUKsRaH9nENufWWM7AQlAA5+w9s8rgg6V2au3RgT6GRinYd9JcQq+Wg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703427937; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iEVFhfY4oohe+jfw0RpnZELXNZ3fdm8hrmMRUbS2O4U=; b=ybRzD28HeG/cDX17XF9OVbLuTz15WK2iPpdrGQyGQek3e4Z/p1KIokJiD/HyOSyKMAtxhk DdUGzd9Yg+IeaZ91UOTKTnnsezm868W+0Vv5qJxEhHwYc883N4y1efgyg2M/TG82E5kTD2 KHb9PZNUpMWmFNDULGvhiH93tzQQv1+x8u8BdJwaTT/mz/rEGOi5dmu9TYjH5SnEk7EBo/ Spo+sRUr+NZ21qUt0riTS5MD7xTmSfxSQ1IHiEZKw+YHxXY5PR/wmraZ8pZQl+3VlIwyMP 8ygk8EXvwYTBwsQkz7C2WV4Be0h/332YweuDTz3sXfTz503KB1N6tqPyoNWUMw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SyjxX6Bb4zXT9 for ; Sun, 24 Dec 2023 14:25:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BOEPaJA082287 for ; Sun, 24 Dec 2023 14:25:36 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BOEPamt082286 for fs@FreeBSD.org; Sun, 24 Dec 2023 14:25:36 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275905] nfs client: mount becomes unresponsive Date: Sun, 24 Dec 2023 14:25:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: lexi.freebsd@le-fay.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 --- Comment #1 from Lexi --- this seems to be connected to Kerberos ticket expiry: the problem recurred = just now when my ticket expired. when this happens, klist shows: % klist Credentials cache: FILE:/tmp/krb5cc_10006 Principal: lexi@EDEN.LE-FAY.ORG Issued Expires Principal Dec 24 04:13:34 2023 >>>Expired<<< krbtgt/EDEN.LE-FAY.ORG@EDEN.LE-FAY.ORG Dec 24 06:58:04 2023 >>>Expired<<<=20 host/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG Dec 24 13:37:42 2023 >>>Expired<<<=20 nfs/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG Dec 24 13:37:42 2023 >>>Expired<<<=20 nfs/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG Dec 24 13:37:42 2023 >>>Expired<<<=20 nfs/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG Dec 24 13:37:42 2023 >>>Expired<<<=20 nfs/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG [... many identical entries elided...] Dec 24 13:37:42 2023 >>>Expired<<<=20 nfs/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG Dec 24 13:37:42 2023 >>>Expired<<<=20 nfs/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG Dec 24 06:58:04 2023 >>>Expired<<<=20 host/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG Dec 24 06:58:04 2023 >>>Expired<<<=20 host/hemlock.eden.le-fay.org@EDEN.LE-FAY.ORG in additional, the 'Invalid' counter in nfsstat -cE increases to 64 (it was previously 0): Rpc Info: TimedOut Invalid X Replies Retries Requests 0 64 0 0 4832814 however, the mount never recovers even after renewing the tickets. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 15:23:33 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SylDQ5Cjhz54lWd for ; Sun, 24 Dec 2023 15:23:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SylDQ3XYhz4Lky for ; Sun, 24 Dec 2023 15:23:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703431414; a=rsa-sha256; cv=none; b=YFWaLqHtqOkKBUqKdmTbn/TP5kus/FxGA7jOpbUA29L4kjuwUFpFp/B/3YODCRGAAWPAaU AuCCXVIOUOilKtpH1JioY3M0te25KqP8acitoVrWeyZqtwqjinUXqfWlzxzuP1uJzRxbhY SlC4nk4DBurYYoX9+gnoEO+gy8u1eaah6ggEQwhbLXlbpEAc7ub1tR/7nsxa9Lcmdff6zj aGqxttWjDzBugMaPXY0PmUCT5TD0XV0d62Kr0nlqKndc6AWQ5OENQbWMPkDeo5RAbmwB58 LFa7RgW0h8JfCFQmU7Qdh/cX9Ms7+12tnoduphPJafAfb6VDw4oL2/HvyGnaxw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703431414; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vEpNwExMq8UsXdNLDTC24BSShQH9fOTqkF+IjBKRjI8=; b=lZ5KBpjIMR9NZNKFU8sXlDcKmUZTSKEeM7vSPUrDf6Jp5hBvf2ufzQ69fk4LOBTPzc457T gvZZg9AKffr9JP6j4/ZT5zjHaB1d1S3WceGAgMx3CLUOJcymb0Q1e9jmhQSOg7p+Qn/G5P fsBl5tdvhFaieOn70jtDHSPnar6gitcA0AS8EaP7Ps1Li0nINOiz2XTi2UFPNPufZImF7C TPT6mr430J00RvV8ClzWBpZWqxHJRh3+t91YjouNqsyUuFqzudTQ4lCeNE/xo8474M716t FD4fI/mhkxdBHfg6R4AGQkoaq1ZYw5kJbrKDxPsJNKZaTt4Zv+7zNv9oFOJZ1w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SylDQ2crlzYVW for ; Sun, 24 Dec 2023 15:23:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BOFNYak065805 for ; Sun, 24 Dec 2023 15:23:34 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BOFNY8G065804 for fs@FreeBSD.org; Sun, 24 Dec 2023 15:23:34 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275597] Samba: smbd sometimes aborts by PANIC when 'vfs objects = cap' Date: Sun, 24 Dec 2023 15:23:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jrm@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: timur@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275597 Joseph Mingrone changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jrm@freebsd.org Assignee|fs@FreeBSD.org |timur@FreeBSD.org --- Comment #8 from Joseph Mingrone --- Thanks for the thorough description of the problem and proposed solution. I extracted your changes, created an i386-only patch for net/samba416, and added the maintainer, timur@FreeBSD.org. https://reviews.freebsd.org/D43171 I also reassigned the bug here to timur@, but I believe he's been unavailab= le for the past few months, so we might have to wait two weeks for a maintainer timeout before committing.=20=20 Ideally, we would understand why the first memcpy() is failing. Maybe ther= e is something to submit upstream. Finally, unless there is a compelling reason to use i386, you might consider using amd64 for your home server if your current hardware supports it.=20 Starting with FreeBSD 13, i386 was demoted to a tier 2 architecture, and it= 's projected to be unsupported as of FreeBSD 15.=20 https://www.freebsd.org/platforms/ --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 15:24:52 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SylFw2KXHz54lQc for ; Sun, 24 Dec 2023 15:24:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SylFw0kHdz4MSk for ; Sun, 24 Dec 2023 15:24:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703431492; a=rsa-sha256; cv=none; b=a2v8mg3qDDH0JvbksJ32N/55/KNFB8vH93nLIh+wDRHXlHWth+PVBU+YHQNbumh3wV/go1 FJS30bPVqvCLQSONzhx/f5JAbuTIDUjUaG5gv2gvHouNFY0y44OR4xRTqGCCDFVKOWOYEV Cp49r2FON1NfOqLiJlq/Q+M/rbadgR6rZ/ED774eGzpqiIeHYqFUfmMa6oADDpVVTeY2RB UqU00a67iuXjD/u0AxZMJxnbAACeB86lPxiHMuE46q5siIvuP15UyR+rhS+iJsE4shBp0w OkO22XxC3Mza/Xuba+p5spLV6eHhu8TwxlYI7cerLD7sDnAbreyJFtvC+5aYqA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703431492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S+VZobrxXqNK6mfcVfF2yrh7048NnP3VtNqKTBvsoBM=; b=bl9KwWZ7+WxSuTlsRb0uOXsmIUvbdBvi0dJ4MCxtyh6kZ5fA2D4dFHIxHk3sKkloLPOVgM 5jk/37Bxwuycb2Kq0wCNbfD7kAWe9wA0Dd3J2VhRs3UfCEN4vDW9b3zHKrTqcuoBcZycNU 4E6jrogj/8M9Mhkt8XKlXB6isJmAW9gjF1fOQP6b7lsgGSXaibhxNipo9Ha5JIp5ypKMrt +ZL9mZ9WjNTVygMrg/Hz9mciIoQOJUvGpZmSebxc2tkcoy57aX8VDXCOqf48Lf7g8U+xwA E1W7EwHG177WdQjharvW2THrkyoHRc2DgDOfnPTNeUBmYAPlQAo84B9JVI+05w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SylFv6w8szYrw for ; Sun, 24 Dec 2023 15:24:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BOFOpue066207 for ; Sun, 24 Dec 2023 15:24:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BOFOp4B066206 for fs@FreeBSD.org; Sun, 24 Dec 2023 15:24:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275905] nfs client: mount becomes unresponsive Date: Sun, 24 Dec 2023 15:24:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rmacklem@FreeBSD.org --- Comment #2 from Rick Macklem --- You haven't used the "intr" and/or "soft" mount options by any chance? These are guaranteed to break NFSv4 mounts. The renewthread is stuck in nfsv4_sequencelookup() because there are no session slots available. - This can happen for "soft" or "intr" mounts because the RPC reply does not get processed properly to release the session slot. - If the mount does not use "soft" nor "intr", then there is some other reason that the replies are not being processed. --> I'll think about this. It might be a side effect of a RPCSEC_GSS session being expired so the client cannot de-encrypt the RPC reply, or something like that. Typically, the RPCSEC_GSS session (not the same thing as a NFSv4 session, just to make the terminology confusing) expires when the TGT used to acquire it expires. It will be re-created upon the next RPC if there is a valid TGT, but that might be too late for processing of an RPC reply. (With sec=3Dkrb5p the reply is encrypted, so it cannot be processed without a valid RPCSEC_GSS session. Once all session slots are in use and no RPC reply is processed to release one, the mount is hung and the only fix is to "umount -N " or reboot the client machine. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 15:36:46 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SylWf3476z54m4h for ; Sun, 24 Dec 2023 15:36:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SylWf23cfz4NfL for ; Sun, 24 Dec 2023 15:36:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703432206; a=rsa-sha256; cv=none; b=UQxQYMJolvqHqOOAMM+a2Hq0ouwNISTIWrtP3s1/fssBB8YLeY7nes1ZJtCz3WKFKk7mcz N75gmfEV5RWPGGU+5lPm4QLCCWZKrsSJr2Y/Qv2hje2bviFJ22/YDZhu96xrQ4FtdwJ/0u bH1uIkSELNvIU5o4LYIHYNFXEFeRLq/ohoHE3loeVDQSo9xCY1qciXAF0T0NgG1whzH1G8 egPkH49ZQnC63eiAJPdOHJ4RuKTvlXjBuUFJKv7FcW3LG/xmB3/fYhC8n4FsWnfx/i8w2q czP9/swLWdd42sWYBjBW1VC3MHkWNYb4DJG9DoNogu85T3itzddJaMgIHf3YLg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703432206; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Pu9WVzeV/1YoiCoutiLGzlwxQ/xfoih+v8yhYdo7RdM=; b=nMTsSnvBQTGQ7pgpuMYBOnOEs4zBAzZid0Zct+3YolhvHRFfpFJjyER7l27ksnLIKAi+EG 1ds4xuIEMsXZpQb/ZHPdT1JDyqKca4knwQJXsYcHA8SyS6ITTO2KmO97XZyHPkaxpm9A8u EdWafa2pHkISmY+4J0eYSdiKG7KOe7PhMcGRCvHAJ/kFCcHfhdqvn+sKYLRSt+6vqvU4aI JEIXJ8WfiP5n7/2Tbmd4Q/4269DJyWgTgSo556Hi9zESkzFpbq4iI+IypYPhyp0ue5BFLn S6ZUJLV5mfDftGo2H9/A/uvNDGUPEt/bCzXChtHDVGUCPWiKMDL8Keq4H7OU6A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SylWf11TbzYgB for ; Sun, 24 Dec 2023 15:36:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BOFakLu083881 for ; Sun, 24 Dec 2023 15:36:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BOFakUe083880 for fs@FreeBSD.org; Sun, 24 Dec 2023 15:36:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275905] nfs client: mount becomes unresponsive Date: Sun, 24 Dec 2023 15:36:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: lexi.freebsd@le-fay.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 --- Comment #3 from Lexi --- no, i'm not using soft or intr (assuming the default is hard,nointr): hemlock.eden.le-fay.org:/public /data/public nfs rw,nfsv4,minorversion=3D2,sec=3Dkrb5p,gssname=3Dhost,proto=3Dtcp6 0 0 in case it makes a difference, the max lifetime on my user TGT (lexi@EDEN.LE-FAY.ORG) is very long, about 30 days. in the second comment,= i'd only requested a normal 10 hour ticket. in the original report, i think the lifetime may have been much longer - which is why i thought the ticket hadn= 't expired. i'm not sure if this is relevant (i'm sure it *shouldn't* be, but perhaps this configuration hasn't been tested as much). --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 15:50:38 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sylqf1kLyz54nQ5 for ; Sun, 24 Dec 2023 15:50:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sylqf0hFMz4Q60 for ; Sun, 24 Dec 2023 15:50:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703433038; a=rsa-sha256; cv=none; b=lxslbA8fTCXig0TtRDkc036VPWHuO/8wIkjMJM4WjRY/4O1WCY7jkgCBcdqsW2JnFHqLxC vPSKsr0Wlb0dQvMF3tKeKpgEKBagAEORhIUS72vOkOEYEFlP+Nj+xByW+b/QHBoYJ3itDG xGpgzVs8kTu1RN3AL25g0odsqsmTi5WDfZa3/oS1nbTga8FSyCzEx8g9D58+5dn/p3QhLD NJk1WVuhFcwwvwgp3TMPzmdOEN3l6IhiwJvTLKbvtrb8t110If9ourNBAU8Dkfz3VlDKVs bujqONVjIrquKnvKrH9JJ/+aZbxXhuNJt8jw13k/h5Ki6gIQKbsboaeg2RJJcA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703433038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TW+oWNB3T+iVXjpN16/fJPLk3wf65VcZndL5v+aAOdU=; b=NnsmkUUj6xQVN+WK0pMaV8nxl4lyQK1hcm70W9KehWYO/vPbmF/cJ2xZW4+dga0v3SXa/s ph4deeB3HZBBqmQ83vTeXY+42yPlw939HHSWXLDBGrmX3fPhJZEwXgxfbSP3K9ZePusYi5 NFUS7xwkz7xkzBpxa/VvI+/Oihwc9pW1woD1Lxqjpj/wWuKV/gF/aKzyrYYsMSjCZOHk38 deAheHPgFGqVRWsyfmn/QKLimYTjcvEFtkU28kqWYzixix1Zy3FA3cEDqpS3KIKLxjyONX gDaFuL3RkOOJKoG9vrtVc7PfYrCMJ0jyGYn+qlD/y+aOGnjL295Oevq50VivMQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Sylqd6tGTzZ90 for ; Sun, 24 Dec 2023 15:50:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BOFob2D099023 for ; Sun, 24 Dec 2023 15:50:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BOFobW0099022 for fs@FreeBSD.org; Sun, 24 Dec 2023 15:50:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275905] nfs client: mount becomes unresponsive Date: Sun, 24 Dec 2023 15:50:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 --- Comment #4 from Rick Macklem --- The renew thread RPC (the one that your backtrace shows as hung in nfsv4_sequencelookup()) would be using the host/FQDN@REALM entry in the default /etc/krb5.keytab. (ie. Your TGT would not be used for this, since the renew thread runs as "root".) Could the client's keytab entry be broken somehow? (As you probably are aware, when you create a new keytab for a principal, old keytabs no longer work.) You could go onto the client and do # kinit -k host/FQDN to see that the keytab entry for "host" works ok. (I am still wondering what will happen if the TGT for the keytab expires just after the RPC request is sent to the server. I'll look into that.) There could also be a problem where other RPCs were using all slots, so the renew thread slept in nfsv4_sequencelookup() and then another thread needed the renew thread to do some work. --> The renew thread should really have a dedicated slot in the NFSv4 session, so that no other thread can cause it to block. --> I might work on a patch for that. Does this hang occur frequently or was this a "one time" hang on a mount that usually behaves ok? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 16:01:02 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sym3g2zBLz54pg8 for ; Sun, 24 Dec 2023 16:01:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sym3f5KFpz4Rb0 for ; Sun, 24 Dec 2023 16:01:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703433662; a=rsa-sha256; cv=none; b=Tqd2WH9/hvwwX12omAre6BVI8eVDSU2KC1mvNgn5dFZcFayym9GmdskDosO3W4o+x9Veo8 KRIXVGT0yArYWP+W2xm0OqNo86kpxc+xNb5kPYdhueg0WtmO/2i5/vUVCfeLs9N73qubir ioDNKQTargpUMj1jC9JwBZ1t5U7m7riT5PaTLz6JCrsvkjt/Or834RPOE7BX+Tl+O1Xvrl PH4hWSi0+wBlLOFK7PYGJFGrWjecS67FEG+q7XC3RL9RjP/NEUQ9zNM5gII45bad7sW/+m a8EAxD8iWRANmV+AnvofYpdOAT80/U1QgZDpTQS1mNlkAU84zrs90B1C+OygDQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703433662; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZXk9qe2sfo8hniYtZinmd24sHDkZeHJf+4dIj9Rv1/I=; b=tdAT+MvJAj2ngTNeqLpLJvtt5HNUjWFTBaiT9nbQYWTjPD4uN7CSkkyXDf3EU8M4aWVQpe 8CleGkiDOGC+QtSiX3rEW0QCnDlrPxHt907Qt107BDolgwQqhNqvDswEjfLoF/ucun0TsO LM5m64qKI4NLU5/LF4kzeKi32YdD0LBX2iKWnxrPNlYQN19XlybxK7fI3pyBXLi1Obm/bZ mU7GiaTeJXvflXEpzTb9DR+JaCqt8sXXiCS4u+hTgQVr4TaTLXSf22DLLG2FeXMnP3tUS4 XFXWB75YRH1VKkQbShqtybBKMaH+H2ZjvglsZCM84bPqv7doSv0eG6BeRe8vdg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Sym3f4Pd2zZjX for ; Sun, 24 Dec 2023 16:01:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BOG12vU013564 for ; Sun, 24 Dec 2023 16:01:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BOG127H013563 for fs@FreeBSD.org; Sun, 24 Dec 2023 16:01:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275905] nfs client: mount becomes unresponsive Date: Sun, 24 Dec 2023 16:01:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: lexi.freebsd@le-fay.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 --- Comment #5 from Lexi --- the client keytab should be fine, since i only created it about 2 days ago = when i installed this host and there was no previous principal for this fqdn. 'kinit -k host/fqdn' on the client seems to work: # kinit -k host/ilythia.eden.le-fay.org # klist Credentials cache: FILE:/tmp/krb5cc_0 Principal: host/ilythia.eden.le-fay.org@EDEN.LE-FAY.ORG Issued Expires Principal Dec 24 15:54:16 2023 Dec 25 01:54:16 2023=20 krbtgt/EDEN.LE-FAY.ORG@EDEN.LE-FAY.ORG > Does this hang occur frequently or was this a "one time" hang on a mount = that usually behaves ok? as i mentioned, i only installed this host 2 days ago, so i can't say for s= ure, but so far it has reliably occurred twice about 10 hours after booting so it seems to be 100% reproducible. (iow, it never *hasn't* occurred at that ti= me.) in the mean time, i've temporarily switched the mount from sec=3Dkrb5p to sec=3Dkrb5 to see if this fixes the issue; aiui, this means no GSS should be involved after the initial mount, so i expect it will, but i'm happy to do = any other testing you need. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 21:00:13 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sytht2pJPz54T4F for ; Sun, 24 Dec 2023 21:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Syths75C9z3GdP for ; Sun, 24 Dec 2023 21:00:13 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703451614; a=rsa-sha256; cv=none; b=Rb86DHpKQn0muG0gmMeevBO8I/VPpuBzSqcRm+yCoqpT9+DyrFWuVyCOj/8ldMY+4HduvH IadTT0n1xBv79Eh8mjJIhBKxAPsiIZ1JA3l6L4luPwk2i8iqyYE46aHHHwY2FZ1Ur5cIP7 RuoLXu+p8W3FX/qKB6o0qcOTlg71DOPcOCJQvoAOGaFkADnlefzS54+iyBbSgL6eJaKlQm 7x1r7GhgBn12+zw8wRQlErpXDX0pgSQq2NwSIky17dVTus+WmEjGQ5HpxdoN1OOSAxoWRB 7rBHHE2yjyF+CXq9gbJZl+Q3tnuEOxZjSCrKQCq/mYDeRZl4162Ct2TQsk6beQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703451614; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=rOAe9aOzVqdDz3JfUGgfQwlszKD9mgtaZctHbUJDsXY=; b=ca5/15W57ogAVesxiYqOOktn4tpyorRHZvxtTQitsVgchDU54WGxgROy6MmXf0gau+SUV7 lFOm5Pd5JpdTAWKJRT5vZeL8EMS1PP+eP8n9Y7FhUe/xBHM/SwU0bw0WBh870pE2fnViSh Sl5ObxnWNkttz3nyzckD+FQmPscxRyMBn+/3qiEyod9kkiZh9Yoh89BcNL0nu7Y6qnLP/x 25UyUWpK+W85kmRC3pT5jgfkUT6WqNEzyJNGdG5LxUTIppDdeAY26B7XKvyyWTlqxD2j/7 jOtvdln+lRPrMneyhDEgcWQNZE3ttWQeGlJ5fTyAHT5+kEBl1Sk/DWp3+8jTFA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Syths5dBwzk0B for ; Sun, 24 Dec 2023 21:00:13 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BOL0DFO071864 for ; Sun, 24 Dec 2023 21:00:13 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BOL0DZ2071863 for fs@FreeBSD.org; Sun, 24 Dec 2023 21:00:13 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202312242100.3BOL0DZ2071863@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 24 Dec 2023 21:00:13 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17034516138.c57dD7B.69967" Content-Transfer-Encoding: 7bit --17034516138.c57dD7B.69967 Date: Sun, 24 Dec 2023 21:00:13 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 231794 | zfs: Panic due to ARC related KVA memory exhausti Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 244692 | gjournal: Does not support TRIM Open | 251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W Open | 264139 | ata: NCQ_DSM_TRIM trim method for Seagate IronWol Open | 266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy Open | 268162 | zfskeys_enable: each successful load of a key is Open | 269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit Open | 271384 | zfs_load is not suitably documented Open | 226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 = 10 problems total for which you should take action. --17034516138.c57dD7B.69967 Date: Sun, 24 Dec 2023 21:00:13 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    231794 | zfs: Panic due to ARC related KVA memory exhausti
Open        |    237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w
Open        |    244692 | gjournal: Does not support TRIM
Open        |    251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W
Open        |    264139 | ata: NCQ_DSM_TRIM trim method for Seagate IronWol
Open        |    266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy 
Open        |    268162 | zfskeys_enable: each successful load of a key is 
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit
Open        |    271384 | zfs_load is not suitably documented
Open        |    226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =

10 problems total for which you should take action.
--17034516138.c57dD7B.69967-- From nobody Sun Dec 24 21:31:37 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SyvPP6PMHz54Wxt for ; Sun, 24 Dec 2023 21:31:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-21.consmr.mail.gq1.yahoo.com (sonic305-21.consmr.mail.gq1.yahoo.com [98.137.64.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SyvPN5xK0z3QBc for ; Sun, 24 Dec 2023 21:31:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=TF8e22oW; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703453510; bh=L5u6bHdxQ0n9y73nAeGAweGqZiZ4uFRzkdf2DDuKZ1E=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=TF8e22oWAM+011CWp0h/a7tD+scRp3KCUaawNioBuznTnibZyrpckL78moOwuWD0yTxuEa/chTH+VtnmYPXGohnLp6VfgUmbFEzbEsBS7NoULdmsurP5kAV8/dQ+2f8xEbJlfb9zlLxIrpasA1Pp9JshFW6Uh6vuLeGx1Qars9ips6HRv8gR2RMTz2KJjE4NCQT/9VfqN2GXGXASryGnIG86R07NL2c89E8tI83gkLsINdqZX+sMmxRUkJ6viNGyXKhztAJdZ0EQwBkFjOjBAewEdX9vCCH0cjZpJpi1mVp04Dg/4QOn/cmotXDByVAnY2VK6CCUER+Pyah4faZerA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703453510; bh=g3nKcj7kjqE21e6bTHhODAmSIcaR/ikLL3/TxcMpJ6O=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=AG54SNh2MhbGRrymvi8kI6Hnr1BThXXVweWJyVkGMzZPHROP+VLv9MJfknIcA49MivAxrV1WEhZqrt/98atSndgbeaCcncNJ39UOBxOTBnQBIkkxc+BTFffawG056pttKMZF2zNAlBijv7DD+2LOu2jBom5t4yAYMortdCvgWf92kpTwGXm46kaguNEaXZFpS1JVhq7Rc/FPKQdyB9ajGLo4O5fezE0MfQicPb5VmrxG8i7R5ZANq8FBx/g63zW5VT6WvUVmY0nePIk72Phry6fF/rqGGvtT2h4bgMDsHdV8GDMG1MTCwhf4eoD+G7tBwfxykwDBTRwoNOZCTBYV6g== X-YMail-OSG: iNKZzMEVM1lgcZjThdv3ZvVLkWowrZxzQayh3Nu7vOkXw6fbcFswMH5wZc7DH_b nCJgayrG3cMPA.Gfz3OeM5_GNAQyY1fxxNM2fslglddjyMYBakPkscGGkIljQGnqv709jS1Nb78Q xvDJAV06obrLRxncqB1duWXXlV64WKSMN14cAtlaGmZnyxn47slcWwqpXXImPFLTcFtv7UtU8rqQ XzlMtOgpFh9pSOeGcvzdiXOqrNsI4PzCaTwjJoevNOYc_k.E1pSZr.4UExtFsS6lfhfLcVadObzw ezlezQcRPhoPH6t_kjxbjS3O23Lewtg58rI_U0JsjOGeDw4R69efDOhVtZFXHdbks6yXPok0KJ.F gR9n5zlXuuOrsnjpzdNU7BUjJSdup9C1k42_tN18A_iEQqAqqHLwPzoXYc1_h53RxFazu8rcGrjT rCELsGjkC5fMH22yjJBVh9Uft2oPMTU7qbXzobreFqyvZ1ypCNfVqJ5E6xHkKVgsNOZiS.zHy6KV V0BnH0Q5Dp1p.KxT0dYW2xVf1B3loU1toZGZtWjqJtzkGJVI_Bk30iv415uMcNE3bIKgwbCJmx_y DyreJRypOtEm_0HBgUZK8uNPCK0e6oqsQIF_tSVUGUCorrbFkZQaAqQ6vmctK1C0oOUpiOVBW03M 8OYu53UQNGJhz787EzEFhVNCIYGZtpYbUlXyJyPhSo3EqcKuwWsdBmH6yefgK3FIJuHXHEnz9ZbU Ji5BMObc2EZx69H9ZjjiJfI1RhpqwTakv2VT0fpeiUbVU.ORKqCVZHX7_lZjnidYlpL0OyvoKEpd ApWYY0EmamyfDynAaio0MpMb55r9ZIm29aORKdOBJ_TPptQIOhkazv5dwDBM_effnR4BpM3I38z9 bLugO3lWMvxYAioTy87p6iqq7pY.lpuBH9JE3y2TL79NMcf3yd02nzlwoeIy0kBhT2_eldEuJonJ VWRdTrHOqY6Gz7omPzJ4GjNUPbxrgvtTHMDXugaSORq2XmYeXP4GMz7BlnH7Ck9jrwCuImGCWpOc RN1TT5_vbe0_SI5LOAkBHvyKEgGditPpi4JlJ1C6Flb6TMBqOSRjX.xxDUAbgWSuhvYOEn4C3uAB cZpJ.yKHJ83nCWmokHEhmxk_YtKAzAANebfP306xANM8i6p4fFrN0J6rHxT2ngVEyBNJZyUVk2ku Ocr9XTkqZYRVZqHD_xEiKlf3Xt6D1bk8TSnWE2DMTx41Jc1ZVQjnFLKRH5yvw7pBxRhpNHo233OM YbMuSUpspt2mdwnXHVjhrBdp28ZkTEH8Op7a7xAPkemaC4mqp1egME3HTvob3zq2WRLuH4va976Z jUIiPx2jZHMLY7Smv9RKkvLWGPR2W8TSk0WK0CM53aLoQfcflCrrzCiw0C7ZPte0vHKHhnkKg0wk Kglrr9POvCTue7hONy1rBCQMEbMta3wncgQ.RrbBQjO71RQMk8zgIK4eGGHrFAymnsrOMU7DgVEu 0yODzWd2wZiQ1zXYyLMpiJrSx1BUSSMHF9dh1K5497vpnHbRTPrbp49JioweJNvsJwk9bytozmWl 5csUyXWoeOlMZvixnOr.2iyjoD8Je5EomEREoHgOCQA2y44X90gW6wnt4_DBkKpnih_7.9bebQtv jhCYvbltsWn1pfFM5rrntuX3Cd1_.vxIOQtLDATzsQhNsh8hQsf96CdvERytY.F5TLdBbSKKBVOu M6D1I51nyIVUZtXDh9kPgljiYc5GMhRd7dszXRYBImULM7yVQRe4aDLeL_saxWxGft_f1R_Yf6LG tyigjckvcfYcHj8aCfV4QEVgfYbpUGo2ZsLgbtQZxg771Wgh7hA3iGQN4eghFjqMr3aue8vVL1Gt rrXd.lV0quLsK8A.IMCLoQv4qui11iju1k.sGFvc61kONKxH.mlts_ZgO9NYltP5IPp_JZzz5VAZ e0OPwFdlGPTsnscB91OSw8E33KwpV6SMBuk6biu2UgBr7dT3xLcRgc45kP__L.xTqttFm0ZSkyvI zmy5G4v33ak5e9thT1Wj0C2C9aD5ei2rIvRzx0rdcsbkg1.RVhDdaa3tzm9piVCdTJgsvixX.PRn bUSl1FEC4Q7LEbzA4wiL.xq85GetfatrNWFL7ULAr1gRpLMPz3R_I.09ULVyzOhVO1lvlr0TvZv1 UgTNrUm2lJ8HKhCmgvRKHe3r89ayGfgeoG0apDb0tq1peRBY7dXJRJZMzSAIUq5HF2L4rakBBaHW EvgreSHPFAr6spNaWRCAr09RJJLL46p4PZmOtJ2f8nqT_gwN3GA8inCHGf3Et_hf6gp_SwVnKcvJ uK5pnF.Q- X-Sonic-MF: X-Sonic-ID: 049c3491-2ee0-4967-a313-59a638fdc77a Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Sun, 24 Dec 2023 21:31:50 +0000 Received: by hermes--production-gq1-6949d6d8f9-c9pk7 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID fb55ae36ebc6fd2c72a35ee06edce5a4; Sun, 24 Dec 2023 21:31:48 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: measuring swap partition speed Message-Id: <0D286183-ACD6-4818-8E27-E03D7F16D288@yahoo.com> Date: Sun, 24 Dec 2023 13:31:37 -0800 To: void , freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3774.300.61.1.2) References: <0D286183-ACD6-4818-8E27-E03D7F16D288.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.84:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.84:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[f-m.fm,freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SyvPN5xK0z3QBc X-Spamd-Bar: --- void wrote on Date: Sun, 24 Dec 2023 12:28:05 UTC : > On Sat, Dec 23, 2023 at 11:55:43AM -0800, Mark Millard wrote: >=20 > >If you had the resources to test avoiding the encrypted > >partition for your type of media, that might prove > >interesting. >=20 > The problem remains even with UFS. It remains after any non-swap i/o > load stops, even when swap is not otherwise in use.=20 >=20 > I took /dev/da0p5, formatted it with newfs -U and put it at /mnt. >=20 > then >=20 > root@pi4:/mnt# date && git clone https://git.freebsd.org/ports.git > Sun Dec 24 12:06:50 GMT 2023 > Cloning into 'ports'... > remote: Enumerating objects: 6044522, done. > remote: Counting objects: 100% (941/941), done. > remote: Compressing objects: 100% (125/125), done. > remote: Total 6044522 (delta 923), reused 821 (delta 816), pack-reused = 6043581 > Receiving objects: 100% (6044522/6044522), 1.16 GiB | 6.94 MiB/s, = done. > ^Csolving deltas: 2% (101172/3649394) >=20 > date && dd if=3D/dev/urandom of=3D/dev/da0p6 bs=3D8k conv=3Dsync = status=3Dprogress > Sun Dec 24 12:06:52 GMT 2023 > ^C203841536 bytes (204 MB, 194 MiB) transferred 205.012s, 994 kB/s=20 > 25009+0 records in > 25008+0 records out > 204865536 bytes transferred in 205.914885 secs (994904 bytes/sec) >=20 > cancelled both processes, checked swap >=20 > # date && swapinfo -h > Sun Dec 24 12:11:56 GMT 2023 > Device Size Used Avail Capacity > /dev/da0p2 2.0G 0B 2.0G 0% > /dev/da0p4 2.0G 0B 2.0G 0% > /dev/da0p6 2.0G 0B 2.0G 0% > /dev/da0p8 2.0G 0B 2.0G 0% > Total 8.0G 0B 8.0G 0% >=20 > while /dev/da0p6 is available as swap to the system >=20 > # date && dd if=3D/dev/urandom of=3D/dev/da0p6 bs=3D8k conv=3Dsync = status=3Dprogress > Sun Dec 24 12:10:38 GMT 2023 > ^C78790656 bytes (79 MB, 75 MiB) transferred 96.008s, 821 kB/s=20 > 9767+0 records in > 9766+0 records out > 80003072 bytes transferred in 96.986518 secs (824889 bytes/sec) >=20 > # swapoff /dev/da0p6 >=20 > # date && dd if=3D/dev/urandom of=3D/dev/da0p6 bs=3D8k conv=3Dsync = status=3Dprogress > Sun Dec 24 12:21:28 GMT 2023 > dd: /dev/da0p6: end of device2031 MiB) transferred 105.007s, 20 MB/s >=20 > 262145+0 records in > 262144+0 records out > 2147483648 bytes transferred in 105.837991 secs (20290291 bytes/sec) Somewhat analogous swap-active vs. swap-inactive sequences follow. I've updated to the new snaphot (that has a UFS limit increased to be big enough to again allow "poudriere bulk -a" to finish). Note that this was tried in a context with: # sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels = dev.cpu.0.freq dev.bcm2835_cpufreq.0.freq_settings: 1500/-1 600/-1 dev.cpu.0.freq_levels: 1500/-1 600/-1 dev.cpu.0.freq: 600 # sysctl hw.cpufreq.sdram_freq hw.cpufreq.sdram_freq: 400000000 no powerd use or the like. I may later do some of the activity with my usual settings for such. # gpart show -p =3D> 63 468862065 da0 MBR (224G) 63 1985 - free - (993K) 2048 102400 da0s1 fat32lba [active] (50M) 104448 468757680 da0s2 freebsd (224G) =3D> 0 468757680 da0s2 BSD (224G) 0 128 - free - (64K) 128 451979136 da0s2a freebsd-ufs (216G) 451979264 16777216 da0s2b freebsd-swap (8.0G) 468756480 1200 - free - (600K) # swapinfo -h Device Size Used Avail Capacity /dev/label/growfs_swap 8.0G 0B 8.0G 0% # dd if=3D/dev/urandom of=3D/dev/label/growfs_swap bs=3D8k conv=3Dsync = status=3Dprogress ^C506142720 bytes (506 MB, 483 MiB) transferred 31.001s, 16 MB/s 62278+0 records in 62277+0 records out 510173184 bytes transferred in 31.247613 secs (16326789 bytes/sec) # dd if=3D/dev/urandom of=3D/dev/label/growfs_swap bs=3D8k conv=3Dsync = status=3Dprogress ^C783056896 bytes (783 MB, 747 MiB) transferred 48.001s, 16 MB/s 96653+0 records in 96652+0 records out 791773184 bytes transferred in 48.533448 secs (16313969 bytes/sec) # swapoff /dev/label/growfs_swap # dd if=3D/dev/urandom of=3D/dev/label/growfs_swap bs=3D8k conv=3Dsync = status=3Dprogress ^C749903872 bytes (750 MB, 715 MiB) transferred 46.037s, 16 MB/s 92085+0 records in 92084+0 records out 754352128 bytes transferred in 46.308913 secs (16289567 bytes/sec) What I observe via "gstat -spod" is the likes of: dT: 1.004s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name 1 1999 0 0 0 0.0 1999 8 15992 0.3 = 0 0 0 0.0 0 0.0 67.0| da0 Sometimes L(q) (length of queue) is 0 instead. The ops/s and w/s vary = some but 1999 looks to be the most frequent value. %busy also vary some. ms/w seems rather stable once it gets going. I'll note that ms/r counts how long it was in the queue but w/s does not involve the in-queue time. The above applies to both dd's. FYI: in my RPi4B environment: # find / -name "*.core" -mtime +2 -ls lasts a few seconds at most but there are no source trees or other significant additions to the snapshot content. The same goes for: # periodic daily So I may need to clone to populate a /usr/ports or some such. . . . # cd /usr # pkg install git-lite . . . # git clone ssh://anongit@git.FreeBSD.org/ports.git . . . Updating files: 100% (156990/156990), done. # swapinfo Device 1K-blocks Used Avail Capacity /dev/label/growfs_swap 8388604 0 8388604 0% # dd if=3D/dev/urandom of=3D/dev/label/growfs_swap bs=3D8k conv=3Dsync = status=3Dprogress ^C554303488 bytes (554 MB, 529 MiB) transferred 34.000s, 16 MB/s 69046+0 records in 69045+0 records out 565616640 bytes transferred in 34.691537 secs (16304168 bytes/sec) # swapoff /dev/label/growfs_swap # dd if=3D/dev/urandom of=3D/dev/label/growfs_swap bs=3D8k conv=3Dsync = status=3Dprogress ^C553779200 bytes (554 MB, 528 MiB) transferred 34.001s, 16 MB/s 69145+0 records in 69145+0 records out 566435840 bytes transferred in 34.774020 secs (16289052 bytes/sec) So I do not see the variability --including not in the "gstat -spod" display updates. I'll note that with /usr/ports/ populated: # find / -name "*.core" -mtime +2 -ls now takes more time when first tried. During it "gstat -spod" showed the likes of: dT: 1.066s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name 0 506 506 4 2049 0.4 0 0 0 0.0 = 0 0 0 0.0 0 0.0 18.0| da0 (L(q) is 1 some of the time.) But that is only for the 1st run. Rerunning afterwards does little I/O (caching). Other forms of prior activity could produce such variations. Adjusting with swap on vs. off status does not change this behavior. Looks like a reboot or some such is required to have the find do a bunch of I/O. This likely messes up using periodic daily as well, until a reboot or some such. Note: so far I've not seen any "gstat -spod" information for any of your types of experiments. Below I instead use bs=3D4k . I also use "time -l" # swapinfo Device 1K-blocks Used Avail Capacity /dev/label/growfs_swap 8388604 0 8388604 0% # time -l dd if=3D/dev/urandom of=3D/dev/label/growfs_swap bs=3D4k = conv=3Dsync status=3Dprogress ^C598851584 bytes (599 MB, 571 MiB) transferred 63.000s, 9506 kB/s 147514+0 records in 147513+0 records out 604213248 bytes transferred in 63.563283 secs (9505696 bytes/sec) time: command terminated abnormally 63.56 real 0.46 user 14.32 sys 2176 maximum resident set size 16 average shared memory size 4 average unshared data size 134 average unshared stack size 130 page reclaims 0 page faults 0 swaps 0 block input operations 147514 block output operations 0 messages sent 0 messages received 64 signals received 147516 voluntary context switches 14 involuntary context switches Example gstat -spod display: dT: 1.004s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name 1 2299 0 0 0 0.0 2299 4 9195 0.3 = 0 0 0 0.0 0 0.0 76.3| da0 # swapoff /dev/label/growfs_swap # time -l dd if=3D/dev/urandom of=3D/dev/label/growfs_swap bs=3D4k = conv=3Dsync status=3Dprogress ^C579633152 bytes (580 MB, 553 MiB) transferred 61.001s, 9502 kB/s 141615+0 records in 141615+0 records out 580055040 bytes transferred in 61.045478 secs (9502015 bytes/sec) time: command terminated abnormally 61.05 real 0.48 user 13.73 sys 2176 maximum resident set size 16 average shared memory size 4 average unshared data size 135 average unshared stack size 131 page reclaims 0 page faults 0 swaps 0 block input operations 141615 block output operations 0 messages sent 0 messages received 62 signals received 141618 voluntary context switches 14 involuntary context switches dT: 1.004s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name 1 2328 0 0 0 0.0 2328 4 9311 0.3 = 0 0 0 0.0 0 0.0 76.7| da0 Note the "time -l" "block output operations" being just a little smaller than "voluntary context switches" for each. Also, it appears that bs=3D4k is limited by whatever contributes to the ops/s being under, say, 2400 rather than being more like 2*1999 (1999 being from the 8k=3D=3D2*4k dd context). Another systematic difference in our contexts is my use of one swap partition (the default for the small arm board snapshots these days). =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Dec 24 21:42:25 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Syvds53J9z54ZVP for ; Sun, 24 Dec 2023 21:42:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-55.consmr.mail.gq1.yahoo.com (sonic308-55.consmr.mail.gq1.yahoo.com [98.137.68.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Syvdr6Cjzz3Swt for ; Sun, 24 Dec 2023 21:42:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=mqd3Ij9+; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703454158; bh=bFqVhMgGDyUzNGyBvMCrEDrExoHDplftzNN8xWEhqow=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=mqd3Ij9+H9TpTLdArUwubK73r6kCVH/F3r4x4x5E9e6R8AqbujqjkDOQMQRJgjg0WnR66pR3m6JcJnzfGr0bk+6yuH6S/7Ww6gdzczLNz0nCzoOJmEBDXghxRI44nTVxfXp2Wv9/WiN8nc3+f9kdH2+Yl/PzJ9XFUZbf/wmjhVOUjx20xWKWZt5ZR+dmPoErlPiJz0kD48DaKRaGeu0+z9xCCJqmXbLVWtcD21DXBHWycGTRuvybLs12rqeQHGLXcyKKCXlDLHmQkBZPD7CXHDGHjSZ5YjZ6vhjadET6ejcYO/3Jvc0XQlHNZvzcSF/nrJxa5xGEEWKH1o+puDncFw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703454158; bh=puJwToX4vx7Wz1xp2zUrve2i8FdaBa5iTxou8V2zBMI=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=MPXDKZx0hyCrWHzt0UE+DKnzhyuemBh+jwimEavA/ucwLw9GH5ITGX+rM0ZY0YU0bMfGocOY1wHPtVfdV3NCifMYanctoUM26B21rkt0RagNmwrmL4Jr35AArCzNSRxn7KhLKI2OpOj9GWHQTWvDAstIE6PH/3C5ULtzw+0oiyOKp4y8cYsuK2zrOYxebKqyH75vdKKRbcUXGsA6V0almlIsHqOHHjSvGjje2sSM3BKvKW5ugDaxzTA7It3dij9XeJmZUsAU+1og1+8EYpJrfcNe3TxjpSFUB6J1lAGYaWoqloRGEejoK7c7IAHIBFZ3KDVIOoo24Jqu5qT13CoyBQ== X-YMail-OSG: DJYDUjkVM1kUiPdFCW8aF63MVUspDNACglY.S0erZGI3z6OHgSLkMUEwE6JhxQE _1autBSfLDlSI_CUEnM38LE0spl.vH1a7g3qILd0D8pnhDOvSvEu9oWFfbikSIcOR7SdI1Z43EDW Oow4ip.gzy79L5P6fO5h3HSXgtLnzur_IYMIl7Pcguzu1gesO74AEjd2acdj9v5PG.w9yliz1Adu 8xAIHIpECMLSa0rN1skLU49ZH9dB8E1fCo_ay4UQ4nK0Ju_tcUm7YPzMtsWqvKMJtq.I.hBzFBTd L6c32X7W7jeVggdcTrO1YROQeB7kzMIClPviRsemvBaSGNY2CMJXJcql3mikclcDgvkiLwi7fTGz Gqv3t.4BrH5.0y0K_gwHnlDr_y4AieYWAa6O_47qcOT.BBn.QwQLESVPbzCAV.6Ff6OoqtrFNze3 IYos_68HXR6C50N4wT9Od2GvYF0oS0dRgTQkd9g8a78LMSNdE7WUy4v8cDZ4_m7s0_jRhenSuDMW g8wid4cRqz8ELUbWdUz83YBy3Q51eqdimC537YqlutVkSsOtiw4Fd_GEXwBe3ZpJPlaiYdBqgTmg D7.o4U_82Nqc4udtS7l.2qdQykXAPYeqidod596472md.sZY5vG5kKugXupp8J3qaDDo_S109hDp PMRpOAhOycRsFmRyremDswa2RzyZwqo4oTu2qw_6t4Te1hH6PCHGgV148tQDn3iYzDalj7o7k5FY y6lLpBw.7uDrZwiQhWZ8GJjmLSSpPxVxb8frrc8NUUv0aFA6gRAi3iYh_Ws3TDo3oCO.C1iARb7h X6F3dNwlSYd5592ncTw5jeTAuJ70quRMEAVM5sXXu_6q_fb7buNUcNq0SnnI66oIxlEzGIv2blZd FrDLJTU0jfSpvBmVBFp5ASMPHLio.Ijejs98qWPpmavzuyDd3f.DWqU95IIEmBgAoBf11tpQW_42 UtV1SrrUbdgfzPqFYwmHRqQp_jXpxy.KwXExAICr0F4kZ_HACvCTfagUiKfAxCx6jwj8SUjXCMvZ 02D737Lqk2aUcjj7_q1zuET4OhgxYUDKdIQohzo64gr7SWO1FdWOTFt.KHFF9X6Kj_ADsoGbL9eq 6.spPP4QfKjRpHzk.QdmxziQr3u4V4fckNr.bpBuH5glkoCi9.nvkH.uLf0EWsYXSW3f_JLLqNAY jTPcWZCM01qizod2eVmtyEmW3nsTl5Xel4p.CwbpkbDLseShiBXuFANDOlNPeE5cQIJgLtDhkSrd u4MoTpNErP9eJTl.xwlXL_toXU2oyeShQXxrNXvPc69gxG0hE0IOjNIrht6FgBUYiDfYLTowh9mz wdoFvurGKFBd84kQgBpP1i6CEVfeNlCSZnCd.dbbL1PZN7lO2lt_bPka0mJHUtvkJv1HFfGvg2dL ueqmXFuVeFO5AeHiq0sLdnf5fYspLp4GOrwGPZtbujPz9ZfyD5s.RRncuIXYd1xaZ0Niw37lCLNg wEK9oJjYTQCQtM6KbdJSlY3FP_hBYkMuOReH87s7Pd1cQEMD0IQ0.xgrCey_sSCaXlEAZRCtyWPr X4v.VFJh1z6zzxFA948b_XxWeF4S6C4NuTCC7C6OHFwNDD4IbRq4piaTedwmpfs1g3R5h1Bl3dUo _wfN0R6.xUhuWzmuZMQUvEouh9s0JM63_abefrLO4z25WA0f4oqVqd80ah0UCE.0Yht.7IqDAsu7 9Bw0Uv9tD9rVDsKBEMvT3BjvTNycM4w1Sgf.t1ANkli1_tIOVej0FxaHI6W2XwRgDjD4Z35D48Fw 40c4ClDNrikYrt7.lk9HLYu6TjbwEsfGCbklAFnepmemzDQxBioZ7ZT.KdbQQrcqiwE31fjhYOki 030WD4J7SeOxOYwPJZN_0Cmu9xpBfDBuLWRUhAUT4I0tvGc6SP.9mdE199CbEXchBJzcM1aml.a5 52TYjFjTCEgULlydtf_2vtp43iDk40tjOYdreDZkz2goLuD5iFBKsVqCjLab4m.1mXv20cqByC7n wME7n9KjecLBBlKuT5vBeLqqrmpwZW878_2J0E4nx6hOM95LKnLPiHv_S0iRdQEmavZsyK7WOz28 A_3vbIEQIfG2laL7zt2jLeWpbszJd7pLT2A99blczuH_J6D25NABmFqxQAVQhRpkRCk6VGi_uEQF sNUeCRFIItKAvxIDSnkhIXanW7BHhUQjgYtpe8W6f7ufjlSjaWxJtMCsdgbnwg0zazJGAT_QSGzb jtatXrAq0SwSVznqdoESZYRC1tHv4P2OvwEXweO68g05ieCdTp7jyeuBeoRMKA5fFx97HtaRApYg - X-Sonic-MF: X-Sonic-ID: 7aa2f527-b98f-4b46-af47-5720a0f85e5e Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Sun, 24 Dec 2023 21:42:38 +0000 Received: by hermes--production-gq1-6949d6d8f9-k52jv (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8524ba17a2f55de1e17f40836eb13afd; Sun, 24 Dec 2023 21:42:37 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: measuring swap partition speed Date: Sun, 24 Dec 2023 13:42:25 -0800 References: <0D286183-ACD6-4818-8E27-E03D7F16D288@yahoo.com> To: void , freebsd-fs@freebsd.org In-Reply-To: <0D286183-ACD6-4818-8E27-E03D7F16D288@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.31:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.31:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[f-m.fm,freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Syvdr6Cjzz3Swt X-Spamd-Bar: --- I just thought of another relevant setting that my normal context has but that the RPi4B snapshot does not have (by default) and you might not have: use of noatime (or analogous for ZFS) The default /etc/fstab for the snapshot: # more /ets/fstab # Custom /etc/fstab for FreeBSD embedded images /dev/ufs/rootfs / ufs rw 1 = 1 /dev/msdosfs/EFI /boot/efi msdosfs rw,noatime = 0 0 tmpfs /tmp tmpfs rw,mode=3D1777 0 = 0 /dev/label/growfs_swap none swap sw 0 = 0 Keeping access times updated causes lots of extra I/O activity. My future snapshot testing will likely mostly be with: # more /etc/fstab # Custom /etc/fstab for FreeBSD embedded images /dev/ufs/rootfs / ufs rw,noatime = 1 1 /dev/msdosfs/EFI /boot/efi msdosfs rw,noatime = 0 0 tmpfs /tmp tmpfs rw,mode=3D1777 0 = 0 /dev/label/growfs_swap none swap sw 0 = 0 =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Dec 24 23:00:17 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SyxMd27tyz54lK0 for ; Sun, 24 Dec 2023 23:00:29 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SyxMd0S4kz3Zlx; Sun, 24 Dec 2023 23:00:29 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-2044d093b3fso1471638fac.3; Sun, 24 Dec 2023 15:00:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703458828; x=1704063628; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=1SgrR8ZDFy9ABmSf9Ro25yoUpEUIc8vxFqR12WgAzg8=; b=gn0yLAbP9YhZ5m8EY93sgnA19MozGvr9w3OmrGrinuAoSfQEh4KqHVVw6tZnFzqyAS vkaqC39Y/Q+dg06x/zh7W5eHmRZmQjgj3jPCtkazmJwekY8J1gwnGkAwihOrluIAic6y BiaM1Q0P+eSXd5ZDqpbJsHYl7shcpflSX2KdktJKTzy2ZCEMFcsMstuyfrPGOps6Imx3 okWz4SzZoQPI2gTwNYDYw/bapj/zYVhQNeG8Wt90RqvwdoP8L6Gphfba2bmcmKOwvQlq 0DrcTqW6MIFL6mBA1vMIua+imi/XvKOgfYatYM+1KAeZT8bhrFAuAH7+5Jt8HDo8mvHH PKOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703458828; x=1704063628; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1SgrR8ZDFy9ABmSf9Ro25yoUpEUIc8vxFqR12WgAzg8=; b=g2MYhwctbEhtmxsuEGDH+3q9BO40kc8vo5dgxiLrA33fhi4UzZb4JKC489O5A7fHkW rvtAJLF+8VuBI4j2BKjHTnEbsA0S8vtlPJvnpKD1gUZ4uv3E15LOFN3KKcZaHeQTpT9c Ou2NkYaD4gTi0JkQ6SpzwhhUoY2FoHqC4I/aYbcNk8Ax9j5zptmETDjFUui1kGZAaNZ9 +8baJa6FXjvvtNKmtXA9kFlNoc79U5q8FYEAkJSc+hgaT2aIfCTar0WWeRe7L2aD/72l oH5uJQnQoTqpU19QbsTdu+URPd9x57KbUq27j4eTOzPybH4pfqBZAWnQFN+SkN14OEbY ozhA== X-Gm-Message-State: AOJu0YxvUzjU1+8/nXmiLuQgCE20ptUlx0jkevkzMpJmLYv4c26RSjKv RgAbTbvNhFhIEl8JmpqbJUA//bH8jNp7OGpJNP4mQM6LGQ== X-Google-Smtp-Source: AGHT+IFwNEXVOWOuC61JdBcNStQKS7Ab+pndhGy8TXleg19zKMt+GBqdm8aU+yLYy2q0mrTFKQ1LglUgWavimRatrPk= X-Received: by 2002:a05:6870:797:b0:204:7f1c:1ec8 with SMTP id en23-20020a056870079700b002047f1c1ec8mr1854986oab.2.1703458827845; Sun, 24 Dec 2023 15:00:27 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Sun, 24 Dec 2023 15:00:17 -0800 Message-ID: Subject: Re: [Bug 275905] nfs client: mount becomes unresponsive To: bugzilla-noreply@freebsd.org Cc: fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; TAGGED_FROM(0.00)[] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SyxMd0S4kz3Zlx On Sun, Dec 24, 2023 at 8:01=E2=80=AFAM wrot= e: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 > > --- Comment #5 from Lexi --- > the client keytab should be fine, since i only created it about 2 days ag= o when > i installed this host and there was no previous principal for this fqdn. > > 'kinit -k host/fqdn' on the client seems to work: > > # kinit -k host/ilythia.eden.le-fay.org > # klist > Credentials cache: FILE:/tmp/krb5cc_0 > Principal: host/ilythia.eden.le-fay.org@EDEN.LE-FAY.ORG > > Issued Expires Principal > Dec 24 15:54:16 2023 Dec 25 01:54:16 2023 > krbtgt/EDEN.LE-FAY.ORG@EDEN.LE-FAY.ORG > > > Does this hang occur frequently or was this a "one time" hang on a moun= t that usually behaves ok? > > as i mentioned, i only installed this host 2 days ago, so i can't say for= sure, > but so far it has reliably occurred twice about 10 hours after booting so= it > seems to be 100% reproducible. (iow, it never *hasn't* occurred at that = time.) > > in the mean time, i've temporarily switched the mount from sec=3Dkrb5p to > sec=3Dkrb5 to see if this fixes the issue; aiui, this means no GSS should= be > involved after the initial mount, so i expect it will, but i'm happy to d= o any > other testing you need. Not exactly. sec=3Dkrb5 says that a RPCSEC_GSS (think Kerberos) session is used to identify the user for all RPCs. The difference w.r.t. krb5p is t= hat it does not encrypt the NFS payload. I doubt it will make any difference, but it sounds like you'll know soon en= ough. I now have a hunch w.r.t. what might be broken, but I need to look at the c= ode (and if my hunch seems correct) and maybe come up with a patch. You could try the "syskrb5" mount option, which avoids use of the keytab an= d allows "system operations that maintain the state" to use AUTH_SYS, while t= he rest (all involving file data) use Kerberos. (If my hunch is correct, this will not fix the problem, but might cause it to behave better.) rick > > -- > You are receiving this mail because: > You are the assignee for the bug. From nobody Sun Dec 24 23:22:32 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Syxs50zLcz54mWt for ; Sun, 24 Dec 2023 23:22:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Syxs45lw7z3dLv for ; Sun, 24 Dec 2023 23:22:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703460152; a=rsa-sha256; cv=none; b=A4SKK9/aA7hwMgHA/kRUnYcld2OafFZ3+8OWy1CnarWRu0CiK8+TJmBsCPRb3FzIMwMdn9 RIqzEtC6tfSP9S0Eq3v9R0fhz3Z60NcqbyDAqICA7xJsE49zHmg4OOjY7fCpo5GGWy8KYE P5AmNdlk2ASZpszwvaWvDGQsMhYPUa0ar5rmZyxzLDkBFk2XagbPhSPSnb/AFUrvjoXaHF AjvLZyYaQuzC1YxSG25OYSkqbmT0XXm88YDpfex8Eu9WcL39zeQxXWyLFNSx9IcjmjI+Ng c0qDTxcFcjxZtAbRcS8SVDZjZ9nJ3lZlOGCI80Uzq5ON2W4bsbYvzWJ/2xgwvg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703460152; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YfBzglgHTcW/670z/O9QDB88ND3kWr2PTGrvDA057jo=; b=JXgA9nOqOnRouzxoBtX9ejAxjIGpqfAqxkWlWUGID/Rq7M08AAY12qH2VftmTHniWwVK2D gImiU4LR9G5vY/WvswxpgHQSt3Tl65MXJXdn+1y86rxTtX0n3c3rGTZtjHAocvZqN8hrnw ljy+e3W/mBWYgnM9nHa1nbEwVIvfWeH9Wgq8482ikiXX/385pDdazUE925ljPteyyIayJy 830usZh31yJQm1FfuuylF0k/cWk2isjYG59qOCYanZKeFs2wN6HIWUSCTCfjholssMBEcQ ytHjw+QneKEu1oSBU9hgDLO1VaaI/sZPf5aGaFBD+BSSu0A/Y8QHY2zdcqOZtw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Syxs44pxZzmjw for ; Sun, 24 Dec 2023 23:22:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BONMWFY000935 for ; Sun, 24 Dec 2023 23:22:32 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BONMWMP000934 for fs@FreeBSD.org; Sun, 24 Dec 2023 23:22:32 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275905] nfs client: mount becomes unresponsive Date: Sun, 24 Dec 2023 23:22:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 --- Comment #6 from Rick Macklem --- Not exactly. sec=3Dkrb5 says that a RPCSEC_GSS (think Kerberos) session is used to identify the user for all RPCs. The difference w.r.t. krb5p is t= hat it does not encrypt the NFS payload. I doubt it will make any difference, but it sounds like you'll know soon enough. I now have a hunch w.r.t. what might be broken, but I need to look at the c= ode (and if my hunch seems correct) and maybe come up with a patch. You could try the "syskrb5" mount option, which avoids use of the keytab and allows "system operations that maintain the state" to use AUTH_SYS, while t= he rest (all involving file data) use Kerberos. (If my hunch is correct, this will not fix the problem, but might cause it to behave better.) rick ps: I accidentally did this as an email reply... --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Dec 24 23:31:01 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Syy3900ctz54n8f for ; Sun, 24 Dec 2023 23:31:17 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Syy38535Rz3f2x for ; Sun, 24 Dec 2023 23:31:16 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-a265d4f78bfso291402366b.0 for ; Sun, 24 Dec 2023 15:31:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1703460672; x=1704065472; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=9XuYINp1Lor0EZClZQRpkhplAORR5Ng6sa3bC+gYuHw=; b=HKVeZNqdCAjzJ/hVfLFGpbzlQdOwT9BMj58JhgcCg9I45CCS/2wDFSRzb/V6zNAnGD j4Q27c4R/c6MlE2ZYKmHfCrhKgkPrI6mfG5Ji+vJjcmLiS1V2BnRoFyf1M5STJjvioQ2 RCj+/P8q/UG2hF3/fo4q/KlI5t0TNnf0EzpHOL+txrkcOyURVoyrHQlLRKqDi2B/9p9d SxWKT4Z+4fTFBe+g4zSO1Vo63vK5Cbu1PU/XGUgyZE/7KBHmpQtoB5CLW+TN4kJVGpx3 zqAMdh+EkpPVdT2X9ZhSomc6+2j3kvILdNbaZFDj/q7+0Zr5yt8SvEM9p+W3i/s8t75r 4DFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703460672; x=1704065472; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9XuYINp1Lor0EZClZQRpkhplAORR5Ng6sa3bC+gYuHw=; b=DDNDWqG2/sT/h6JIAYrxESqyPSMRaTXlK912GU2rM4RabAPqEczyuZN/frDWMbzTyi orTdjKu2vkCtLp8NIaIWc5q2a+3UHR3eIyst+4kPT9abmtyX6wq7IXZd/Cw+hOe9X6Qy F9UqQ5+GexsypuSQIx08C7WaYkIZNi+35ltCHgeDoxZdxikdpx506pC+lXfBWdatB1ME fChAjt/jQOnbtpQMr6OUsjPXzSPO8v62LVE4ed/emBmjcOrvwOLogTFldnQAPoQOzQoo vF5BQXqNZJgyxTl18VtUV1hWszTHuwrl29x5yYd0BUzy1wYvMDjFXix8Q6V5M1w4/05h zTcA== X-Gm-Message-State: AOJu0YzW9JSTY/MkXRka2rx8Hs0M7Hah4QkZnCKMzoEYaPElpXRGUzsn ztKLIpqs2RyJwBheOt43f8LVzJxTroVH82OrB9DiJ/WSEYc3F2RnGDdGyQlg4SI= X-Google-Smtp-Source: AGHT+IGulrRjd7mvj5MFmraMi+kjCK92l8gnKYOKXx7Ovj9Z4/rwGiNGgf7L6CLTIiBJ7V8c8PQfYaAppt+Gn8nhzvQ= X-Received: by 2002:a17:906:2210:b0:a26:c2ed:851b with SMTP id s16-20020a170906221000b00a26c2ed851bmr2023158ejs.28.1703460672579; Sun, 24 Dec 2023 15:31:12 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <2dcc21bb-f446-41fa-871b-2c37cdf9afe0@app.fastmail.com> <0ac6ceb8-cdf4-49f7-99c3-026a56527fd3@app.fastmail.com> In-Reply-To: <0ac6ceb8-cdf4-49f7-99c3-026a56527fd3@app.fastmail.com> From: Warner Losh Date: Sun, 24 Dec 2023 16:31:01 -0700 Message-ID: Subject: Re: measuring swap partition speed To: void Cc: freebsd-fs Content-Type: multipart/alternative; boundary="000000000000473a71060d49d8a2" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Syy38535Rz3f2x --000000000000473a71060d49d8a2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Dec 24, 2023, 4:17=E2=80=AFAM void wrote: > On Sat, 23 Dec 2023, at 21:19, Warner Losh wrote: > > > 5s+ of latency on the average, max latency of 12s! Woof. No wonder you > > hate life. > > LOL! > > Thing is, it's not all the time. But when it happens, it tends to stay > happening, unless something like swapflush runs or if I manually swapoff > then on. > Yea. I think that the swapping tunables may need tweaking... if you must swap. The swapin/swapoff evictions suggest as much. But my fu for that is at least 20 years stale. There may be tuneables in tuning(9) related to vm that might help. Warner > --000000000000473a71060d49d8a2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Sun, Dec 24, 2023, 4:17=E2=80=AFAM void <void@f-m.fm> wrote:
On Sat, 23 Dec 2023, at 21:19, Warner Losh wrote:

> 5s+ of latency on the average, max latency of 12s!=C2=A0 Woof. No wond= er you
> hate life.

LOL!

Thing is, it's not all the time. But when it happens, it tends to stay = happening, unless something like swapflush runs or if I manually swapoff th= en on.

Yea. I think that the swapping tunables may need tweaking... if you m= ust swap. The swapin/swapoff evictions suggest as much. But my fu for that = is at least 20 years stale. There may be tuneables in tuning(9) related to = vm that might help.

Warn= er
--000000000000473a71060d49d8a2-- From nobody Mon Dec 25 00:16:54 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Syz3q0CjQz54tVy for ; Mon, 25 Dec 2023 00:16:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Syz3p5TVKz4FGd for ; Mon, 25 Dec 2023 00:16:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703463414; a=rsa-sha256; cv=none; b=yAa2XajA14uWb/UloHmpN8Y6vcYTh4FjIMV07maMoA+cP8KDKM/cokGZ/4o9yuRC4Eiico LU1h8HoIEhYwLsSPu6CPcjGSuDVHBT0hM0wAtMj0YsaZgcu1r4pn5euxPTkNR5YrtIpowg aZll+2IVY4ssEjzO8cl06wyYZfUOZ1t07kth5ka2BX9Jru2JTnKEXQxcrkxkL6/um/Qb8/ vfykMiSP+Rbw+zkDF/9qlKKOWvOxAVG9SeE3tfUWr5NaL0M2rimbZgUU7SYYXRSYT5kloB A4cjdj/WIzDF/wSls2e/5ahhnLmzu+VhbRcGQw/+BCWZVPWpG+pGmDkVLtm8AA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703463414; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fnLQBew42rk9jkAASDCGcHEbUvmLAw0EfxmNxE4Pjds=; b=QqpIEXFSngzcGrdHrV4v/EkrdqqeMsZ7DhBCOuew3MY0KjHFjohvGEFFFGC54GEhmiNYNo fDtRWwzrhDsJmd8M3jhiheBZdM3FpU8LnPvrKpa/Gx9vgjQ+u5wZAoY8obFAVmOpzf/7u5 8pq/yEBW8mMONl6Rw9mKhwvm22Y+zgkBAjyeTjsI6FNeA70W7SdSVUNsxES56KgyAMyClS JW7VH5v0/IuGtBRtVpFO0YKuD/pjXyDtgpi7vbh2z6kSR7sUmeTb605J8fq/N2mpI0WBps qEGgjUIb0O9QiMk2O25rMAIxi9lDzJ3M37fmDeopAU6yJjFb4+6TS4AHlIYC+Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Syz3p4RfJzq1F for ; Mon, 25 Dec 2023 00:16:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BP0GsbN086521 for ; Mon, 25 Dec 2023 00:16:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BP0Gs88086520 for fs@FreeBSD.org; Mon, 25 Dec 2023 00:16:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275905] nfs client: mount becomes unresponsive Date: Mon, 25 Dec 2023 00:16:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275905 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|fs@FreeBSD.org |rmacklem@FreeBSD.org --- Comment #7 from Rick Macklem --- Created attachment 247232 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D247232&action= =3Dedit Free a session slot when a server replies RPC_AUTHERR This patch might fix the problem. At least it will give us a little more information w.r.t. what is causing the hang. If the reporter can test it on their client, that could be useful. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Dec 25 09:27:32 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SzCHX4rtmz54NN8 for ; Mon, 25 Dec 2023 09:27:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-22.consmr.mail.gq1.yahoo.com (sonic317-22.consmr.mail.gq1.yahoo.com [98.137.66.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SzCHV5FfMz4KXs for ; Mon, 25 Dec 2023 09:27:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=QuiqlKRl; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703496467; bh=rG+1DyWJUAe18FaF7WwbA4y4KTo6Qk0a+nzmcj6nUd8=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=QuiqlKRlnoSK5kMPcOF8D85kxKFUKsgw9DfjGPtyuv5I7okm8bFWxukmvREedOohPPAjykL/FG/p9JlrsYaQrab9dJ/YjI0ZmUszu6qTwHtfEvSRW3gqyyoLYuufYU3xCCppMgK7ka8OI+3c57+ToaJBKo9j3oGyxrYp3WNCw5XeZzDizcJhV7WCUJNTRqxT400/32/si/6QJsBH4pQWOsm5WxGKV0scqLrvboLulZKaECFtyca7KIOYu+j7FtJ4WzNkrRdpVbsUKe3mQ+5IUIAFAGpdPFlKhwmou8/3shPn/FvkkBHS0bi0j3+RWFQDvzMwHVPWCt4/sLuYc2W+4g== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703496467; bh=ROOEal+EzkzYx3lb61VKMKRTmoA2pN4axNoZ2YmrPRs=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=aVkbr1AvtlwXdFxlNmfm9aeoJ4FAHpp7hgI4i2Num9FRJvr3+Aygl+4z90AAQqtj597tUgVHC3CaCaPZBySDjzsJ2t9mifAh3PfTo+rH7DcFIrpYLgNqmfSG/Hi7Uahm9CgNshV/us1Vqtyt6GzeAoNw7vW3XpE2n3smP95iNlCJn4aEXO6bztZkf5kbit9LCEx5eNtCk88MZpXkRHq86mnLecs/IQmG+6OPlmvAkjW8WOG1mVOvfNT7lZv2yHK31UoZLgaqecnYkog/aPjnGU9QVkxPuRvnz2hz2OuyNuBKOc6YVQSMNbHXoN9L10pHTQzb61WpOprUBY45kTR16A== X-YMail-OSG: utTIteoVM1luglpmSs.6n9VcFSfgT_sPijOTpdNmb8o7jVH_O1ZZ90pnmNPkK7P EjOxzc8kAcVUZ8LaoQT6JmRYYiKVgEINwwuc8CC3TCm.CoOC8ZrSxMVSMKkPDbxqrc1tFnG2._vn XrEjt6RUnS70SWIEx2vV_USXeeVJvz5monoDT73p4Z5b377NDldfSQwaLp8GGmGpUl7gy37LbbKS pIRmh1f0uX3Klz.DvXFZSdRBpKW0IfpYvaug0gZRjbgYM8WFVn8UGMr.XrkEyx0VEmnvsCGpeEAt 03Th9WYXsphwlrhYu1mgxjHjpU2J2O8PqI7shH0TQa9kT9oTwcvcFdzkD00SebnvWmLAjC66cquw LgXr55AP2ON1MLf2DVaHI_Ao8h5emwBvjWdn8dYMnJqSdEnJWWoqarl3gdIrmnLNFXm9ZGjOswBA bKWihqPnTpquKyRYaBMgqGAfkrr7Nr4wjpfQb6_ZkMN1kR7YK90TQhjmxew1d5Fq.b7REU_Bcu3J CgpQH8KlGA7V1QkPghXFOwUH6KgCnQHW7Xuk_6ZayGN3imN5mS1yg5tXWYs3DuBI5DrlEdyna6OR T4_e.vkS8Eh_o_uDp6H.ZQ0SYhfbpxlK7I8mc_OSDAeFdfzLVtjQOaKE4jxw3ITtWZzq.7H7rMT3 QnXLvPCE7psYHsnpwYcNITbCKcrXorX4dJDy0eGASI1vT5qQwMHJhzBUfnSy5NC7W0d5cJ0Y4lVv Ve622jxMosbvfdfC.QgYs4ovhJRv6sXqB_urHZI08bxeC6V0xfXwYXBq.w_bgNanP8Chs1N4IltN .cbVbE9TIoE2Q4IvYQ7RU_9EaAEZ94lT068n8qPk1kVAVayydepsiQiO.nhr73m0mX98ird8sT_B eJazJJKIfx8RnH6dovrII88lby9FJu2.biwHvL9cYGk3ik0MwVAQmNbJGeW9yy5LePzUO7vSnGr8 pRlgSlCHiIRxNDzqB9llCAXHPcVGJBk8VIo3YyTd2Kp2nyPSOcaEQWGsMZRu8PAaGEbD.8hAAa7S i0VqqD8GkL0AwuX5hBFzhTFih6GUgU3.Z9LkQ7LSOSlxVS422XP5r501eVNekItxUBPYe1vzga.T YEwK9_FGOB5nnghVi_8M6MbS1xqP8ObPipPBSmqrmydCgX0I1XmyeF4A1sla8Li.XJqZPdT96ZMR 8pOsMak86aRCuGBqsX.mwisK9p14tSBb5OMDLCfZ2tyfufmQBF.BWZPtM_fxzgelCfT1o3.RsLB1 2j1ehf.D24f0cqGB9WQ141FhVUMaIhV3TALdUa_uCXBOsyjb.SnetDFBzIHoYYSBA6DYo3ySywsG 4NdnvjzbUXal3mY_koVxE4p.f6zLnz0vyCNiXICNxdZ43uAGDse9dHTioIxkFj2lrkor_J7Zd22o DBKk6opa03BgXJegouRNLfTyMYLBUbLeMW3nR4Yj83h66WTBgB63mpJq8.hctf2drQIEsFpBL4eU 0MCz9zGjFsMrMSF95KdpyWXUhELxph1zALZaSWkETeivHvI_WSSKCHGWusJypLXCO1rhxV.eilSB 1aiEAfuBJt.jTBLO.PlaSnaOy.VAnGUVcQyYGdjpQYD3xvKpRjg_ewA7WQez1cuY94btAgK_OOS3 pLJMrLFP21eC2DY2vEm_AvP5zkbhhbdn6yNHXinlI66zlGe1jyTRXrC7vYKU_ZzZ3EWklYNb5Yfh _INZr5I8fIB6LB4VyO7DLXb3Consyr_RK9Li_7kOt62VFxcVTvQXgxE85LjXsJAHDvGd9PZQNQTY D7WUT9ipPwv13H5WNlpnQtRjfHNNPZW3T_TgHVfpPQIIr2moWDuwV.P8C7b_8n8c0MESj5t87ZRx yv2HtpJYGb9EMl2NziBRiIkhEJU4wMuicZ4oLRiMx1VdYqgedoIAhEsqI47qtICxhXPl4lvIMcfB FBESpIQ30ArxOT2C9kUWMPkHtdmhY5oQaHR0Z4jXhmrUBuG3as6hAUTaNG.04HXJ.h50w9l0NTLg aEfXDxupgWdLsSIuZXIl1qpVXIpBvqcB.s6Wkhr7W0qJ0HK7YROXGhLmPR1ZG_7soCLTNQGvCdcA g.fXxY9PcVOdjl5.svqYyz0.rg7sfTTVdqSQ9i3NXc3V1CpkSWqiC34zsTvQIvsYiafx2Bk4kjf_ QFNxr_6GtPi63WxarJ_Jl7E0Byr9YMBIlKqyFmt6P8Qhn.1ctb..PIPd_uxpy5E9OezelmL2jNXP dyJ2EDUkW76f7FSyT1mCjsYjRCZfTKPwbPxZKdiBpMLDRpAqknQqhMbGuAar.1Xe2uWSLEaqHll. m X-Sonic-MF: X-Sonic-ID: 0b34c4b1-bba0-4000-952d-ce6a002e3c48 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Mon, 25 Dec 2023 09:27:47 +0000 Received: by hermes--production-gq1-6949d6d8f9-c9pk7 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID c26a4972b6540f9c331e1844d2af1852; Mon, 25 Dec 2023 09:27:43 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: measuring swap partition speed Message-Id: Date: Mon, 25 Dec 2023 01:27:32 -0800 To: void , freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3774.300.61.1.2) References: X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.148:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.148:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[f-m.fm,freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SzCHV5FfMz4KXs X-Spamd-Bar: --- void wrote on Date: Sat, 23 Dec 2023 12:36:50 UTC : > On Fri, 22 Dec 2023, at 20:08, Mark Millard wrote: > > I normally use -j$(sysctl -n hw.ncpu) for installworld but, > > if I gather right, you effectively used -j1 (implicit), so > > I'll do both styles, -j4 being appropriate for RPi4B's. > > Interesting. Advice long ago was to not use -j for the installworld part > so I've never used it at that stage. https://cgit.freebsd.org/src/commit/?id=ad6481dddcaa reports: author Cy Schubert 2017-06-28 19:05:04 +0000 committer Cy Schubert 2017-06-28 19:05:04 +0000 . . . src-ad6481dddcaab3363fa5f76489ba8ec11d5a235e.zip Allow parallel installworld (-j N) and poudriere installworld (poudriere jail -c and poudriere jail -u) to proceed. I'll note that at the time: #define __FreeBSD_version 1200036 was in use. But even older is . . . FreeBSD 7.0-RELEASE was the first where the man 7 build page reported: installworld Install everything built by a preceding buildworld step into the directory hierarchy pointed to by make(1) variable DESTDIR. If installing onto an NFS file system and running make(1) with the -j option, make sure that rpc.lockd(8) is running on both client and server. See rc.conf(5) on how to make it start at boot time. So, as near as I can tell, only bugs have interfered with use of -jN for installworld since sometime back. It has been some time since I last ran into problems with such -jN use for installworld ( or installkernel ). Also, if one is installing to a (sub-)directory tree that is not the live boot environment's tree, there are fewer potential issues. I have such for chroot use and poudriere jail use. === Mark Millard marklmi at yahoo.com From nobody Mon Dec 25 10:17:35 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SzDNv73SNz54S2F for ; Mon, 25 Dec 2023 10:17:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SzDNv60zQz4PP1 for ; Mon, 25 Dec 2023 10:17:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703499455; a=rsa-sha256; cv=none; b=GZJUYjckrNCP6SPv0XICpFzmUX5B8xRL4+ZcURWOfbOZVTVViSMYp6VO1UVrsC1DpqYWyD 5IKf+NFCV1nbHHTDwjKKCpW1usv1PJV0YPQSBXNNkcp0u89IPUkcPEr0I+szEFft63uyoV TOXd3Wwt1pHNdppt4KkefLdBl5SpafnMUv1ULPlIHRtppmPwOuyx6UBZWle8UiI0YWpjVj PNyE05qHJQkCIUpxSpzp9p72GQzL2W2zcFfLHdadFBQ9cw4uRWAmJngUPKXJD7YK2aZGHC c+566wj2P3uHvdpC6hN8zQ9ZNXf3xVCBAvO7gsg0TJGozaluz5PtT2QGg0BBIA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703499455; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A95Xn5ggOQjzHWwA2BkNHA491qiN7a8bp3xK0YRKtf0=; b=pIf1FBOgZENSgT8+gJYhhGwTHOynGuO5F/06k6RtAljfyEW2yHw+Q8ef1CheL9cbtTJaL2 vP2Dfy3tMm/eLFBTCt/xqGTC9/tFKi6cSddzFWxMSmEiJL4LQ5EbHqqVHNrLGg/I3IEYRv Ea2riype5Eha4NCSDxtzuuGy17q/Cf5JnHvG2O4MGeoew5+HnmLvXCJ/EhraXctw9PYYz4 eSFu5ZeGkyTiZeJO+Vc0pythSsOOqqaWd/b23+ZnEfDCP3VDEXyDo7SjgWbEpM4p8mzAEd aXATYOpKB6DvqqJ5XTYkGTdY0FxVhqPdkZr06jOsku6mdGKwHm2Z/TePQO5QwA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SzDNv4yPCz16TB for ; Mon, 25 Dec 2023 10:17:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BPAHZkK003765 for ; Mon, 25 Dec 2023 10:17:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BPAHZxB003764 for fs@FreeBSD.org; Mon, 25 Dec 2023 10:17:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Mon, 25 Dec 2023 10:17:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #13 from Seigo Tanimura --- Created attachment 247238 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D247238&action= =3Dedit The measurement result charts for Comment 13. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Dec 25 10:18:23 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SzDPr10yBz54SJN for ; Mon, 25 Dec 2023 10:18:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SzDPq7452z4Pt1 for ; Mon, 25 Dec 2023 10:18:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703499504; a=rsa-sha256; cv=none; b=I7XhosapgvPPNudBIosTNRGiWVPzyfLJ5EpOWmNu0SWcxpn1oQx7sx2P0cL3+RkXpEJUlE n3iYhxwDCjUtCDNHB3EQyhRzTI33Fx9qBL9fUGYBEY0gzJ+zCIBBiv2RvfLcUm71dHTsI1 IEOW5kNeOaBFNBt+VgSsMB5w1aCQImKio3rwh63X7ZR7Wov6RrK7tgJ8BZG9I2cUE/jPg7 BxHdEaX9Qa/aeoO7FOEwxHkWyQ4Rhx2S4zEO9xz8btM6YSpW9msIFaxbsZzUDMaUKJrfNj i+TIwmBcoP38Wwwcfvo7PhhzexbES/lnkqOAv9fZfSSc25trQ81xq+dpm0hDTw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703499504; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4oJu6vZklGfCMxiOUwF1arq3RH8hx9UVJBpL+LRsK68=; b=bL5L0PrGztmRLG0kE+nmETUtTzyPgpAcNj76glC3iT375io+T400NaZxEC4vN0bNW8auL1 IuLvwtl9uYq63w/WWNfRurzv+O1gANYXxr8Jp9efkFpO9zEZENT5IMrbp6VcGxW+o62Ubn qKF1u2yljwmy8qJHz00RTQRGFZteXdta4TD4t5XEBfKr4TgqweUkuZPPrY2QobQdcuhs06 FbsgK1zZGbzjPm6vr9h2UA8F9CKm4YOMwXRqWvyFiRN6Zvjfg+qq5hO+3TTNvACP0DHseH ja1HeU5IdVbSAQcC+8be5KOSj7IKP4lBmedeIQQufQKosqI0tqS8A8i9riRWJQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SzDPq683rz16TC for ; Mon, 25 Dec 2023 10:18:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BPAINSF004176 for ; Mon, 25 Dec 2023 10:18:23 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BPAINI4004175 for fs@FreeBSD.org; Mon, 25 Dec 2023 10:18:23 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Mon, 25 Dec 2023 10:18:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.description Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 Seigo Tanimura changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #247238|The measurement result |The measurement result description|charts for Comment 13. |charts for Comment 14. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Dec 25 10:21:27 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SzDTM5fm7z54SGd for ; Mon, 25 Dec 2023 10:21:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SzDTM2SpDz4QBK for ; Mon, 25 Dec 2023 10:21:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703499687; a=rsa-sha256; cv=none; b=xUWr5FV3K1Zj6yv29meWVBM3zCBDIOmqiW71SJb+fbLEtsKfSQDIrSWScgs3WXy8s8ATh+ aeXs8fv3q/6j5+xBZie7xBy8G0hA6fk3ixQ1bKN3texj/eqKUchQU2kOZss8yQ8iWrgAb4 A8/xNwQzVFTnO2nNHA132K4SI69rik985u/j0pRrfezEty/scZfH4oHpffP/z/qi27CK2h NXpezIVLIMKd7rmAPVc2j/UywaCxntegLOfex3FAzWEh6p7bkHrhLIHPi5rBGUkDPa5Ywu RuZAinznfEsSdB8QcIlCpQRhNGvcFxC8sruLvtsI4Uxjnj/ga89VG5IQwy+OIg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703499687; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w8Z10sCuzYU1DLQd7hDJFlsYDsaoRkfRfaQwL4/8vH8=; b=eedlDxCNUO5hGREi5Mivt1EUjifiTNQmQiIMlENKzr/23vu5NLo+eqVBk1f0hvgoc7eYo2 LbTK8khp8RKyBYJTSXEO5lHKfe48ygvy5bsdWIsoKfmYEcFvQcg84H7UnSrD6PM1/JA73W qrQFk3EHS1nryH1cTZJQp5smkFR5m56a5g2msM2K8LDpQbcKJlJV+fBlX0Ulrafhm5uQGf smDC+k4rdfziEX1lvSmwh8qhkskXnY/xu/HmNr2tPmw5CJnNpPQZB6aNobM9FKWqdFqIqr 2KCNcFwhfF7SN68TkxjGx/o+VKM94TrJhFEO+CkFtBrkrttMOtPTzCFa3QxLhA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SzDTM1Xcqz16fn for ; Mon, 25 Dec 2023 10:21:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BPALRBd010156 for ; Mon, 25 Dec 2023 10:21:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BPALRFu010155 for fs@FreeBSD.org; Mon, 25 Dec 2023 10:21:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Mon, 25 Dec 2023 10:21:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #14 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #12) Thanks for your patience, I have spent quite some days to set up the more precise and easy measurement by Fluent Bit, Elasticsearch and Kibana. It is now working somehow and ready to share the test results and the analysis on them. ----- Sources on GitHub: - FreeBSD - Repo - https://github.com/altimeter-130ft/freebsd-freebsd-src - Branches - release/14.0.0/release-14_0_0-p2-topic-openzfs-arc_prune-interval-fix - No changes since the last test. - release/14.0.0/release-14_0_0-p2-topic-openzfs-arc_prune-interval-counters - Add the vnode tag counting. (disabled by default) - Refactor the ZFS ARC stats for the precise investigation. - The changes may be squashed or removed in the future. - Fluent Bit - Repo - https://github.com/altimeter-130ft/fluent-fluent-bit - Branch - topic-plugin-in-sysctl - The input plugin for sysctl(3). - The other fixes included. - No documents yet. - Note - "-DFLB_CORO_STACK_SIZE=3D1048576" recommended on cmake(1). - The default coroutine thread stack size is only a few kilobytes= on FreeBSD. ----- Test Summary: - Date: 21 Dec 2023 17:50Z - 22 Dec 2023 00:50Z - Build time: 07:00:59 (304 pkgs / hr) - Failed ports: 3 - Setup - vfs.vnode.vnlru.max_free_per_call: 4000000 (=3D=3D vfs.vnode.vnlru.max_free_per_call) - vfs.zfs.arc.prune_interval: 1000 (my fix for arc_prune interval enabled) - vfs.vnode.vnlru.extra_recycle: 1 (extra vnode recycle fix enabled) - vfs.zfs.arc.dnode_limit=3D2684354560 (2.5G, larger than the max actual = value observed so far) ----- Findings Summary: - Vnode Behavior - The vnodes circulate between the creation and pruning. - Almost all of the unrecyclable vnodes have one use count, which makes o= ne extra hold count as well. - ZFS Behaviour - The direct trigger of the ARC pruning is the low ARC meta value. - There is a massaive amount of the ghost data hits. ----- Finding Detail: The discussion below refer to the charts in the archive attached as poudriere-bulk-2023-12-22_02h55m25s. The times on the chart horizontal axes are in the UTC. - Vnode Behavior - The vnodes circulate between the creation and pruning. * Charts: number-of-vnodes.png, zfs-vnode-free-calls.png, zfs-freed-vnodes.= png. Although zfs-freed-vnodes.png shows the steady vnode recycling in some time sections, the number of the vnodes is almost flat. (Typical case: 23:20Z - 00:40Z) This means that the vnodes are both recycled and created at almost= the same rate. - Almost all of the unrecyclable vnodes have one use count, which makes o= ne extra hold count as well. * Charts: zfs-vnode-recycle-phase2-reg-retries.png, zfs-vnode-recycle-phase2-dir-retries.png. The non-zero traces on the charts are the overlaps of: - free_phase2_retry_*: the skipped vnodes, - free_phase2_retry_*_inuse: the vnodes with the non-zero v_usecount, and - free_phase2_retry_*_unknown1: the vnodes with one unknown extra v_holdc= nt. The non-zero v_usecount is sufficient to explain the "unknown" extra one v_holdcnt. The current vnode recycling code does not work on the vnodes wi= th the non-zero v_usecount, and one v_usecount makes one v_holdcnt on FreeBSD. - ZFS Behaviour - The direct trigger of the ARC pruning is the low ARC meta value. * Charts: zfs-arc-meta.png, zfs-arc-dnode-size.png. arc_evict() triggers the async ARC pruning when either of the following conditions stand: A) More than 3/4 of the wanted metadata size target in the ARC is not evictable, or B) The size allocated for the dnode exceeds vfs.zfs.arc.dnode_limit. B) is not the case in this test; the max ZFS ARC dnode size was ~1.6GB, whi= ch was less than vfs.zfs.arc.dnode_limit. (2.5GB) The ZFS ARC meta is the ratio of the wanted metadata size with respect to t= he whole ARC size, initialized to 25% scaled up to 1G as the 32bit fixed point decimal. The higher (lower) value means the more (less) size for the ARC metadata. The reason for the sharp drop of this value is discussed next. - There is a massaive amount of the ghost data hits. * Charts: zfs-arc-mru-ghost-data-hits.png, zfs-arc-mfu-ghost-data-hits.png, zfs-arc-mru-ghost-metadata-hits.png, zfs-arc-mfu-ghost-metadata-hits.png. Please watch out for the vertical axes of the charts: - The "hits" are the sizes, not the counts in ZFS. - The scales of the data hits are about 10 times larger than the metadata h= its. The ZFS ARC meta value is automatically tuned by the sizes of the ghost hits passed to arc_evict_adj(), called in arc_evict(). The idea is to favour ei= ther the data or metadata experiencing the more ghost hits. The ghost hits on the MFU and MRU data is about 10 times larger than their counterparts on the metadata. This explains the sharp drop of the ZFS ARC = meta starting at about 18:20Z. Also, the large size of the ghost data hits happ= ened occasionally, while the ghost metadata hits seldom exceeded the data. This shows that the ZFS ARC meta stayed low until the end of the build test. ----- Next Work: - Check if the vnodes with the non-zero v_usecount are actually open(2)ed. - Compare the count to kern.openfiles. - Estimation out of the current results: - Max free_phase2_retry_reg_inuse per 10 minutes: ~270K @ 00:20Z. - The ZFS vnode free call period at the time above: ~1.57 [s]. (<=3D 1 / (382.5 / (10 * 60))) - The hits into the vnodes in use per ZFS vnode free call: ~707. (<=3D = 270K / (10 * 60) * 1.57) - If this figure is reasonably close to kern.openfiles, the phase2 retr= ies are indeed caused by the open(2)ed vnodes and such the vnodes cannot be evicted. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Dec 25 10:57:17 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SzFGk6sRbz54Vdh for ; Mon, 25 Dec 2023 10:57:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SzFGk5cTKz4WZg for ; Mon, 25 Dec 2023 10:57:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703501838; a=rsa-sha256; cv=none; b=o77JzumOLMDfsa1JHzrZvYMsMGTL8aGv4GIgSIFeTp9J1zATf826UopI+Xli4AbsWYOy2h Wm616d0cqRPoVpFYyK4opbEbGAQ5HpnMTmnA4sGgTJhzKFwpTyLnwq9CJQOg91lPr5M/0z JHSzTk3H0+yo28N5ARd6x490e0Kt28VZJmxjfD3XQSxLWCFeihiFEu716rBVLl4yVCmaAe Q3kMvj5/5czqBD5asHMVxzCS1JDKhvuDXlXB1+wwtHHXnBIrgRuG1xuTsQH+QIJrMIYH70 ZeX/BXm/F/sirXH94DbfOKHPlWy1MTzBl3XWCQDDpE+kEPUD+Ain8AvxwjOfrw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703501838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eKV/zo3MLn3Rj1IXwnl+ahhgVCVfvWTGFD6emuINcX4=; b=VRHaxtirS/Xz07NWBogXr8a47Ox4XK6PEnN8HWz59w3yAxoDOxTjhHmWYXp0ZTUycE4OM4 8ZFwi2vp8rFMbNq65wdKffMuBvy1kyMTvKl1mswCSGAcLvVYzRyab0g326eiLbDO3KBCOJ YGPnt3gT1zFsmdNjN5nt+5VN3QkE7m234G6jnETtU35MMShHV0WLNfoBpoapF3Y8DQWm4U N9yNsFEOprwNUdJTkI0MpawTDV49jgzHMbwGYbdv29dBVJT9X9Y8emDND1W+hTbwGC8iRr 8PPjyj5XHqk7kxXX6CX3gAWCo6pLyh+oMv66HP+p+7EwkJjXufIq1xxFKwNpFw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SzFGk4jHFz17BC for ; Mon, 25 Dec 2023 10:57:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BPAvI93064582 for ; Mon, 25 Dec 2023 10:57:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BPAvIRI064581 for fs@FreeBSD.org; Mon, 25 Dec 2023 10:57:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Mon, 25 Dec 2023 10:57:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #15 from Seigo Tanimura --- Created attachment 247245 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D247245&action= =3Dedit The additional measurement result charts for Comment 16. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Dec 25 10:58:46 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SzFJQ50cwz54WFt for ; Mon, 25 Dec 2023 10:58:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SzFJQ3y8jz4Wm7 for ; Mon, 25 Dec 2023 10:58:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703501926; a=rsa-sha256; cv=none; b=CGLXKLxbxKdeHteu2sESoHPDPu2l5MA1Ve+zIouMVCuUFUXSSWMp5ccW7ZIH/EGlCu3kJL GIS2r+zeo2llijctJcTFAs59L5l7auo1ugWeraGG5GMfcLeaGJhCaJsLR1NzWFEh+v2uvm o2+TSTEi2wY0WX/yUae32nXpLbrKw2oXxrZccH8QQTU+UhLpIAWGWSDxmU6t5ZBNRR3V43 ihQ6yBem2BG8SkC6JD1WeU50v6dhxn0mqKOyOqM8N8mLNS+Y0pvGejAt5+8d7hkuuFeQDl v+6+b7bXc0bizUc/0vgrhP4CM6cMeb2PTxWsbTJib1hp717Ays5CEZDJtQqv0Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703501926; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KuHwaI/PsAWe4cF0ylPzVBvxYm3VHNvPud+/PBgLEw4=; b=abSdjfEcCluzh4hj+QPZKrWLzVIm8IloSxL7waJuqy/askkRGHxMvx01ulQb5KbNg+euND UeRaBg3qzJsRv9Vt2nh6wcpcHupF3iZwJzyK2ha9Ej+qKD8pazQ3dQxOnerMydosWoLMDi Yzb7AOUpu9v+GkcCgkAHD3riDek6EAiLuYx63nhilZrMGs3ygiuL7F24fxEl38QixeZp6G jGyMXhHQXWtg5PYSD5B16mYBtDntLQbQIRrY/khFBvaVol6C8Eb6rwi3yD2XuzWNBK0PAi 6HYllzoRYGEr88ppnwnBknHEONprVETMCOvkhlEHAQXvgZkVxBOm1kGkGZvL3w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SzFJQ2pMBz17dy for ; Mon, 25 Dec 2023 10:58:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BPAwkdB065046 for ; Mon, 25 Dec 2023 10:58:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BPAwkYW065045 for fs@FreeBSD.org; Mon, 25 Dec 2023 10:58:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Mon, 25 Dec 2023 10:58:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #16 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #14) * Notes on the comparision to kern.openfiles I will assume that ~70% of kern.openfiles is for ZFS in the next test. This ratio has been observed from a preliminary test conducted separately from t= he last one, where the vnode tag counting in enabled. * Chart Archive: poudriere-bulk-2023-12-21_18h42m38s.7z * Charts: vnode-vnodes.png, vnode-tags.png. vnode-tags.png shows the vnode counts classified by its tags. The section = of 21 Dec 2023 15:00Z - 16:30Z has ~2.5M vnodes, out of which ~1.8M are for ZF= S. (The yellow area on vnode-vnodes.png) 2.5M / 1.8M =3D=3D 0.72. The vnode tag counting will be disabled in the next test (and so by default) because it puts the load as heavy as the ZFS ARC pruning and is hence likel= y to affect the test results. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Dec 26 14:22:48 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Szxnp13Dqz54w4s for ; Tue, 26 Dec 2023 14:23:10 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Szxnn0BpSz4HkR for ; Tue, 26 Dec 2023 14:23:08 +0000 (UTC) (envelope-from trashcan@ellael.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ellael.org header.s=dkim header.b=TZ3Ql6Ve; spf=pass (mx1.freebsd.org: domain of trashcan@ellael.org designates 91.121.41.56 as permitted sender) smtp.mailfrom=trashcan@ellael.org; dmarc=pass (policy=quarantine) header.from=ellael.org Received: from smtpclient.apple (p5b2e5c97.dip0.t-ipconnect.de [91.46.92.151]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 4Szxnb2z6Mzl10 for ; Tue, 26 Dec 2023 15:22:59 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ellael.org; s=dkim; t=1703600579; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fpHvpBgjCzeVUOt2e0mLonXMxZQHvr919YTvSZTMe1E=; b=TZ3Ql6VeGSsXies0jnfnjYiJ58jZVX2pTtl08t6UX+ZOiLJOmahxoHbPXpE2M++31j7zMv s7MO/QkfBCGqbzT5tBrVGe8dWRPinZ8cNYau50v/XIuQbkXbG+gsH3sU5FlpaNP9tyjLoA yeh01zi4eWageU+MxdJQcKR+rnTD9kRav3GcdVdqapSlfwhf6VMmO6Tqtgvdi8K+qJq6Bq 80bnSxkAMFoc2w5ZX9czvcrpNiy5sx7MklqHt36BCcQImbbICMdeXtA4XarkstVY4aWidb Rtvpi9LrJGbN8rInEAbgaee/m1VwJ/dRHazZOSbRMBz1TnfSvwINBm2Nil6H2Q== From: Michael Grimm Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Howto migrate a FreeBSD cloud instance to another one without native FreeBSD support? Message-Id: <4A6398FE-5DC1-45A1-85A9-7C5054CF5BF7@ellael.org> Date: Tue, 26 Dec 2023 15:22:48 +0100 To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Spamd-Result: default: False [-2.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; DMARC_POLICY_ALLOW(-0.50)[ellael.org,quarantine]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[ellael.org:s=dkim]; R_SPF_ALLOW(-0.20)[+ip4:91.121.41.56]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[ellael.org:+]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4Szxnn0BpSz4HkR X-Spamd-Bar: -- Hi, I do run FreeBSD 14.0-STABLE instances in OVH's public cloud for some = years already. Now, I do need to migrate my FBSD instances to new = instances (old ones are no longer supported). In the past one could create new instances running FBSD from the start, = but OVH has removed that option. I do have used Ubuntu rescue systems in = the past for my FBSD instances, thus I am familiar in using Ubuntu's = OpenZFS functionality. Sadly, OVH doesn't offer Linux systems installed on ZFS pools. Thus I do = have to stick to Linux systems without. Here are my questions regarding using Linux (Ubuntu) rescue systems: # What would be the closest equivalent to FBSD's gpart tool? gdisk, =E2=80=A6? # Would it be possible at all to go for legacy booting of FBSD?=20 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 # Or, would it be better/easier to consider UEFI booting instead? rsync /boot/loader.efi to rescue system # Are there other promising strategies for migrating feasible? Thanks in advance and regards, Michael= From nobody Tue Dec 26 17:47:39 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T02Km0hPjz55F4G for ; Tue, 26 Dec 2023 17:47:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T02Kl6lhJz4YSj for ; Tue, 26 Dec 2023 17:47:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703612859; a=rsa-sha256; cv=none; b=GG5qYyvR1SseY+e3xSrHUuid1/v2sZfmdOE7ag/hQbxPjIqZCn6bclG5Ds2UMMEELQpaQ2 caa8A+TYYtvUvWjobZm4RupIkjsoBJNkNEwJmxzwrO+iKQrQDySga57ImOXnslMED3xm5g xy9KdArg1j6rG5/kgfoLk4Hb0Dv4l1wGWarkOWiBu35G4wvT7PodLUEm572rm2Bdlk2XOX EbeoPtNTH9JQUcTBkg6qNPA+lq7WfYPRtoSE2ncaQD1OO8qpnKwPhaVcNxFICPbjllzwaJ i2xJXZQMMpfJJFovUS5Cgj6L7RORdglsCsWcRF6fOO3hshtlnAOzQS4zTWVP0A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703612859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qUDoNW/YKn+Lz8rvjrPW2CNnCc8HpKdiM3Zdl8LM7FU=; b=pHXwcsW9W6ZbX1mz8lmtH6AY65RuiYh5gU1rRVCPuT0zRPtzqcbZmigmLBzrA+NunYjRZY 6QSB5MmrUrjyPJqlmFcRY4x2zp+RVFLnx/O6I+5fG2TdXiGLvuzPbvBVE5Sm1JkCs8wAn9 EFb3DCxwS/2UfeD5W3yHEyLZDbN+JiBI3RzJ+yM85TNjGqy7HzCwoiM7Rk3uhhC+1W+ry/ tsPTLvkds8q+VidbT2tkDL6vuxK+Ao1ptTRkknNpIklLKhdFhIoP4IY4AcZF/7xYt9eMgR Wsk/QG+gp7/hQUZ8oCafu23KR6ydm1CqkWpDbDUaruzPlVMRTgY1RYFKjO2q4w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T02Kl5sVZz13nG for ; Tue, 26 Dec 2023 17:47:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BQHld4G056493 for ; Tue, 26 Dec 2023 17:47:39 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BQHldk4056492 for fs@FreeBSD.org; Tue, 26 Dec 2023 17:47:39 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Tue, 26 Dec 2023 17:47:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pi@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 Kurt Jaeger changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Dec 26 17:47:49 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T02Kx1SGjz55DhM for ; Tue, 26 Dec 2023 17:47:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T02Kx0R6dz4Z6Y for ; Tue, 26 Dec 2023 17:47:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703612869; a=rsa-sha256; cv=none; b=bcG20XA9JvrDDy0lU+XynSHrWF8xQpXxSPJ3mxzM8nMpf+8OkmeOpXehgMj1DVpi/EnFGm GlODocsD/mRaSp3MxT8y7v+YGuAX+XM+ydGel0sh8JDTC3iIy2qlwiyBwNsJ5x/GDZEGA0 6KqdnsrpOCbXExRTAoqMak9x1S26A99ErhqEJI9uNbmOGRGrcnJT05IZXs5SnlZwbzz6ja nw2htyCt9u56Zs3uhUb+oN5+XrTX/Mvr429nqrdADGn95COl2R0G7wULvjCnOIycll+n6g lW6tUH+pIOmybCrKdmStgOO0Qi0sKurWB9wxPj7gREOZSamQG9yKP0DKBdYYjw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703612869; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hroPquMHy6Nw1uC1CnyC4oZnk5Q9+PJ/STpzsglv1e0=; b=Cko0TrRXxazLKPFX2a/7N0rFXMftJhQQa+Qx91jIv1EGgyma2aoNQLmRaltM44V6U7YbGH mu5isO2H0IoHIJjJ2Bap/ET9yCUAYUmzxmwEhElfVlXS4NGzCHoCgelsOVpPnol+ppgyP4 qJatIepUua94pGm6aBJShwvctO2XVVI6Gjd794EP6X+9Jr1Ny6Y3Qf4o1xfUcJJu/kuXZg RTC/F75IB8lV6Kdyyj3WfJHiz4g8DsgEEJj7f9sgNWmwwi/5IOvnmlbsDSYIwbmR+4es1C ytzcCdcb5yC6o6xgE/Zfptt89Ho5SY8lIGDX9nSJGiBHxyQYiXkuOJHtm3Mvpw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T02Kw6fprz142Q for ; Tue, 26 Dec 2023 17:47:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BQHlmxB056535 for ; Tue, 26 Dec 2023 17:47:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BQHlmvx056534 for fs@FreeBSD.org; Tue, 26 Dec 2023 17:47:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Tue, 26 Dec 2023 17:47:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pi@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: version Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 Kurt Jaeger changed: What |Removed |Added ---------------------------------------------------------------------------- Version|13.0-STABLE |15.0-CURRENT --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Dec 26 17:47:58 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T02L65GnQz55DfB for ; Tue, 26 Dec 2023 17:47:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T02L60l48z4ZZh for ; Tue, 26 Dec 2023 17:47:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703612878; a=rsa-sha256; cv=none; b=gqmCYNxQr0L+Xvwee4XOmh99HEyoySmYDGK5+43zSfLaeQ4e+5BBK3x8pVsJ5tYHZYoFkD Ji/FB/LwsW9KfZAi2AGOUFton23u7CcRYeMGXAZGD7mC8TyyClV2xh8Fwex6ph7XCIapzc GwDF+IPcdDI9EfKNU08gGo0DmZx0DcZOAMPa09pviz1uDJbJh8Neimv4505qAn4hq2DiQU zMTcWd28uAp2q6mbyE3fHosJP9FVowSi+rAWzvjt4hPIpMtvxrNphIMNa2PSS47Bs7NAWF O41FKsK+hLBptGlLnfZLP/cmym6hBeFCgsmuAXxESwbEFgsVzq8TtkxFQDembw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703612878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RnYlSo/dSCaEdYg+QM2LvFbLlmAoZ2H93XWhw3vhXLo=; b=ij2jCRmd7W+lz7JcYUMl4EpuUw/G3i6HujJhE5VzW/ghMJxTA+bmmFaY+I/1wQaFgWW5+U RxEoAB5UmeSqyKNqX0QkmO2TrSwM24WN3tciviv0QGHIB47VWt91g7iVCKoT9DbfqlfZ/S 6ejzzlCSDsAcSSVoWCsrmSPDHD3L1iIuaqITnx7ewW4HtcO6caWVPpPm4Yrb5KGoRTshH1 BXoVHPxKr/ngslOpgrUDDAJSdVs/2dYaF1QLmvVYJykYlihJgEe9AhfKLckwRRT/Gehcb2 Wa5VzH6FOL+K4JwUVtR5fom1nPAsVFrZ6p7L3ymBWKmeGABxdNmtWsyyrsgMiA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T02L56xMJz142R for ; Tue, 26 Dec 2023 17:47:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BQHlvAC056587 for ; Tue, 26 Dec 2023 17:47:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BQHlvQj056586 for fs@FreeBSD.org; Tue, 26 Dec 2023 17:47:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Tue, 26 Dec 2023 17:47:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pi@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: rep_platform Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 Kurt Jaeger changed: What |Removed |Added ---------------------------------------------------------------------------- Hardware|arm64 |amd64 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Dec 26 18:19:21 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T032X1zkJz55H9N for ; Tue, 26 Dec 2023 18:19:32 +0000 (UTC) (envelope-from SRS0=2BNT=IF=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T032W6BdZz3Fdn for ; Tue, 26 Dec 2023 18:19:31 +0000 (UTC) (envelope-from SRS0=2BNT=IF=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 30831D788C; Tue, 26 Dec 2023 19:19:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quip.cz; s=private; t=1703614763; bh=6kt+U0dLuxF9Oc3LocdpACT3CgoYkYF5xq+evn3orNE=; h=Date:Subject:To:References:From:In-Reply-To; b=zppxDdi4noWl8bTFqW4lTNiy53+tTrC7Ub9meZkarQyuFCTgLlt3jW7AQIfhGtmqQ owQ0tKCYLeIJpil+AV0gjN7rjOVSzoSlgYvKwrvdMOnODQE+IszfmTMhTySX4u5i8e 3FFH9jhunFOSu/knhFqLH75BM7uvvALnvr+8pDtg= Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 2E77ED7887; Tue, 26 Dec 2023 19:19:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quip.cz; s=private; t=1703614762; bh=6kt+U0dLuxF9Oc3LocdpACT3CgoYkYF5xq+evn3orNE=; h=Date:Subject:To:References:From:In-Reply-To; b=W4s2d9/Clm35KKaPVwadnb8JbQy3lTxWlS6uBBqcOK/UXy1yCwXZSCi+VhKi6GZcs 7LJTWqOMqfp/ig4+n42dwBAG5Z0+a82VBxedGOmDddrj71C4cPrgxMZTEJhvvFOjiC /ZSOi286AMZrWbZpQAZ2u7B4AFOUsfQW3wc5Ahew= Message-ID: Date: Tue, 26 Dec 2023 19:19:21 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Howto migrate a FreeBSD cloud instance to another one without native FreeBSD support? To: Michael Grimm , freebsd-fs@freebsd.org References: <4A6398FE-5DC1-45A1-85A9-7C5054CF5BF7@ellael.org> Content-Language: cs-Cestina, en-US From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: <4A6398FE-5DC1-45A1-85A9-7C5054CF5BF7@ellael.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4T032W6BdZz3Fdn On 26/12/2023 15:22, Michael Grimm wrote: > Hi, > > I do run FreeBSD 14.0-STABLE instances in OVH's public cloud for some years already. Now, I do need to migrate my FBSD instances to new instances (old ones are no longer supported). > > In the past one could create new instances running FBSD from the start, but OVH has removed that option. I do have used Ubuntu rescue systems in the past for my FBSD instances, thus I am familiar in using Ubuntu's OpenZFS functionality. > > Sadly, OVH doesn't offer Linux systems installed on ZFS pools. Thus I do have to stick to Linux systems without. > > > Here are my questions regarding using Linux (Ubuntu) rescue systems: > > # What would be the closest equivalent to FBSD's gpart tool? > gdisk, …? > # Would it be possible at all to go for legacy booting of FBSD? > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 > > # Or, would it be better/easier to consider UEFI booting instead? > rsync /boot/loader.efi to rescue system > > # Are there other promising strategies for migrating feasible? I know nothing about OVH and almost nothing about what kind of FreeBSD setup you want to migrate (partitions, UFS vs ZFS etc.) but one thing I can imagine in you situation is creating everything from Linux rescue (partitions, formatting to UFS or ZFS, boot sector, rsync old FreeBSD instance to newly created...) And the second is: you can prepare small minimalistic FreeBSD image in Bhyve / VirtualBox etc. make a copy of it with dd and then write it by dd to your new instance's disk with the help of Linux rescue system. Then you can reboot to a minimalistic FreeBSD system, enlarge partition(s), use growfs or zpool / zfs enlargement and then finally move data from the old instance to this new instance by rsync, or zfs send, or whatever tool you prefer. Kind regards Miroslav Lachman From nobody Tue Dec 26 18:41:15 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T03Ws5JkCz55KRH for ; Tue, 26 Dec 2023 18:41:29 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T03Ws3RVxz3HJn for ; Tue, 26 Dec 2023 18:41:29 +0000 (UTC) (envelope-from trashcan@ellael.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (p5b2e5c97.dip0.t-ipconnect.de [91.46.92.151]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 4T03Wp2X3xzmS2; Tue, 26 Dec 2023 19:41:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ellael.org; s=dkim; t=1703616086; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uTqFAVp7EFJcWFBlOV/41veJaAJ7XSFSTeeGxwXP35M=; b=jO/GMZJLkXw/r3vzOEKXZnqICk4b+F9ytxgQznKgMoGUworYddGJx218cSwOPl1G60eJoj jwCLlFxrUfggOrcrFrVmvjOo6F5zMVd+ElxpW+9AEJO8ekEq1TN4/hso3ZIfYfrPBXeD5x mBqsPpPNumD4yCPD6yblJLdQwG3iLq08tvKHPfE8hwgYJTYZI+RGSBVQ/QM0yPcqBGayU6 3q/GXtz7nG2PcKXSCEoDBAwx+aNKCMpYefh3qRdM9nVMt2VhSPA/vsUiBimk7di5KgBSXL F/SgwxgTAVPf75Ach5MWRFkTbT3HNlL8S2+DdKEv4YuDQdjPJV2CxCHDD20TnA== Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: Howto migrate a FreeBSD cloud instance to another one without native FreeBSD support? From: Michael Grimm In-Reply-To: Date: Tue, 26 Dec 2023 19:41:15 +0100 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A6398FE-5DC1-45A1-85A9-7C5054CF5BF7@ellael.org> To: Miroslav Lachman <000.fbsd@quip.cz> X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4T03Ws3RVxz3HJn Miroslav Lachman <000.fbsd@quip.cz> wrote; > On 26/12/2023 15:22, Michael Grimm wrote: >> Hi, >> I do run FreeBSD 14.0-STABLE instances in OVH's public cloud for some = years already. Now, I do need to migrate my FBSD instances to new = instances (old ones are no longer supported). >> In the past one could create new instances running FBSD from the = start, but OVH has removed that option. I do have used Ubuntu rescue = systems in the past for my FBSD instances, thus I am familiar in using = Ubuntu's OpenZFS functionality. >> Sadly, OVH doesn't offer Linux systems installed on ZFS pools. Thus I = do have to stick to Linux systems without. >> Here are my questions regarding using Linux (Ubuntu) rescue systems: >> # What would be the closest equivalent to FBSD's gpart tool? >> gdisk, =E2=80=A6? >=20 >> # Would it be possible at all to go for legacy booting of FBSD? >> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 >> # Or, would it be better/easier to consider UEFI booting instead? >> rsync /boot/loader.efi to rescue system >> # Are there other promising strategies for migrating feasible? >=20 > I know nothing about OVH and almost nothing about what kind of FreeBSD = setup you want to migrate (partitions, UFS vs ZFS etc.) but one thing I = can imagine in you situation is creating everything from Linux rescue = (partitions, formatting to UFS or ZFS, boot sector, rsync old FreeBSD = instance to newly created...) Sorry, if I haven't been clear enough: #) I do want to migrate an existing ZFS pool (from one instance to = another (zfs send | zfs receive).=20 This I have done numerous times before. #) New to me is the preparation of the target disk out of an = Ubuntu/Linux rescue system. Namely, how to partition with Linux functionality (gdisk, cfdisk, = =E2=80=A6) lacking gpart? Namely, which kind of booting to use (legacy versus EFI)? > And the second is: you can prepare small minimalistic FreeBSD image in = Bhyve / VirtualBox etc. make a copy of it with dd and then write it by = dd to your new instance's disk with the help of Linux rescue system. = Then you can reboot to a minimalistic FreeBSD system, enlarge = partition(s), use growfs or zpool / zfs enlargement and then finally = move data from the old instance to this new instance by rsync, or zfs = send, or whatever tool you prefer. Thanks for this info, very much appreciated. Regards, Michael= From nobody Tue Dec 26 19:12:04 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T04C81nkRz55Mkh for ; Tue, 26 Dec 2023 19:12:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T04C76wdjz3KwP for ; Tue, 26 Dec 2023 19:12:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703617924; a=rsa-sha256; cv=none; b=i7R4KvTb+ywqOCEtZj0L972JIiG0hXdBBRvF2BZyZzOwh5MJF1Bnzcm/IvvtJJC7W8PNgQ +ti6Nxy1ZepK/CFm4JbEk77/kXxdjLcycuiD3eiz0IhuHvrUsWqguXy0j1+IqCSGFCScF/ KEhn28AA6vBjtoiKx2ezwNO640a3cuIE4bRABiSGIU+f/Xm7gnibavh3efc3N2hgeTU5xr 1ph3xYt1tGCZE5yNGr17lM+PL17H4dHFKLbbs1CRfB3UvKwX2AMzfOxIPiOWynsqEzzsNQ SgttnlNV8OqDGi5T3Y3qs0RBWPu4X3Q/Cuau888p+y3Wx1gboXH4H6GuJ50Zeg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703617924; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+DovPD1NpM8SF+Vxo7LxGUE56n+fWJxZJQ+1kXdmStI=; b=OIe+exhYs6Cp5zXFp9DN4w4nu3qGBGn2Qs+BZ106PaVILe2Yts5e4vNyh3xKkVtvfAohTx sEmb8rIzvZrI4s2ll5XVQL2YRkbQDqeHaAnfnTwJyHvpE4V7oETjKfxuTcMMpfNDsowCtL vWD/e2PsBpZswFThqhA0/rycS97FFc/pfnnp6iC83ToRQWWglYhc+t/pnTZQFNFp+iaAkD 6lfwun9HAt83wcoPY6ajVf9YvtSjh6WIeFYyfSlifhr/rQglzM3oCDS7XkzC1/pvLhq4LS gatorlm8m65jDZxLdOtG14axEuEurP1nPf6Coi4BPoe1+hb3DorDHGWKUzx4Eg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T04C760t2z16LP for ; Tue, 26 Dec 2023 19:12:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BQJC3YK093559 for ; Tue, 26 Dec 2023 19:12:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BQJC3T9093558 for fs@FreeBSD.org; Tue, 26 Dec 2023 19:12:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Tue, 26 Dec 2023 19:12:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_severity rep_platform Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 John F. Carr changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Affects Only Me |Affects Some People Hardware|amd64 |Any --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Dec 26 20:14:01 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T05Zl04yyz55S64 for ; Tue, 26 Dec 2023 20:14:07 +0000 (UTC) (envelope-from SRS0=2BNT=IF=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T05Zk4CtVz3RC4 for ; Tue, 26 Dec 2023 20:14:06 +0000 (UTC) (envelope-from SRS0=2BNT=IF=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 0BC54D7887; Tue, 26 Dec 2023 21:14:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quip.cz; s=private; t=1703621644; bh=cmUGiX8vPze5mfFdh/ULA4+R1FOErBIB6JRo2ejIwCM=; h=Date:Subject:To:Cc:References:From:In-Reply-To; b=MwjE0MW1IqHmpl4LVg4m2cJz6t2ZfldhEwTsp13Wz9q4vCig9PwMY4JP3flbEE5x0 vLLK4/VO+gZSgrNTww75iMgZ9Uv1rlFmDG89EPZpgPiNOtLXymUxysrlnjbfZVQvpe USv2aUordtujRDlnMzjiE9w/WvRxaG3eI+lzq1pw= Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id B46A9D7884; Tue, 26 Dec 2023 21:14:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quip.cz; s=private; t=1703621642; bh=cmUGiX8vPze5mfFdh/ULA4+R1FOErBIB6JRo2ejIwCM=; h=Date:Subject:To:Cc:References:From:In-Reply-To; b=feN5ujPOTm2BDX1NdLXNjp2eIKfapv+T1dV6DA+f9hrrn8aHde9MObxWzHVGrNmET aWc5bwmW1IK+fTupVE7xH2I2zF3nTzdnggCH88JOHzdaF6vYZ6TxL1yd3m8INX4HLU u1lr56zREJlH7MkzjpEwnAhqE1G24aNoXcfRAHQo= Message-ID: Date: Tue, 26 Dec 2023 21:14:01 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Howto migrate a FreeBSD cloud instance to another one without native FreeBSD support? Content-Language: en-US To: Michael Grimm Cc: freebsd-fs@freebsd.org References: <4A6398FE-5DC1-45A1-85A9-7C5054CF5BF7@ellael.org> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4T05Zk4CtVz3RC4 On 26/12/2023 19:41, Michael Grimm wrote: > Miroslav Lachman <000.fbsd@quip.cz> wrote; [..] >> I know nothing about OVH and almost nothing about what kind of FreeBSD setup you want to migrate (partitions, UFS vs ZFS etc.) but one thing I can imagine in you situation is creating everything from Linux rescue (partitions, formatting to UFS or ZFS, boot sector, rsync old FreeBSD instance to newly created...) > > Sorry, if I haven't been clear enough: > > #) I do want to migrate an existing ZFS pool (from one instance to another (zfs send | zfs receive). > This I have done numerous times before. > > #) New to me is the preparation of the target disk out of an Ubuntu/Linux rescue system. > Namely, how to partition with Linux functionality (gdisk, cfdisk, …) lacking gpart? I used Linux fdisk about a 15 years ago so I am not the right person to help in this. > Namely, which kind of booting to use (legacy versus EFI)? Legacy (BIOS) or EFI boot is matter of what OVH supports. FreeBSD and Linux supports both. Both will work for a years. >> And the second is: you can prepare small minimalistic FreeBSD image in Bhyve / VirtualBox etc. make a copy of it with dd and then write it by dd to your new instance's disk with the help of Linux rescue system. Then you can reboot to a minimalistic FreeBSD system, enlarge partition(s), use growfs or zpool / zfs enlargement and then finally move data from the old instance to this new instance by rsync, or zfs send, or whatever tool you prefer. > > Thanks for this info, very much appreciated. Do you use swap partition on your old instances? If yes, I would go with the path of creating minimalistic FreeBSD UFS image which you can dd in to target disk with the help of Linux rescue system. You can prepare the image of the size of your swap, write it by dd on to target disk (with the boot sector / boot partition), then you reboot to this FreeBSD instance where you can work with all the FreeBSD tools you like, edit or add partitions, create ZFS pool, use zfs recv etc.. And after all work is done, you can wipe content of this partition and use it as you swap on newly migrated FreeBSD instance. This partition can be few hundreds to 1 GB of size, just to fit the base installation of FreeBSD and tools you need to have for migration. Kind regards Miroslav Lachman From nobody Tue Dec 26 21:15:02 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T06x257Qzz55XnM for ; Tue, 26 Dec 2023 21:15:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T06x244tlz3WYK for ; Tue, 26 Dec 2023 21:15:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703625302; a=rsa-sha256; cv=none; b=eCkGx1Zt26Lul8IxtZNZGcpTgzcH9z9v2xTxUuMDYeF4S6l4IF6cb4pzuBZSEFqYD9fcSK u8K7kQ2Cn56HIiD9/4xyyx/Fw258l5ARIRlJ7B3EqNcQptL/uMjmbh4cTzc+nY+YjRjk0X Fs1yjGmjF3aa5JNebIIRSLW0msLm/dejZerURoNObrpi2+AF2u8navVKlsrfng5aL/7ctw q++jt/AVwJVMU1PVoZpjrI+8IVDOyx8pJ3MBTYAZkQ7TUSvppIDq0sjhTIHYxFPAVihYZl QtPvnoQ91GkxB4OpycKZ5eXjNi4xO40eSdpKnvXvVrvmXaqyAtXK3XwCoKQ1Pg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703625302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=90WcxG5XBTa7h5oIyM7Wku31EyMTgmw44nPG0yLC9w0=; b=nK7gSfq3KVXHl3guc6wlJSzcIB/2xRvyaElosWvAtf8T0x5KL6XhPXkaOFyU/vWLlmmob9 mo63t+bR5Fv4MqAfFCCk+pMVMKflhzu2cB+M0RhpDm3APd48Qq2a0oN1/HL775TiOioSk5 kS+MeBjRpgsTY+Py2x0zX+8v6kRPKAyAFZQR5DDK7zbi40eEs/Xdg2E2tiWkmnBCK2/tCX bfrxbZwZUXYIa1G85Ykl2RZfWihMyxtvLsuO3/fbRUfopVyZDi5iL5KFQKYmAccmI0eV7S xzBOahNsCZkOmyqsTvHQ3tzdm+oXrasKhhC8dcRj/Ai1FjKvXto7P3E41q7Xsg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T06x238FCz19cT for ; Tue, 26 Dec 2023 21:15:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BQLF2Hq076533 for ; Tue, 26 Dec 2023 21:15:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BQLF2Oc076532 for fs@FreeBSD.org; Tue, 26 Dec 2023 21:15:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Tue, 26 Dec 2023 21:15:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 Mark Millard changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marklmi26-fbsd@yahoo.com --- Comment #7 from Mark Millard --- (In reply to Robert Clausecker from comment #3) (Kurt too?) FreeBSD version? Prior version? There has recently been (this month): Fri, 08 Dec 2023 . . . git: 3494f7c019fc - main - Notable upstream pull request merges: . . . Mart= in Matuska (It had a very long summary line.) Fri, 15 Dec 2023 git: 5fb307d29b36 - main - zfs: merge openzfs/zfs@86e115e21 Martin Matuska Tue, 19 Dec 2023 . . . git: 188408da9f7c - main - zfs: merge openzfs/zfs@dbda45160 Martin Matuska With no other added reports other until yours and Kurt's, I wonder if more recent changes are involved than John ran into. In part this is via your report of likely lack of dedup as context for your failure. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Dec 26 21:18:42 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T071H2Kc7z55Y9c for ; Tue, 26 Dec 2023 21:18:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T071H17fsz3Wtv for ; Tue, 26 Dec 2023 21:18:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703625523; a=rsa-sha256; cv=none; b=KJlAqkksiP53xghBXfM/+GuH+NErfpgFIsv3yJFI0r03HIacdfDZD/V0qV1hXRzhWauJMC 3rfz0w1SemfX8muUR9EG+4j/lE1VRNOhkOfZ9f2RiAvaie/S1/Aw1MKuCQRLmuy4tN8xnl HDaDP6qj8GIGT3TXnPYz+Rg+/GGXuPl9JauW147fHzLCv+Q2rA3LuYcXkilkJwKIAUXJhK FnDu4H8c6DHPqrKYRBCu3gXjFgpSNej6xUESmIaAVT9JLhfZP6FnkohJlQYGoJkWNzSw1C ycisHC5vVZjRrj7rotYkAxYT9P11aBBBzcgiisjQ5rgrZ8Kcm9qxETt+3H+6AQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703625523; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JSvhaPm6mbGnuHs+DATjMUAacVvZFwAC3TYUVPXWzUU=; b=GCvbZo/6HiYir6RmkpaVCgZTinMkBbQ1FMZLYG4Y2aY7qA1MFJ1UNBLQAQCYeQgExaTMRD XTuLheyo6vYylITQNLhmEp/ZgwB/o/HBpnjVWfT4HtyuiLAwDdFP0vj64YnCZ6EoCzZrmG a55RFRUJ+xCHK5sL9nt+M/4h4uT8/jw2JPSahOUli1rocskmHxE2xBDR6AizuvS8j1ok8B taGTNOtwxlVJU+m2ISmZ/kg51Fb37LR2Eozot1BWbLYgoUstnPh0/MugjDihiN70ZLRESx 2HVmGXjE0WsgwLQbk7e0r7uvUM0S1mtNWGi1ce78+HhNkobr3Hgky4p3KjIuNw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T071H07gYz19jP for ; Tue, 26 Dec 2023 21:18:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BQLIg1G077646 for ; Tue, 26 Dec 2023 21:18:42 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BQLIgAD077645 for fs@FreeBSD.org; Tue, 26 Dec 2023 21:18:42 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Tue, 26 Dec 2023 21:18:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 --- Comment #8 from John F. Carr --- Block cloning acts a lot like deduplication and may trigger the assertion failure without deduplication being explicitly enabled. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Dec 26 22:24:05 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T08V26SJTz54QdN for ; Tue, 26 Dec 2023 22:25:14 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T08V22QsWz3bqL for ; Tue, 26 Dec 2023 22:25:14 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1703629499; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=GBcWlFFUMTgsIWdJb2V9dERZ0x/ZR1kZAb0/+D2zdGY=; b=rM716WQXluljnXTW8IrjoZlrsQ1Snj01qjbKorIJJ36X6iHS7egBfG+cEXomYgQDLemPbR AYKQJdX5voMgYGwaV3VCLJSMIU/Q7S3kOGs7xCm1YC2SrMGOjvdjNM0MzWQXApDYBe5lls b3EZnYfQ9Nsjk5MT2eqLV/cvQFRYEiCAbAvcTai4/fhi8C3lLHFp5k3x5LeLv6vlo5LSxv ZnoDDnvsFpmwW9B0JgHrPDUrSBaYVsnRcYE1tq2CglDow+3p3zPaL3i7Vm+V5QjGFPU4EM 5HounYE7j0uJjUXL2uF5VXUV80mTjBuwU7kGljqnxf9zrM9hEKBZhEiibIZJbQ== Date: Tue, 26 Dec 2023 23:24:05 +0100 From: Alexander Leidinger To: Michael Grimm Cc: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs@freebsd.org Subject: Re: Howto migrate a FreeBSD cloud instance to another one without native FreeBSD support? In-Reply-To: References: <4A6398FE-5DC1-45A1-85A9-7C5054CF5BF7@ellael.org> Message-ID: Organization: No organization, this is a private message. Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=_1ebca81641bc5a170a85d90bc4b02ee6"; micalg=pgp-sha256 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4T08V22QsWz3bqL This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_1ebca81641bc5a170a85d90bc4b02ee6 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8; format=flowed Am 2023-12-26 19:41, schrieb Michael Grimm: > Miroslav Lachman <000.fbsd@quip.cz> wrote; >> On 26/12/2023 15:22, Michael Grimm wrote: > Sorry, if I haven't been clear enough: > > #) I do want to migrate an existing ZFS pool (from one instance to > another (zfs send | zfs receive). > This I have done numerous times before. > > #) New to me is the preparation of the target disk out of an > Ubuntu/Linux rescue system. > Namely, how to partition with Linux functionality (gdisk, cfdisk, …) > lacking gpart? > Namely, which kind of booting to use (legacy versus EFI)? > >> And the second is: you can prepare small minimalistic FreeBSD image in >> Bhyve / VirtualBox etc. make a copy of it with dd and then write it by >> dd to your new instance's disk with the help of Linux rescue system. >> Then you can reboot to a minimalistic FreeBSD system, enlarge >> partition(s), use growfs or zpool / zfs enlargement and then finally >> move data from the old instance to this new instance by rsync, or zfs >> send, or whatever tool you prefer. > > Thanks for this info, very much appreciated. This is more or less what I did for such a case (local provider, not OVH): I used an existing FreeBSD system, created a file of a few GB, dd-ed /dev/zero to it completeley (for a compressed transfer), created a md(4) out of the file, partitioned it, created a root pool, installed FreeBSD into it, created a suitable rc.conf and such to boot from it, and wrote it to the linux rescue system on the target host "ssh -C me@target dd of=/dev/sda ; Tue, 26 Dec 2023 22:35:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T08k20L9Mz3csW for ; Tue, 26 Dec 2023 22:35:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703630138; a=rsa-sha256; cv=none; b=d1OqoIL+jK51bd4f4q5SaZvkx6KZ2s2aN7/7KiQcDVdibGvjO2ZNpwx0FRQnxd7BzB7xwz 0GkPJb2ZjxwcsEZsosiLozpf5m+NwrPho9DcYXfCJSutyWmduzpkj1EXjEd7kxbjYmFwSC /Gz9pyrLMl8pcdHIkVhb2qBktYL4e63GzJNkFMuS6BdCbuEoD7Bi/Hx6ImbAJ1Dk1Z8jRr yMfavS6GW31yGiBxlbRiPai0BZhgQHISRVRZhK8j1QwvwZx+sYUEzv4jCaaltyZ81eup0b zFtG5SLB+Z1Ovc0EdlYGXTPF2+AWGtAfwe9LBmzLT7cfOVPcVX50S+auIFoQ3A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703630138; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bJl7FH/mqEz/vkSOCOV2exvmpnhAXzU+LLtB0NVfO6w=; b=RWnwjL+HlXy16622XdU+OT6G9szoMB4mjQxiLa8fkNOYvYoARUFPkK/Tv/plKXRFqXf09Q TbLV4fC3pIBlC8IBGR9mMgwFMqH0qklHeFRkx3dPfSKmXXWR6maPX0cUzW3Mmg1I1WLfm+ eTa16NAuHqFBTFHvRk8up32uGmE7OoWMLeS/z1neip4f3g5gokjGFwirRNQV9fA5lUXd6o gC3guHJanU4CBvlyCmTzcRYqlLROvJJEo7ab+wC7z2O1rbMpU/mPVr6P/NeOrUEjluXHht JKNPM684X37NUCkN45NIT0d47WSoNAdzX0MjjAwIKnOsq58lt5TcUSN7QIOrww== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T08k16RJSz1Cdd for ; Tue, 26 Dec 2023 22:35:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BQMZbOK099795 for ; Tue, 26 Dec 2023 22:35:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BQMZbD9099793 for fs@FreeBSD.org; Tue, 26 Dec 2023 22:35:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Tue, 26 Dec 2023 22:35:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 --- Comment #9 from Mark Millard --- (In reply to John F. Carr from comment #8) > Block cloning acts a lot like deduplication and may > trigger the assertion failure without deduplication > being explicitly enabled. Okay. It still leaves the early 2022 to late 2023 time frame difference and 2 reports in short order in late 2023 as suggestive of recent changes being involved in the recent reports. Robert and Kurt giving more details about the versions used for similar-context-to-recent failures over that 2022 to late 2023 time frame could be somewhat useful if they have been using various versions over that time frame, used for the same kind of activity fairly frequently but without problems until recently. No claim such information would be definitive. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Dec 27 08:05:45 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T0PMs2gzzz557SN for ; Wed, 27 Dec 2023 08:05:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T0PMs1dBlz3Sh2 for ; Wed, 27 Dec 2023 08:05:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703664345; a=rsa-sha256; cv=none; b=ReR58h+53it5mFxv6Z6JdSLdHscnYP4mgC4WqeUEoP4ZlhQvpkTbGYaKDXXGI5Cjf6wdzm /16wZncetLx+CybETx5E8kfYhNTfb7A8hFd3WIltTN+3HRIY7h0mlJoFsO1FcUG2NKbg3y t0oydSGKQKNCjBlzNRkHKPeeBZ5NIFMv/cYqWGkBFJx2+yWu8X1CZlcc37XjoPjIFBwjmU 7Cq3vHIHya53Tr3y77wUKNWN3+3YHA5dyds5fecSXhNLM1foxi3j3RP+S5onIGpjYpO/YR uKVU5Wc0ABMhVUO+ChcdksJNHJRpzcXyT2CR54S44uZjggd9yib9gyWaQ2iWrw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703664345; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3YKANO1B18i7X6nPURrtw0VY8ltsBSWkerZLTC7tZ0s=; b=NBYP5tFx1jNYRWTpxze3CTetk+2uyrSnA0H1yOoadEZUPUaAtq4C1WfM2Gyybz8QWqizja WsL3+3UhKboiogOSepFL4sIzr5QMLz0HBMU4/t+Hiq8B2SySIJyMKTYtcbrVeLHX/EJ/Xd GxsNGzp/m3gb1+sXf4Qg7wRammJizo7sGpkUqwUCMeddwPBsCoTAcD1+0fAbM53QMMwUJA 97+eVjxTqCpFFS3eefFeDv9rC29qGSNPEDus8xDBcJ9rrMLN4O9HQuDmdZTdoKQx/JlRMx WdFf9n4N8o1m7YDCSZvEPzsnEgOK/bZ7CXREwEdLFXrqiFr75cWcw+Dav8srZQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T0PMs0cbqzWMb for ; Wed, 27 Dec 2023 08:05:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BR85jEN059309 for ; Wed, 27 Dec 2023 08:05:45 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BR85jv5059308 for fs@FreeBSD.org; Wed, 27 Dec 2023 08:05:45 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Wed, 27 Dec 2023 08:05:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pi@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 --- Comment #10 from Kurt Jaeger --- (In reply to Mark Millard from comment #7) I upgraded to 15 around 8th of September 2023, had some crashes due to out-of-memory during poudriere runs. I upgraded the system to the latestet 15 on the 23th of December, and after that it had series of crashes with the panic given in the PR (and some others), some during poudriere, but tonight, probably during the daily jobs= , it crashed with a different panic.See https://people.freebsd.org/~pi/crash/ for all those where I still have the textdumps. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Dec 27 08:53:17 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T0QQj5sVXz55Crj for ; Wed, 27 Dec 2023 08:53:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T0QQj4T53z3bBh for ; Wed, 27 Dec 2023 08:53:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703667197; a=rsa-sha256; cv=none; b=c2AS3JxKa+AAErQE9ogrdUN8Zq4qen2fybmgIEC6jcIk2YTcioHLJGLpUZKO2c98KbHUNQ VPbe2ToC/Vr3dBUUJECzTkhdVTCikL5+R/FksJmIkgm1D8LdMbtNaCLw3q+zktEzH5Mab2 d7SrrSD6KbSr5kbcsc3uoxf05pkf9G2ApslHlx59ia+rSZ0gTVnWz53KvKtlG4RBajIX1v YVhV2wKWekqwBWYR98pg5hPOywAt4Xi/yyPridJSA9f8QYGSvUt1RfuXQAW2fl7XSVyzHH Px3aUbdc7tiaAp1k2gO9UJC29VF98pjdqgoT9BPX8luXNHS9JH0TUqFV6vSIOw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703667197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JnorxTdE6aXYl68WbI95RtsC2jZDRrwQeFheMMAhAfY=; b=IcFtS1V+QqKAPEdHKjq0S6Jg/nMEtE5mnM2esKDyP70e9qhfAy+YLTdZGM8lVNIpWBukg+ lEb0rgw5Fc/ngiOOvCpyRCU2UoTC5k/ltMxvLqk2cBbnBZGPSEd7GdEVpau3ezlwXIETJm h/6Y+wqCGbvDEv0OS3ZkqHB4wGOl1a5wY2ahejIFtdKOdsuoHmLqDJLn5F7EgdC1pdAANc IrYIDKlXiorFcxqq2TPcr7rnPh6xofWsd4IWguyQlzLPDTo5uVboxpRahFMzNIocPfst26 bQHCM6m7vkSCf0r3njDpy0T8Ac64YTLSC+nNqzAnQdftXHEbPABVYy5nuE3tuQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T0QQj3XnFzXsq for ; Wed, 27 Dec 2023 08:53:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BR8rHTw035906 for ; Wed, 27 Dec 2023 08:53:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BR8rHXK035905 for fs@FreeBSD.org; Wed, 27 Dec 2023 08:53:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Wed, 27 Dec 2023 08:53:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pi@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 --- Comment #11 from Kurt Jaeger --- (In reply to Kurt Jaeger from comment #10) One more thing: When I updated in September, I did not upgrade my zpool, so it's still at the state of 14 or so: https://people.freebsd.org/~pi/crash/features.txt --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Dec 27 09:01:54 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T0Qcf0mbpz55D6G for ; Wed, 27 Dec 2023 09:01:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T0Qcd6nsXz3bZ2 for ; Wed, 27 Dec 2023 09:01:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703667714; a=rsa-sha256; cv=none; b=JRm6c/l4pGG2cVwgThfOGL5hi7fiOLIbDABcNf4VWdoGEARVmSEAlnqWS9mziFUKUsZoEZ rvEZ/eN+eTQwMfDjXqvlSo/y4wfMKdxnlA5/VnjfkTdrmLt2mchin5YbHt35SdtgZO3Pri 3/EZXIUWZbi9CKYdtFHEm7/RzTjnKtf2bpiwL1DctiyBUQyEtCqMtZOK0rCfZvMUhkONpO mH4S1uZzTjhbMjsVIkhJOpSP56/PjV89NEavy0M+QwLPUct4SyVZBBPlxm002VLpw6inMw sk9z7kWICHjXKxzDnWesoOmM35U+2S0sJ/N1VzwgsOpfPEMr3kZ4zsgAYOhRPQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703667714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ziNIhRNAMp4k/3x1r9DOnrqcNpv1eI3kN54BrA5Ikh8=; b=QzzKr6a1w3V3oj+oyIxegRW3GlfFRv+AEPgVmkEtks0isFGWGZIP5X84zDaI4dutYFvCtE zeoP6JiRI4biyftO0cX1BoqRH6XLNDDz9O/DzakHvIxerTR8ubWZQd9rk25Z/IyRNYooBB 0SmL66vZq5PG1b14Qx1W0ZUcK5Tk4d8wKA88p7+wTU1S/RnGnmZpsQZUCO6ePpNDDeIIq1 QO2U1iPZnM7fOTAQ0/llf4zIYPCeI1pUQVfI+8MZiOV1H7alaVPH85WXH98hKiLiBcuMXd yhvS9h5tngdqYPH0mhFEwlbZ3BkpddB3Au6pVmCvqRZAJJJEa1x8M6iZ2X8EIw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T0Qcd5rzrzXjL for ; Wed, 27 Dec 2023 09:01:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BR91rC0050351 for ; Wed, 27 Dec 2023 09:01:53 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BR91rOI050350 for fs@FreeBSD.org; Wed, 27 Dec 2023 09:01:53 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 261538] zfs panic: VERIFY3(l->blk_birth == r->blk_birth) failed Date: Wed, 27 Dec 2023 09:01:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pi@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 --- Comment #12 from Kurt Jaeger --- (In reply to Kurt Jaeger from comment #11) Ups, and it had block_cloning active/enabled 8-( --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Dec 27 10:28:59 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T0SY91gtqz55Mm2 for ; Wed, 27 Dec 2023 10:29:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T0SY90gFHz4GlS for ; Wed, 27 Dec 2023 10:29:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703672941; a=rsa-sha256; cv=none; b=Gd+Qzsg3owsrshRV/Sr7oZ3LHhh47iJFITRHkp6KiZA83HMJypy7Jubmk5BdxccbsfFXiV cgQellVv33eC+a3cprrmTQqSuXzwos5IZnM2041XgPjXLvehgjdkKdCm941nkhbHKZC/Qm mBrW4jzy6stTi3R318yZYjrM0H85103X72MwOuK8wjfls4mhd4g5XESK2AEYISwse/RC9J CsquYqnUGJw0T+R09fs4XuI1i6O5u48Y9dvo9+RGTvx0VQg5bbyw2iMymi1P8nSHKISgdm tVNM4VuZolHt67H43vN9J/QmdbidBoAeqLBSYVAsKlgTYkGrDaIMWSmvmb6B+A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703672941; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FOJwInDNjgU1K4LuaEt7zyj2QW1Ej+NXN0W9pB2yVfE=; b=DakvePvZJua4IGdQHkkkcahzazYe6nO+fVBcipQKBKPEAWYS9J0yAB1jCvGxbh4ZBsRsQt 9/+f5MwW8iMYc6RaW6t90L5pztiTLe1NzaYZbMrwhzJElTnD4U1DU68R7q9Q9cLJMMLYMD vUwexkf2qy84IPFwVHIiU5ejZR6dqfBPrfBipAt0TxcJIN/w841gZtpKsuU7HgrbB2cbzU z74BKddguRQYUInxIgOzJ3p6unlSoc5ENFsNvwpCR1FNjWa1feqOLnKgbBQwUNsSOzLtjf /flD3E9hQzjoUAQD3L7kkoMgtMVwJqVBpAsaX92o4yYzxlRHUrB6h826HutpMw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T0SY86t8WzbLK for ; Wed, 27 Dec 2023 10:29:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BRAT0at075698 for ; Wed, 27 Dec 2023 10:29:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BRAT0hY075697 for fs@FreeBSD.org; Wed, 27 Dec 2023 10:29:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Wed, 27 Dec 2023 10:28:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #17 from Seigo Tanimura --- Created attachment 247285 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D247285&action= =3Dedit The test result charts for Comment 18. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Dec 27 10:31:56 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T0ScX4jmmz55Mdy for ; Wed, 27 Dec 2023 10:31:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T0ScX24PWz4HGd for ; Wed, 27 Dec 2023 10:31:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703673116; a=rsa-sha256; cv=none; b=l0lM1wvuh6P2u7IUk6HfWk3MPTuqH3XKVBdJfNsShTRmJB6S05dGQN0qaToHfJv76rJdMC ZoX0opHhDlen3LWds6Fm+9orBO97r4eVNPAxFdCPWSFRshec9EGkQSGCA4IfoZE9zwU+HI kFyACsxdplfSZYuMCfo7ZB0kP9jg94+V/f3Qrd8P3QimYvG9Bf+xr4W0OkespxcNUaOD0N rn6rp13Xp/zVE0269m4k30Ukobd3f3siBj5PXT2PT1STvmdccdETPVpjOkFcKTeiGiC3eU N50zoEWOs2EBWSt7pNUwsfZnEj9A6TEjfG7bEjQCunUEe6e42LqwTiGQeyVlfA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703673116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=egIW8tMkhu6qzAZr+Qa1NKObEYk+uqeYN8qhgqP94oc=; b=A7jTtfL89WHAUefxGH7WwRP3niaGomaXDEyneLsB0rR0wjJBjTt6qiJSyPMU1sr40R99en CBdcUVlViEXLEBPzxIaMR6NcQmXZdYJZBKLIv4nmClH15gq7getWahO99y6PKswLG4wiGu WRpJbZU769q8m9+Mnuxl1+WN6A6hCO2HLpvtN+DDfJguOvKXqeXv4DTkRQmsXE87hvkqU3 9sYt6FH8TxsCNZ6CbsqXcJYrOx/Q/zkcFcoD4/kc97k/pKegwM8YiH9vOXNJ/v8HevLnBu Sl4HNjcR9AbFHopIoH1anTcc6jy5pIsF0OQSA0DfhUizgUdyHWxc5f65B21jcw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T0ScX17nJzbJ0 for ; Wed, 27 Dec 2023 10:31:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BRAVuRT085422 for ; Wed, 27 Dec 2023 10:31:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BRAVuDV085421 for fs@FreeBSD.org; Wed, 27 Dec 2023 10:31:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Wed, 27 Dec 2023 10:31:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #18 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #16) * The results of the comparision between the estimated ZFS open files and kern.openfiles Test Summary: - Date: 26 Dec 2023 00:50Z - 26 Dec 2023 06:42Z - Build time: 06:41:18 (319 pkgs / hr) - Failed ports: 4 - Setup - vfs.vnode.vnlru.max_free_per_call: 4000000 (=3D=3D vfs.vnode.vnlru.max_free_per_call) - vfs.zfs.arc.prune_interval: 1000 (my fix for arc_prune interval enabled) - vfs.vnode.vnlru.extra_recycle: 1 (extra vnode recycle fix enabled) - vfs.zfs.arc.dnode_limit=3D2684354560 (2.5G, larger than the max actual = value observed so far) Results: * Estimated ZFS open files | (A) | (B) | (C) | | Phase 2 regular file retries | | | (Estimated ZFS open files | ZFS = open files UTC Time | Vnode free call period [s] | seen by vnlru_free_impl()) | estimated by kern.openfiles =3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D 02:00Z | 1.27 | 354 |=20= =20=20=20=20=20=20=20=20 491 ---------+----------------------------+------------------------------+-----= ---------------------------- 03:00Z | 1.32 | 411 |=20= =20=20=20=20=20=20=20=20 439 ---------+----------------------------+------------------------------+-----= ---------------------------- 04:00Z | 1.35 | 477 |=20= =20=20=20=20=20=20=20=20 425 ---------+----------------------------+------------------------------+-----= ---------------------------- 05:00Z | 1.69 | 193 |=20= =20=20=20=20=20=20=20=20 242 ---------+----------------------------+------------------------------+-----= ---------------------------- 06:00Z | 1.88 | 702 |=20= =20=20=20=20=20=20=20=20 232 ---------+----------------------------+------------------------------+-----= ---------------------------- 07:00Z | 1.54 | 299 |=20= =20=20=20=20=20=20=20=20 237 where (A): 1 / ((vnode free calls) / (5 * 60)) (5 * 60) is the time granularity on the chart in seconds. This applies to = (B) as well. (B): (number of retries) / (5 * 60) * (A) (C): 0.7 * (kern.openfiles value) 0.7 is the observer general ratio of the ZFS vnodes in the kernel. (bug #27= 5594 comment #16) * Chart archive: poudriere-bulk-2023-12-26_09h50m17s.7z * Charts: zfs-vnode-free-calls.png, zfs-vnode-recycle-phase2-reg-retries.pn= g, kernel-open-files.png. (B) and (C) sometimes match on the most significant figure, and do not in t= he other times. Out of these results, I understand that the unrecyclable ZFS vnodes are caused by opening them in an indirect way. The detail of the "indirect" way is discussed next. ----- * The ZFS vnodes in use by nullfs(5) Nullfs(5) involved in my poudriere jail setup is now suspected for the unrecyclable ZFS vnodes. My poudriere setup uses "-m null" on the poudriere jail. > root@pkgfactory2:/home/pkgfactory2/tanimura/work/freebsd-git/ports-head #= poudriere jail -l > release-13_2_0 13.2-RELEASE amd64 null 2023-04-13 03:14:26 /home/poudri= ere.jailroot/release-13.2.0 > release-14_0_0 14.0-RELEASE amd64 null 2023-11-23 15:14:17 /home/poudri= ere.jailroot/release-14.0.0 > root@pkgfactory2:/home/pkgfactory2/tanimura/work/freebsd-git/ports-head #= =20 Under this setup, poudriere-bulk(8) mounts the jail filesystems onto each builder by nullfs(5). A nullfs(5) vnode adds one v_usecount to the lower v= node (asserted in null_nodeget()) so that the pointer to the lower vnode does not dangle. This lasts even after the nullfs(5) vnode is inactivated and put o= nto the free list, until the nullfs(5) vnode gets reclaimed. The nullfs(5) design above explains the results of the estimation upon the unrecyclable ZFS vnodes. As the builders open the more files in ZFS via nullfs(5), the more unrecyclable ZFS vnodes are made. In the detail, howev= er, the estimation makes the errors because the multiple builders can open the = same ZFS file. The massive free of the vnodes after the build is also explained by the nullfs(5) design. The cleanup of the builder filesystems dismisses a lot of nullfs(5) vnodes, which, in turn, drops v_usecount of the lower ZFS vnodes = so that they can be evicted. ----- The finding above introduces a new question: should the ZFS vnodes used by nullfs(5) be recycled? My answer is no. The major hurdle is the search of the vnode stacking link= .=20 It is essentially a tree with the ZFS (or any non-nullfs(5)) vnode as the r= oot, spanning to multiple nullfs(5) vnode leaves and depth levels. The search is likely to be even more complex than the linear scanning of the vnode list. In addition, all vnodes in the tree must be recyclable for the ZFS vnode at= the tree root to be recyclable as well. This is likely to put a complex depend= ency for the ZFS vnode recycling. ----- My investigation so far, including this one, has proven that it costs too m= uch to scan over all vnodes without any positive estimation in advance. We nee= d a way to check if the ARC pruning will yield the fruitful result in the way m= uch cheaper than the vnode scan. It may be good to account the number of the ZFS vnodes not in use. Before starting an ARC pruning, we can check that count and defer pruning if that = is too low. This has already been implemented in arc_evict_impl() for the eviction of the ARC data and metadata by checking the evictable size. The = ARC data and metadata eviction is skipped if there are zero evictable bytes. * My next work Figure out the requirement and design of the accounting above. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Dec 27 10:35:24 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T0ShX7135z55NNj for ; Wed, 27 Dec 2023 10:35:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T0ShX5lDTz4Hrf for ; Wed, 27 Dec 2023 10:35:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703673324; a=rsa-sha256; cv=none; b=kntnM0wNFO2H7YI3mre2qupVIxLedPHeNURJ+QeSTz5E05Cd5o/cMaP3WYnUbEJ7i9uNJy BqL5unofb06S422t6lr5EY/Xs2fcN1socCBYBQdq6IHSu3sHgxCcil2b88YUvb/p2cyb4C Je7FMQl6P5m7EPm7qPPI5lurX/p3aDU1geF/nnQacwTinIGdr4VjAK1mCzjMCC4n2XvIBa oD7CikhuSoWnbbQizAy4gZ0pT6V3sTbTpqe0d8tEiTw7pXogtsUEeDuurHgkvEBYv4+fi6 5MulG88THs4rwpS8lpVwLnirw4u19y1ziNqhot1qrr2StlP3TCqUDC7PUq7Vyw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703673324; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=T0y3zmST5SHgSWkH7UWbP0Pe+P45FjZkILRW+OBWpz0=; b=l2y/Cyf4C/hRTjPYkFTwOz1GSUx+f+CxfMar/fKBLe87wSjVEEZWjlYm4jUEttLtdxBPP5 BxV5csho67vgcJ2HsVyfs+7abiPMWRvaXx4jPZfsQOl6DGfAuJNwQijw6wuC3XLvpSVgry Pj0bRAvS/YYKZPatFbO+b7t98BgZisONB8avh+kdobHaFtjrgDTzWXB7wgs2e4da9FokJr YyRSpKz2Tvlon2pHJ3f4YLsm+b1E7dheZQzaIRMKZwZoRKaIeHt2wwrB7Eh4gZ5boAITEd L9gsk5/MscUCQeeqOOMD+z8gpomLo4qv7httG1eimADzZ1qsFueReMj2MnxhFg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T0ShX4qJFzbbr for ; Wed, 27 Dec 2023 10:35:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BRAZO5u089550 for ; Wed, 27 Dec 2023 10:35:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BRAZOjI089549 for fs@FreeBSD.org; Wed, 27 Dec 2023 10:35:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Wed, 27 Dec 2023 10:35:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #19 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #18) Correction: * The results of the comparision between the estimated ZFS open files and kern.openfiles Test Summary: - Failed ports: 2 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Dec 28 09:28:22 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T139719kTz55SN6 for ; Thu, 28 Dec 2023 09:28:43 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx2.enfer-du-nord.net (mx2.enfer-du-nord.net [135.125.211.209]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T139572Nfz3QWm for ; Thu, 28 Dec 2023 09:28:41 +0000 (UTC) (envelope-from trashcan@ellael.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ellael.org header.s=dkim header.b=o15lrpFP; spf=pass (mx1.freebsd.org: domain of trashcan@ellael.org designates 135.125.211.209 as permitted sender) smtp.mailfrom=trashcan@ellael.org; dmarc=pass (policy=quarantine) header.from=ellael.org Received: from smtpclient.apple (p5b2e5747.dip0.t-ipconnect.de [91.46.87.71]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx2.enfer-du-nord.net (Postfix) with ESMTPSA id 4T138x2pBLz1JSk; Thu, 28 Dec 2023 10:28:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ellael.org; s=dkim; t=1703755713; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dnHPPFSaODMwe+sBLIeQDCUKGF0vfM2BYnfHLe3fQh8=; b=o15lrpFPrx4SwQNFYbWWTCsV3aI3fMGlCBPvpAj/2+3QB4m9oI5MztgPLfbYSibCH4p8os 3TF9G0zzfDWHh1Rb+BNZ7MF/+RhNIBhQmnYziwS0YBP8GWHdg++o87Zz0BsrumNhzZ3NY8 8t/X6m67bfOkmlEgVsNUoRENc7XrkV1TFrEfl3b3P7JwkhNO4k8iIo7UXDe01TrN01VBU5 ICybxxXbK2gPovo8bJDr3A9WPVfU2spj2944IIdWJREuUStzw3DPRDwzAuchcokSoPnIIa GfDiAp8cZpEbrq9gcYpZvYrNO5rS9WDN38ld9FDhcsjzQ43pFzKYbgoHgy/+Hw== Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: Howto migrate a FreeBSD cloud instance to another one without native FreeBSD support? From: Michael Grimm In-Reply-To: Date: Thu, 28 Dec 2023 10:28:22 +0100 Cc: Miroslav Lachman <000.fbsd@quip.cz>, Alexander@leidinger.net Content-Transfer-Encoding: quoted-printable Message-Id: <40702ABB-1DD2-44D3-B33E-0EAB60D19B5C@ellael.org> References: <4A6398FE-5DC1-45A1-85A9-7C5054CF5BF7@ellael.org> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Spamd-Result: default: False [-2.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[ellael.org,quarantine]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:135.125.211.209]; R_DKIM_ALLOW(-0.20)[ellael.org:s=dkim]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; ASN(0.00)[asn:16276, ipnet:135.125.128.0/17, country:FR]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[ellael.org:+]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4T139572Nfz3QWm X-Spamd-Bar: -- Alexander Leidinger wrote > Am 2023-12-26 19:41, schrieb Michael Grimm: >> Miroslav Lachman <000.fbsd@quip.cz> wrote; >>> On 26/12/2023 15:22, Michael Grimm wrote: >> Sorry, if I haven't been clear enough: >> #) I do want to migrate an existing ZFS pool (from one instance to = another (zfs send | zfs receive). >> This I have done numerous times before. >> #) New to me is the preparation of the target disk out of an = Ubuntu/Linux rescue system. >> Namely, how to partition with Linux functionality (gdisk, cfdisk, = =E2=80=A6) lacking gpart? >> Namely, which kind of booting to use (legacy versus EFI)? >>> And the second is: you can prepare small minimalistic FreeBSD image = in Bhyve / VirtualBox etc. make a copy of it with dd and then write it = by dd to your new instance's disk with the help of Linux rescue system. = Then you can reboot to a minimalistic FreeBSD system, enlarge = partition(s), use growfs or zpool / zfs enlargement and then finally = move data from the old instance to this new instance by rsync, or zfs = send, or whatever tool you prefer. >> Thanks for this info, very much appreciated. >=20 > This is more or less what I did for such a case (local provider, not = OVH): I used an existing FreeBSD system, created a file of a few GB, = dd-ed /dev/zero to it completeley (for a compressed transfer), created a = md(4) out of the file, partitioned it, created a root pool, installed = FreeBSD into it, created a suitable rc.conf and such to boot from it, = and wrote it to the linux rescue system on the target host "ssh -C = me@target dd of=3D/dev/sda ; Thu, 28 Dec 2023 12:07:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T16hF4jZQz4L6w for ; Thu, 28 Dec 2023 12:07:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703765245; a=rsa-sha256; cv=none; b=gXtQ7ayw0dEkUe/4Ccy2rdvRhjtKqhyo29aoUyx5CMsRQPUcCIiPOAJ/tBhBM9imn10fSv iNY5HQ1OIWZ5P50vewmVGqQFzyNepksPzAYeBRG+DVlr+YpfAM3g+7wdxIBIu0e8RFEH/t KrTMK1nmDVGfQLurmPpn03gg4Z78zeOtFOjONeOMM3zxbiHU3qFia78WAtB/sApcycOvTb 0LjJwfZb3Eu/P3xz4SD5plkCWAlToSP6ugnnfsn13CG/AGwobn07t+1u2jesIK99FEDeHR RqlICQnAG0SQ67NB/nTNacRBcqYsZG5Y4/13Zv63orfc4Ett4l8mrtJ6XcysWg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703765245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oGM2cDBnmXOWpL8VEUGh/FGFrg6zgWhYR5Ma77EMpxY=; b=DP6Cm3TisObPqG5o9moF5sYjY+eX2346PZQx0Zo+vWLhdQC+8vwoFQcIm4AFeQaIwLtFAQ gilCLauycWo3Ni30GxljxCuEq/B+WwfvJiiaE/CTmebY6KeVSMInXlNz+A8mZvxVn1gM/H HnrtZhYwfiCgAlwjMDVJNYqFEYIXMQQ5yTqo1yg/bdHoRp2NqLr2FyGmXRTN/TPmB5rhdV QlKt8UejOs/e0DeREJHyoCWV/1xHiEF1T3XKPMWXTI1+uhWKmLkriRr+bs7IuDUAZS6Lz/ EiZv7DgQRH3DnOHR5HioF+1kR6YmFjnFrMbSukWwWnr3X9V4IrdPPTlYfNTflQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T16hF3gd2z6bj for ; Thu, 28 Dec 2023 12:07:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BSC7PJF002182 for ; Thu, 28 Dec 2023 12:07:25 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BSC7PEK002181 for fs@FreeBSD.org; Thu, 28 Dec 2023 12:07:25 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 214326] unionfs+nullfs strange behaviour Date: Thu, 28 Dec 2023 12:07:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.mimetype attachments.ispatch Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D214326 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #176786|application/x-download |text/plain mime type| | Attachment #176786|0 |1 is patch| | --- Comment #1 from Mark Linimon --- Comment on attachment 176786 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D176786 An experimental patch for unionfs internal cache(Nov 8, 2016) ^Triage: convert this to text/plain and set the Patch flag so that the automation can see it. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Dec 28 12:24:14 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T175n2ZPmz55lFj for ; Thu, 28 Dec 2023 12:26:05 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T175m3cnmz4Nhp for ; Thu, 28 Dec 2023 12:26:04 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=azUPjVNC; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=2y+ae8WH; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.28 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 91C4C5C009E for ; Thu, 28 Dec 2023 07:26:03 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Thu, 28 Dec 2023 07:26:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1703766363; x=1703852763; bh=qZP/z5KvIg 2lFYyVgNPIPzfsFejRn8SNcYE6MxazKro=; b=azUPjVNC3RyXIXCHh230QEzojM C3VIkX70F4Fd0JBZUWq8xclXGtGIXE8TbMDWmoyUcQ5KnZrH9AqlDRhDGmoG6Xfq 7BQ2A41TRCt/d3kp++igrC4I/hCZz1YDTcnBcvQOpv0r1WW3yhcFi8/WSB/yPREd XpxWWgG/JMCzo48NgkjzupXdp/3sESyo/HSh9PcrgvE95+iXTCBHlw/HAkfxy1tm 28VVoGl8rGIQ6/PKgpf98TSouVW4gHLOr54TTYRJ+G0lL59l743SoFGjk/nsErn7 4rtBqgJAUc5NSaSkCRNy4q2RghAgmAw+SAGCXscy5ObSQzvIAMWYQhDag9ng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1703766363; x=1703852763; bh=qZP/z5KvIg2lFYyVgNPIPzfsFejR n8SNcYE6MxazKro=; b=2y+ae8WHRnC6R+Jkywr4ubxkVhHxn2o/MGOXJCmK7B0M NKabgOUVKl9RCkIxzQCGnFoWN6G1IBI+g5D2xz3qoe/auGFoZ9CWinv5PvJVpXvD Mi12NODfTEsR+QsI69VJHEb6XkPz1h8/e3MfwEhKlI8edLOXk5xdrYmUtAaRbrE5 GTbfMG3fZxe4zsx3yFBecPrEngnaAiaUI/fjQdhJwQhgrE6TY5VQXopbG8YlxRAE NnTlIsxxhyn5CX3LyWi9ZOYXhH6ew69P3NwnQ4rEa+PYDwm8fHtflYbvyUY/fhr/ b5YbNOk//E1RsjvrhGujB/6GF6g2F9jlU9jIAZr4fw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdefuddggeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeeiieehuedtudeikeduieekvdffgefgfeevvdefheffgedutdehtdetvd fhveekhfenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghrufhi iigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 523722A2008B; Thu, 28 Dec 2023 07:26:03 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-1364-ga51d5fd3b7-fm-20231219.001-ga51d5fd3 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Message-Id: In-Reply-To: References: <0D286183-ACD6-4818-8E27-E03D7F16D288@yahoo.com> Date: Thu, 28 Dec 2023 12:24:14 +0000 From: void To: freebsd-fs Subject: Re: measuring swap partition speed Content-Type: text/plain X-Spamd-Result: default: False [-5.11 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.92)[-0.921]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.28:from]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4T175m3cnmz4Nhp X-Spamd-Bar: ----- Plugged the disk into usb3 interface on amd64 machine and ran the following [a] tests all at roughly the same time. The write test (2x git clone/pull) goes to a ufs partition. Writing 2x swap at the same time never got below 3mb/s tl;dr : 1. in the arm64 context, the extreme decline in swap performance happens if swap gets used, by anything (system) 2. in the [1] context, even when swap later becomes unused by the system, the performance problem persists. 3. the situation is recoverable only via swapoff then on. Then the swap partition can be written to in the tests at full speed. Writing subsequent swap tests also run at full speed. When the system swaps again, the problem returns. 4. the slowdown does not arise in an amd64 context beyond what one would expect 5. ufs or zfs makes no real difference it wouldn't matter to me if the write speed was 3mb, 16mb or 22mb/s. What causes problems is the sub-1mb/s speed (worst I've seen it is 300kb/s) and the non-recovery of that swap space when load eases off. I'm probably doing myself no favours using zfs in the arm64 context though. thank you everyone who helped, I now have an idea of a workaround ########### amd64 tests of the same disk # gpart show da4 => 40 1953525088 da4 GPT (932G) 40 532480 1 efi (260M) 532520 2008 - free - (1.0M) 534528 4194304 2 freebsd-swap (2.0G) 4728832 4194304 4 freebsd-swap (2.0G) 8923136 4194304 5 freebsd-swap (2.0G) 13117440 4194304 6 freebsd-swap (2.0G) 17311744 2048 - free - (1.0M) 17313792 8384512 7 freebsd-ufs (4.0G) 25698304 2048 - free - (1.0M) 25700352 1927823360 3 freebsd-zfs (919G) 1953523712 1416 - free - (708K) # newfs -U /dev/da4p7 /dev/da4p7: 4094.0MB (8384512 sectors) block size 32768, fragment size 4096 using 7 cylinder groups of 625.22MB, 20007 blks, 80128 inodes. with soft updates super-block backups (for fsck_ffs -b #) at: 192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880 # mount /dev/da4p7 /mnt # cd /mnt [a] #### all the following tests were run at roughly the same time, in different terminal windows. 2x git (ufs) & 2x dd. # time -l git clone https://git.freebsd.org/src.git Cloning into 'src'... remote: Enumerating objects: 4437205, done. remote: Counting objects: 100% (383246/383246), done. remote: Compressing objects: 100% (28769/28769), done. remote: Total 4437205 (delta 377514), reused 354477 (delta 354477), pack-reused 4053959 Receiving objects: 100% (4437205/4437205), 1.59 GiB | 6.10 MiB/s, done. Resolving deltas: 100% (3526142/3526142), done. Updating files: 58% (57132/97280) /mnt: write failed, filesystem is full error: unable to write file sys/contrib/dev/acpica/components/utilities/utxface.c Updating files: 58% (57317/97280) /mnt: write failed, filesystem is full Updating files: 100% (97280/97280), done. fatal: unable to checkout working tree warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry with 'git restore --source=HEAD :/' 639.76 real 566.78 user 41.47 sys 1054240 maximum resident set size 2566 average shared memory size 222 average unshared data size 127 average unshared stack size 475518 page reclaims 0 page faults 0 swaps 3 block input operations 15556 block output operations 422759 messages sent 1527197 messages received 606 signals received 3656856 voluntary context switches 851907 involuntary context switches # # time -l git clone https://git.freebsd.org/ports.git Cloning into 'ports'... remote: Enumerating objects: 6046014, done. remote: Counting objects: 100% (936/936), done. remote: Compressing objects: 100% (120/120), done. remote: Total 6046014 (delta 923), reused 816 (delta 816), pack-reused 6045078 Receiving objects: 100% (6046014/6046014), 1.16 GiB | 5.28 MiB/s, done. Resolving deltas: 100% (3650283/3650283), done. /mnt: write failed, filesystem is full fatal: sha1 file '/mnt/ports/.git/objects/pack/tmp_rev_cQdim5' write error. Out of diskspace fatal: fetch-pack: invalid index-pack output 553.87 real 0.41 user 1.47 sys 6092 maximum resident set size 2400 average shared memory size 208 average unshared data size 118 average unshared stack size 451 page reclaims 0 page faults 0 swaps 4 block input operations 2 block output operations 152484 messages sent 414280 messages received 0 signals received 409098 voluntary context switches 571 involuntary context switches # # time -l dd if=/dev/urandom of=/dev/da4p4 bs=8k conv=sync status=progress dd: /dev/da4p4: end of device2040 MiB) transferred 660.001s, 3242 kB/s 262145+0 records in 262144+0 records out 2147483648 bytes transferred in 660.734537 secs (3250146 bytes/sec) 660.87 real 0.44 user 10.61 sys 1992 maximum resident set size 20 average shared memory size 8 average unshared data size 132 average unshared stack size 133 page reclaims 0 page faults 0 swaps 0 block input operations 262145 block output operations 0 messages sent 0 messages received 660 signals received 262424 voluntary context switches 2182 involuntary context switches # # time -l dd if=/dev/urandom of=/dev/da4p5 bs=8k conv=sync status=progress dd: /dev/da4p5: end of device2047 MiB) transferred 666.010s, 3223 kB/s 262145+0 records in 262144+0 records out 2147483648 bytes transferred in 666.162391 secs (3223664 bytes/sec) 666.16 real 0.57 user 10.25 sys 1992 maximum resident set size 21 average shared memory size 8 average unshared data size 138 average unshared stack size 133 page reclaims 0 page faults 0 swaps 0 block input operations 262145 block output operations 0 messages sent 0 messages received 666 signals received 262420 voluntary context switches 2867 involuntary context switches # # gstat -spod dT: 1.007s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w d/s kB kBps ms/d o/s ms/o %busy Name 1 1096 0 0 0 0.0 1096 19 20876 2.2 0 0 0 0.0 0 0.0 107.2| da4 [...] 2 1252 0 0 0 0.0 1252 8 10020 1.5 0 0 0 0.0 0 0.0 100.0| da4 When the simultaneous write test completes, swap speed in an amd64 context reverts to normal, without any user intervention: # time -l dd if=/dev/urandom of=/dev/da4p5 bs=8k conv=sync status=progress dd: /dev/da4p5: end of device2046 MiB) transferred 200.020s, 11 MB/s 262145+0 records in 262144+0 records out 2147483648 bytes transferred in 200.166970 secs (10728462 bytes/sec) 200.16 real 0.45 user 9.69 sys 2016 maximum resident set size 16 average shared memory size 6 average unshared data size 106 average unshared stack size 130 page reclaims 0 page faults 0 swaps 0 block input operations 262145 block output operations 0 messages sent 0 messages received 200 signals received 262147 voluntary context switches 320 involuntary context switches From nobody Thu Dec 28 14:07:55 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T19Md32h5z55twZ for ; Thu, 28 Dec 2023 14:08:13 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-20.consmr.mail.gq1.yahoo.com (sonic309-20.consmr.mail.gq1.yahoo.com [98.137.65.146]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T19Mc0Bvtz4Zy1 for ; Thu, 28 Dec 2023 14:08:11 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=i9vfKeEY; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.146 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703772489; bh=auX9WrgoOnYw6jy393Q2wbV4LgNhMc6CtWYNrIl7J0s=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=i9vfKeEYo5dwc279zpmkGrpIyTt19rtw3OJQ0vDbhkOQLI/jkaSyTNJE4UCva3UIeSVRRNzMJjgFXxfzqEg1fym176l6MupF7QhhN7mKw98xHscFxojWJMQIA9AsXv1BZzNVWFJbir2RpGzauOekgvUY9jdOfPuejXnemU1rbCcWTYvGfcV26uaX/q1HEL+yJcELqN6qfpeV9glS3tmECZYMjK/ByfuNUlurECcWlrF8V3Z6tkrfwTYF4qgt9NOP4ApGtympI0dltTCJ68gRnQpkUvnzL1k9wK+oNgo/mmb12kQUry4SM3X/DvSDZjk7PFzIFKnk9VRO2e1SdjJkBQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703772489; bh=XJmxAlMp8qTs7KZ9h9LM4We/HDGJeQOOyqkZZE4ur02=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=X/MEvAulmPSKfnXcKkZAJGEuLEVmKuLoe6mNUDp4CXJYqSEtCalJ/JEs+jqIlJt3kpDiUGxvvhPxmmhddsdLaZXj4msBkcqWRkJhlDU/eBV5mWjJIkp6UUEWqseklpNWVWf3650FKDM945yRQV/uIDJZ7nelMj5g5Ym0FMG4HhLFFzmtdy+KNMO70Cj7trkEFhsRx9jUwRzqjwZOaU+PnRWb1CgCyYoR6/ck1guW0TQNqAAI2kxhaPj0bv4BbUr353/UN5y10slxpnQFefLYF9/llb1cPgk2rUtLEg0MyjNDI61NWB5u/+ObSQjucUrbeM3ECzhytSEHU/InSRtaRw== X-YMail-OSG: LoiZR8AVM1ni4seqYrlFAn1ZE1g09FgUuv1P6ufQGMFvUU5AqWUOVtjb7MeQBHo vild0L278fyYqzl1gcdmqxQycPlV6Gk6cnI2AI1yihNZK6iuobx23D8gXYaZfBwsCa4OVtvUHyED a9YdIzuIzQB_I6ryoB4AFURkML4vfOIWMTJyjHEx9HGFDmYQEyzvCDi_Z2uiH5QhIs.roQhbzDKy ogRAZsNzaU2ffblbV7V6BUIioUdNSoUiR5dDzTFrFwPHBGhLkqm2VrW7TpoZEERk705GTkGOqzcj KGm5sPCAU0MzISnjhZDgWOwon8dqfcCldUVIKXDEQDX6t2poGb6M3bHmJWLYAEgp49t1nYLL4jJa s3miScKojOUiI5LrmopWNTFC7pmBP5bL06I4g351C_HuqZWLNoludhKhkN5fhXvXCFmAenVJQkIX VyCwZA_gBgRZtf2D8cCC81WHCOx9xmSnqGH6leNUf2P1e_emwi12ibdzgh67Q2oDYpM.36NEsSkD 69CV.ahR3flDa8fiM2_pnjEUF.DhoTV_ok5lQaDyneFonG7FAL4dq4wFdm5RpLihUMzjt9TYMFFd pzWI2FWtM3I61QmaPsAIaAkdybtub18TuTZNE0QxFW6SeFN2974Lh8wECHqGs46x.ZDly4bVf2lN qbPxamA0YEDRo_TVE8j31ZPuyGEsrlHnLTPbm7ni5IFMRJFMgewXwCzwv6KcHaOaLidyuT4Ufl2k CgqpCl2hn3bmgj35P9B5BlxHpftXBwCY1Ns_7Do7gg9HLHQuf.GDowssQIHNSZ9F8S.9Fj6.72ir sbyqBwo2nVcW8Xa5AqXOn25J90Elxb38o7Fp3rPcoIyMvgaQLgCh8pgebU5.EpsLzgngS0FNFfeU AtfZUXvS3QjrAIAUwPuXlxIom.nTlw6OI4c9r3qe47.M.2UDy8ewhCh_vhEkCJW7qkxrCRy_YlXa zsyUWtuXpzGCl_xUVI9T4FcnhA9_1qtHAyI.ZqlNl1ucJOKEftwm15wYlD8JNxjpnamoUKcAN8.8 aP.fC2vRu969tRQyCmIdrRV43n16MJpyCeQ0AubAdasjFa9XcaeGG9v.3noln3XTFnzsJRoBtDU2 C9M9xSGAvcnXYeqzWijktO7t1.05CEEG5awBeX8GmSTtcQ4iwIvBabb76v1UFfTzPpC9AP0OaNYP mcYUT4MVsYJYTROck_vyavVLhy2mX3oTn1LHAbObtDzEJcRmwmHO5N2IcrWON2V2M8JeU8BlNLj0 VmI4qDW95B951bY6PURU4Utw7uqU.YU2kmtuzl9L1ZBHSOjt67VQ3fxAKGVkH9rU74NsIUmqDCN0 .cDuk82FIn1p1Astb8ngGC1LMCxMvLyLKvVxQtOn0viXqMEYu_fMf5qVqigS630Ibhk.ZNmE.AG8 19oDF5iAbUQcUud8uFOPKPqGw56jccLIyj4nehjC9lkMX1q33vxw4YIeEVY4hb6RtOTjz8XDfArT MxLWDGnLhKgpwsMwEKkaOaJTqEvtF13tqGdF2cjE5ol0wK4Z.xYroLGtoKmjTfJ37_Yn7svHCBUp F2r1igjOmjE4CStEddGexf7K6dCr0Y_RVuXv1WZDsnfDR4wdc1T6aqkIlLruKai7UZVTGOzHGCah HF_2U_QTcjX35UJxRdtKVvH_eDjr2pd5YhxEaq1UnyXVp4gDejHY_6s07ygxqWyG3T0NkPvoqf.8 b_N9kEcX6IxmGz966hr3P7C3IIXhTHQkzyUIYlmWsw5ocoIU5._t_2Xofyilt7F8tRsBM.k8zLVb pvNJf.X3arvc4M0.C2fGe4.pWIYnPcRpbsZ4PYNl6LGVFB9up0rqcCyv0p0B6WvJEyMd91kqPdPZ cVmPiWoaaETmuaevRe8rgIBvbxwKdnNlaQiu.YIoaPcH10UYGBY64639yJCQxRDnQYN9BYjZNOcH WepuV62gODDEc2KZiFBVjGDSkiIaUdSXHR9qHVq22_hIgvL7C.eYJu4g2HnAVj25ImzBesglVIHa 3QrgsZ6K5Cgp8oF6DgRDLuiABKLcv3RrEiZEzqv9Bez7Tr1aFWTqoeKjoh5k3ZMgKbiY_lze5wYL em.TdXQeNQcuiOuS_6wwXCuJ5UQ4.BiigkaznsDV9o1v1ty4GTpsg9lytUwsyP8YGP0ubAFmnMLK gxqjq3wLwJNyJvho38eSaqGP5G6tmzMZ.nR8GGhk029eDzaUnmYJ.46ZEKU_OcjdcZ_SxVC4gBAq R0XNQFP3XiNcO7l9.BlxiEe3VKFZjTjkJWcmH3qIJMcnsmu36OrRlB4wECbactSkMTr8rT6t25A- - X-Sonic-MF: X-Sonic-ID: e1458e62-9f37-4adc-a3ac-46d2440c1e54 Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Thu, 28 Dec 2023 14:08:09 +0000 Received: by hermes--production-gq1-6949d6d8f9-qkzts (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 2d91741ff1d27d56ad80dc441e0af043; Thu, 28 Dec 2023 14:08:06 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: measuring swap partition speed Message-Id: <0D486016-8806-414F-B7F9-66519E8B0EBC@yahoo.com> Date: Thu, 28 Dec 2023 06:07:55 -0800 To: void , freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3774.300.61.1.2) References: <0D486016-8806-414F-B7F9-66519E8B0EBC.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.146:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.146:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[f-m.fm,freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4T19Mc0Bvtz4Zy1 X-Spamd-Bar: --- void wrote on Date: Thu, 28 Dec 2023 12:24:14 UTC : > Plugged the disk into usb3 interface on amd64 machine and ran the = following [a] tests all at roughly the same time. The write test (2x git = clone/pull) goes to a ufs partition. Writing 2x swap at the same time = never got below 3mb/s >=20 > tl;dr : >=20 > 1. in the arm64 context, the extreme decline in swap performance = happens if swap=20 > gets used, by anything (system) >=20 > 2. in the [1] context, even when swap later becomes unused by the = system,=20 > the performance problem persists. >=20 > 3. the situation is recoverable only via swapoff then on. > Then the swap partition can be written to in the tests at full speed. > Writing subsequent swap tests also run at full speed. > When the system swaps again, the problem returns. >=20 > 4. the slowdown does not arise in an amd64 context beyond what one = would expect=20 >=20 > 5. ufs or zfs makes no real difference >=20 > it wouldn't matter to me if the write speed was 3mb, 16mb or 22mb/s. > What causes problems is the sub-1mb/s speed (worst I've seen it is = 300kb/s) and the > non-recovery of that swap space when load eases off. I'm probably = doing myself no=20 > favours using zfs in the arm64 context though. >=20 > thank you everyone who helped, I now have an idea of a workaround >=20 > ########### amd64 tests of the same disk Could you produce the same sort of report (gstat -spod and such included) for the arm64 context? The direct (well matched) comparison/contrast to amd64 could prove of interest to folks that work on arm64 support. > # gpart show da4 > =3D> 40 1953525088 da4 GPT (932G) > 40 532480 1 efi (260M) > 532520 2008 - free - (1.0M) > 534528 4194304 2 freebsd-swap (2.0G) > 4728832 4194304 4 freebsd-swap (2.0G) > 8923136 4194304 5 freebsd-swap (2.0G) > 13117440 4194304 6 freebsd-swap (2.0G) > 17311744 2048 - free - (1.0M) > 17313792 8384512 7 freebsd-ufs (4.0G) > 25698304 2048 - free - (1.0M) > 25700352 1927823360 3 freebsd-zfs (919G) > 1953523712 1416 - free - (708K) >=20 > # newfs -U /dev/da4p7 > /dev/da4p7: 4094.0MB (8384512 sectors) block size 32768, fragment size = 4096 > using 7 cylinder groups of 625.22MB, 20007 blks, 80128 inodes. > with soft updates > super-block backups (for fsck_ffs -b #) at: > 192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880 >=20 > # mount /dev/da4p7 /mnt > # cd /mnt >=20 > [a] >=20 > #### all the following tests were run at roughly the same time, in = different=20 > terminal windows. 2x git (ufs) & 2x dd. >=20 > # time -l git clone https://git.freebsd.org/src.git > Cloning into 'src'... > remote: Enumerating objects: 4437205, done. > remote: Counting objects: 100% (383246/383246), done. > remote: Compressing objects: 100% (28769/28769), done. > remote: Total 4437205 (delta 377514), reused 354477 (delta 354477), = pack-reused 4053959 > Receiving objects: 100% (4437205/4437205), 1.59 GiB | 6.10 MiB/s, = done. > Resolving deltas: 100% (3526142/3526142), done. > Updating files: 58% (57132/97280) > /mnt: write failed, filesystem is full > error: unable to write file = sys/contrib/dev/acpica/components/utilities/utxface.c > Updating files: 58% (57317/97280) > /mnt: write failed, filesystem is full > Updating files: 100% (97280/97280), done. > fatal: unable to checkout working tree > warning: Clone succeeded, but checkout failed. > You can inspect what was checked out with 'git status' > and retry with 'git restore --source=3DHEAD :/' >=20 > 639.76 real 566.78 user 41.47 sys > 1054240 maximum resident set size > 2566 average shared memory size > 222 average unshared data size > 127 average unshared stack size > 475518 page reclaims > 0 page faults > 0 swaps > 3 block input operations > 15556 block output operations > 422759 messages sent > 1527197 messages received > 606 signals received > 3656856 voluntary context switches > 851907 involuntary context switches > #=20 >=20 > # time -l git clone https://git.freebsd.org/ports.git > Cloning into 'ports'... > remote: Enumerating objects: 6046014, done. > remote: Counting objects: 100% (936/936), done. > remote: Compressing objects: 100% (120/120), done. > remote: Total 6046014 (delta 923), reused 816 (delta 816), pack-reused = 6045078 > Receiving objects: 100% (6046014/6046014), 1.16 GiB | 5.28 MiB/s, = done. > Resolving deltas: 100% (3650283/3650283), done. >=20 > /mnt: write failed, filesystem is full > fatal: sha1 file '/mnt/ports/.git/objects/pack/tmp_rev_cQdim5' write = error. Out of diskspace > fatal: fetch-pack: invalid index-pack output > 553.87 real 0.41 user 1.47 sys > 6092 maximum resident set size > 2400 average shared memory size > 208 average unshared data size > 118 average unshared stack size > 451 page reclaims > 0 page faults > 0 swaps > 4 block input operations > 2 block output operations > 152484 messages sent > 414280 messages received > 0 signals received > 409098 voluntary context switches > 571 involuntary context switches > #=20 >=20 > # time -l dd if=3D/dev/urandom of=3D/dev/da4p4 bs=3D8k conv=3Dsync = status=3Dprogress > dd: /dev/da4p4: end of device2040 MiB) transferred 660.001s, 3242 kB/s >=20 > 262145+0 records in > 262144+0 records out > 2147483648 bytes transferred in 660.734537 secs (3250146 bytes/sec) > 660.87 real 0.44 user 10.61 sys > 1992 maximum resident set size > 20 average shared memory size > 8 average unshared data size > 132 average unshared stack size > 133 page reclaims > 0 page faults > 0 swaps > 0 block input operations > 262145 block output operations > 0 messages sent > 0 messages received > 660 signals received > 262424 voluntary context switches > 2182 involuntary context switches > #=20 >=20 > # time -l dd if=3D/dev/urandom of=3D/dev/da4p5 bs=3D8k conv=3Dsync = status=3Dprogress > dd: /dev/da4p5: end of device2047 MiB) transferred 666.010s, 3223 kB/s >=20 > 262145+0 records in > 262144+0 records out > 2147483648 bytes transferred in 666.162391 secs (3223664 bytes/sec) > 666.16 real 0.57 user 10.25 sys > 1992 maximum resident set size > 21 average shared memory size > 8 average unshared data size > 138 average unshared stack size > 133 page reclaims > 0 page faults > 0 swaps > 0 block input operations > 262145 block output operations > 0 messages sent > 0 messages received > 666 signals received > 262420 voluntary context switches > 2867 involuntary context switches > #=20 > # gstat -spod >=20 > dT: 1.007s w: 1.000s > L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w d/s kB kBps ms/d o/s ms/o = %busy Name >=20 > 1 1096 0 0 0 0.0 1096 19 20876 2.2 0 0 0 0.0 0 0.0 107.2| da4 > [...] > 2 1252 0 0 0 0.0 1252 8 10020 1.5 0 0 0 0.0 0 0.0 100.0| da4 >=20 > When the simultaneous write test completes, swap speed in an amd64 = context=20 > reverts to normal, without any user intervention: >=20 > # time -l dd if=3D/dev/urandom of=3D/dev/da4p5 bs=3D8k conv=3Dsync = status=3Dprogress > dd: /dev/da4p5: end of device2046 MiB) transferred 200.020s, 11 MB/s >=20 > 262145+0 records in > 262144+0 records out > 2147483648 bytes transferred in 200.166970 secs (10728462 bytes/sec) > 200.16 real 0.45 user 9.69 sys > 2016 maximum resident set size > 16 average shared memory size > 6 average unshared data size > 106 average unshared stack size > 130 page reclaims > 0 page faults > 0 swaps > 0 block input operations > 262145 block output operations > 0 messages sent > 0 messages received > 200 signals received > 262147 voluntary context switches > 320 involuntary context switches =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Dec 28 14:10:11 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T19Q033mdz55v4n for ; Thu, 28 Dec 2023 14:10:16 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T19Q028Gnz4bdK for ; Thu, 28 Dec 2023 14:10:16 +0000 (UTC) (envelope-from lev@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703772616; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SXnqYsRJQwuGGf5mgNoXOxozPe/uqBh6JQxhQlMCs44=; b=aEVGFDRYg4oKKn+J3HhJpeRetLjhhQtOMrZwoBvbeU4esRJ7aCw8zj52syuzxXFOLAYd5J /3G4vFFspelvcfQXdhpaXNSmI6DUG74LjbBurnGfXgPI6kLLyPeLww5zY29oJVeq+LjmE7 kov27f6VUlmdnAeIgI0usqVs47bnCG2v6hRAtsVW+LMjICZJEuPzEqwFRV/+VTxakqjoV5 UVcQGD45t7S0oW3cgX7IAS3W8qF6UBHmqK3mC/rFsl2g98MsTq+JEiD+bt0P35PIqA7AIl qv2i4C3Oq/ega+vGcpqkjiVPtU28uxUNZaTGzmpRMOrrhvR/W5Pedo+ez7Wy3w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703772616; a=rsa-sha256; cv=none; b=ZPAUirjzdfz7AjkuZrr0DAV+zMQhXr44I6xmW7yGV2xMuJuh279xcerv7ae8QkLthWFmY8 5ccG8zsznC4z9Tsq035dL6OaiF8Mp7drQuEpTwh1WH5p0k8WO1MI4tIJ+Gnf9CkngDwJ8T 1TnpP3p624aWlUbsdvzCutdUa6C8h+UQKOamdxdK4ZSXhaEwuzAyIT8lTvaISniYblPrdx HprJpurlATKDlNxa8iP/zf1v+9551uP6i1hy3BwgShmAmk1thMifBY1bGV8b5mNHVXJEWD ndgOCRE615gJhghP6ZM4g3TFLZXO69I2+U9UJwpbHss+/HDYF8xFepliOLTZhQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703772616; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SXnqYsRJQwuGGf5mgNoXOxozPe/uqBh6JQxhQlMCs44=; b=jNKN1yCnGjxhuy2yWasTpLdAJpjVg1LmHGdpRAe3NX/ZTGV1tZGaByKrFL7i3Z20Co1tbf tbn0EzYnNpyQSVQdrzOl7L6yFf14gIxD/tdKTTr8qECTUGH7pT/HFycpLBagGUk7sA3MNL eWyNJbOtRIhaTLEOZQ8Y1SCdn9ZQJwyy5iBd8UwdkO2DkCZmLlhBtdybDefIV1BtGVrozq rf9QVi0UIcHN+BcyQgXnvQnyvVJ9k/PuKoD2e9g8mFCyGpthqbRQQHiJfmL8FK+UvDNZLr 37oEDf//3/NWa0haYYHXcV4TUyjVlZf/Gtjmq7mDrpnGXV9yubqMWIzILRyOTw== Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T19Q00T8fz2d7 for ; Thu, 28 Dec 2023 14:10:16 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [IPV6:2001:470:923f:1:9d97:3145:f9f2:d8bf] (unknown [IPv6:2001:470:923f:1:9d97:3145:f9f2:d8bf]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id B68F0167DC for ; Thu, 28 Dec 2023 17:10:12 +0300 (MSK) Message-ID: <88ae141a-93af-4f23-87c1-f44a99975a28@FreeBSD.org> Date: Thu, 28 Dec 2023 15:10:11 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ZFS pool hangs (live-locks?) after adding L2ARC Content-Language: en-US To: freebsd-fs@freebsd.org References: From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 20.12.2023 14:31, Lev Serebryakov wrote: >   I can not find any discussion about such problem on Internet. Also, "live" system doesn't have these "z_int_X_Y" threads at all. > >   I want my L2ARC, I've payed for this NVMe! Any ideas? -- // Lev Serebryakov From nobody Thu Dec 28 14:16:54 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T19Yh61SFz55vbT for ; Thu, 28 Dec 2023 14:16:56 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T19Yh5HLwz4cdp; Thu, 28 Dec 2023 14:16:56 +0000 (UTC) (envelope-from dim@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703773016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jlrJpg/+IkvDnNfCzlV5rEoqmWVrz697ZrDMv9uoTTw=; b=xaqJnhLaAmP6OnqyBXEUTLDa2KvF/cviCVMcUDUP6CofN1NdHo3yrF78YnRV844b/F8ItC FSUuIRHV6N/h9Hpk3W8YUmzt1RS3McmqWqC+HAzMfPIX6e35T6p3TrQHGmuUWU5ryo9bcM etht1v2lZ8kwdc8j2dU+JML4vVZ7SjywNbhep1iSaJIvMusl+Iy6SiKoYA44mb6y5iLq3N DRA3rGUeiPqhWDfw4fMwD1XU7khQ2ODcr/CWkKKmHlshsMISE+KIFS9iycJ9HCVZu+VYyM uxxLBRYvle+PRRFu/41uf3n1g/rbONTLnyMr4RuJUxADL1MJjtQHM0qoelj4EQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703773016; a=rsa-sha256; cv=none; b=HVyMBKarks1Uw5HP0tMvPlVAm3mxFBgx/Hvi5B2eYRJl4fCARRD8y6bFDljvCN8h5yFt2i hB16EEI+qjG/UNx0NC943HV5oqQUY8YBXlt5Ozzr/OZUNzBCCDYWyCc5NBN3HzHZMtKrNn 70Cy/FrK3p31yHqHfQySaFhzNEDACQVu9sUQeFdDUk3FkgZXgXOfWTQku3X6/sBNJSxBff xXta+1n72ikStbvUQGIUGIWvdQgyrFR9RlFeDviFyBjNhvpImtrHjhPNhVSyGe8sHasJsK Id6TUxD45RGngATZjjCga23wPzskmCCmlmlfnO/98BHt5HQKWAwD5f+3NSSRvw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703773016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jlrJpg/+IkvDnNfCzlV5rEoqmWVrz697ZrDMv9uoTTw=; b=UMXxBfuvf4g/p7lw7bh2fb1KQZ7uy+fsMyhvhdvGlFxLD/Xc84NxepxEEqjh7JCO0dvb0X bgfErw7vtydKr+HICF0Ko8w0jfZPZO4ty9+A6T3JB9Gk7YVZsQbyRaeLb4/2PNTI4KAIQu RLv7rxbY/EQYLCE9Lo/t484t+AmUd43cA7DE2BVHTQ4EmfkIJmewKEoLLiuXx+nhB9hSCs 8IEUNuPTKdOONhjxmQw5UtBS1r1ulDFfAVTRfss4TOijM3IpJ56qe6N7az8FTkO6ZpH4G1 w42N8XcfQwnbXr1E9ejH3wbBTe/htE5/HGU/BApSWBb4Jj5UWijH9KPQYjiMig== Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "tensor.andric.com", Issuer "R3" (verified OK)) (Authenticated sender: dim) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T19Yh3f8Mz379; Thu, 28 Dec 2023 14:16:56 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtpclient.apple (longrow.home.andric.com [192.168.0.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id B87D93D8B9; Thu, 28 Dec 2023 15:16:54 +0100 (CET) Content-Type: text/plain; charset=us-ascii List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS pool hangs (live-locks?) after adding L2ARC From: Dimitry Andric In-Reply-To: <88ae141a-93af-4f23-87c1-f44a99975a28@FreeBSD.org> Date: Thu, 28 Dec 2023 15:16:54 +0100 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <88ae141a-93af-4f23-87c1-f44a99975a28@FreeBSD.org> To: "lev@freebsd.org" X-Mailer: Apple Mail (2.3731.700.6) On 28 Dec 2023, at 15:10, Lev Serebryakov wrote: >=20 > On 20.12.2023 14:31, Lev Serebryakov wrote: >=20 >> I can not find any discussion about such problem on Internet. Also, = "live" system doesn't have these "z_int_X_Y" threads at all. >> I want my L2ARC, I've payed for this NVMe! >=20 > Any ideas? Do you know 100% sure that this new NVMe device is working correctly? = Are there any errors to be found in SMART information, dmesg or syslog? -Dimitry From nobody Thu Dec 28 14:33:08 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T19wt62kYz55wpM for ; Thu, 28 Dec 2023 14:33:34 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T19ws2zS6z4dtr for ; Thu, 28 Dec 2023 14:33:33 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=Bwwls0Nz; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=3hW8AP66; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.28 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 706C65C00B7 for ; Thu, 28 Dec 2023 09:33:32 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Thu, 28 Dec 2023 09:33:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1703774012; x=1703860412; bh=sNMWXEczvn xGykwB0TkkTOOr7MZcjhXQHD7GHuGKoDc=; b=Bwwls0NzaFkSv2eH0N2k7mWEoi TWorf+B2rhHZ3gs7c3UOXdc8CXXJWqvsLBdLlo/j2yTiAp+sq0nctCdV5RhxzCPE MjtT6kLG9o4SdNJoYqmqOljM9yd3DVexmqsAMJddSX4jzUaQJviaxMvc8X3M9RfK 9QgNmG1PrnfmVjweNmy1BVL6DofaMqLm3VetHfZDCxGhcx1ZsmNjfqFIMlXcS+kk htrBIt+C/C6D+i1XbtRa54HBsPBTXupFAAVq257vqgpBzt79bs+8FbU+8ZNaulba kC5wJ+GOyem/iyCHuPC+ahXSHTtozd/mHDLko4DR3a8fWLEumqArIEKzREug== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1703774012; x=1703860412; bh=sNMWXEczvnxGykwB0TkkTOOr7MZc jhXQHD7GHuGKoDc=; b=3hW8AP663rH6eJxYpPwxeoDxFYEjs4/vr+y6mPCC9Zf4 QoQRQfR0XaR3Aeey44kB2xog222+fffxx0AT0F1XL6NiIXKmMvfv99Di3vyTparJ guNisHh0LqvmerL3JtQ94sXOMJiZEnRYsF/30T8yO+ycvpVb+9AQYdWrM/pFRzDI sw+0PvDI4yKNfv249upV11GqudJlfglgADzlVc3jZCoTXP8GqTmsd9m4/gY8fHxG 7DcUgbl8CUamldA3RumM7lukabt/65R9gLqFFPJdPoOCr13ErwcuNvmEeSbyMrHM NKTJeuv7smAOT8GOBPh/WGJU3C1f1IzpSyKzMJvMuQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdefuddgieekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeeitedvueehtdehtddvhfeuhfevhedvieelvdeiffehveelheegfedule ejudekvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm pehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 202402A2008B; Thu, 28 Dec 2023 09:33:32 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-1364-ga51d5fd3b7-fm-20231219.001-ga51d5fd3 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Message-Id: <949758cf-4ce3-4ad3-a99d-b8e4eb7cb3b7@app.fastmail.com> In-Reply-To: <0D486016-8806-414F-B7F9-66519E8B0EBC@yahoo.com> References: <0D486016-8806-414F-B7F9-66519E8B0EBC.ref@yahoo.com> <0D486016-8806-414F-B7F9-66519E8B0EBC@yahoo.com> Date: Thu, 28 Dec 2023 14:33:08 +0000 From: void To: freebsd-fs Subject: Re: measuring swap partition speed Content-Type: text/plain X-Spamd-Result: default: False [-5.19 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.28:from]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4T19ws2zS6z4dtr X-Spamd-Bar: ----- > Could you produce the same sort of report (gstat -spod > and such included) for the arm64 context? The direct > (well matched) comparison/contrast to amd64 could prove > of interest to folks that work on arm64 support. I can, it'll take a few hrs though to re-create. I still have the hardware but the installation was deleted a few hrs ago in order to try some other things. -- From nobody Thu Dec 28 14:39:08 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T1B3R1yTbz55xX5 for ; Thu, 28 Dec 2023 14:39:15 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T1B3R1Tg5z4fZL; Thu, 28 Dec 2023 14:39:15 +0000 (UTC) (envelope-from lev@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703774355; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VVJYHzQn5bMtuFkzYg5NpYLzJqFh9JeukLbsFDfvlwQ=; b=Cyi69yROSyNyFqe7AqMI/nfKv+76pwiVMh4rhBR+CRPWAXxgOAVGwDrJlSkn5h97Fg045h czEZ/XSpjqRCAibpzfyrspcisNz10SwzuIh0sNa0ZH1EipKQ1CpmeNyMpzkeAB8N9G4zWy rXjUKoPhsfq1NVSVE1ebCoS0jPKD8iyP+ydpg17M0thgDe7jMgKzx/5YEl1J1/MJ0ojAco FVTmzDM0QN07icLPEAtfXrHfPl1M+txKWIpXEroavUvibnAQCtHRlRo4W0LSwk+U0Vtqwo ieY5URKQ+BPG8qt2buIqiQWUqKQ1LVef2lCQZsYPxmrALoBuJU1LcmdJm3Tlwg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703774355; a=rsa-sha256; cv=none; b=ZpD9V/YAsjbP+oQnxfPpFjHRjFR8wTLvTy6gb61VVAhM+0Y9aFXUYwu2YvMmNRp0E/7333 aBXFK5uzbhZzuOojKrK4fJ/Gqc040HreuS+zfRi+/3DUtfaVQiNdQxhLnve5t7hrdsg4ea aZ4L/sdyni9MBY3JQFsK1kPT3dbGGi117DMvhOCkmOVIDhYi2osagi4eB1DSned2Yc2Fk+ FChYl5YwLIxCTu7rDLg6lIGeOx5E+NnVtIXe+boFcFI7I7FEjeGupk3+PpnDKxCeTNjGFJ i4o0F0bW044tFYF975zRSWdKikdwDs9VrH34zbbv9Lait5sZ7+YIkwgbkYigaA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703774355; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VVJYHzQn5bMtuFkzYg5NpYLzJqFh9JeukLbsFDfvlwQ=; b=Qh2ZoSZTEapHIh2xJKsk0o/U5WYyeP9jo9Jow/qLgXCObK49jhdF3LHzFhvc790DuDSTf8 hvukbqHvCQ9EKh6GHOVKz2piJ3K/TeM6Mn/eaGBBciNLhq7h00K+7ihqOq3I/U2utB+QSZ lloFXuAvz1OHAvvF4EuLzJ4N5wwFolJJ9JaxZQ9m1PLlKdQ5k+cSPxaQsaw/5Ck70gjyCh /CMgwuIJAsfkIQiL0gVJwRNB9Z32YmcFnLtZ5zAoP+P4yHunRMQDvUqWV+FOkv0cQqDW5e aB2IZvjgrRswdc+N3KsYww8tE6Ogb1akxpHRSTYdctN1yd1EbKeEmeXIWgBVGg== Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T1B3Q6yN9z3Qh; Thu, 28 Dec 2023 14:39:14 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [IPV6:2001:470:923f:1:9d97:3145:f9f2:d8bf] (unknown [IPv6:2001:470:923f:1:9d97:3145:f9f2:d8bf]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id C11691685E; Thu, 28 Dec 2023 17:39:09 +0300 (MSK) Message-ID: <24d15dbb-43e7-4a20-b42a-b4b1969a4089@FreeBSD.org> Date: Thu, 28 Dec 2023 15:39:08 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: lev@FreeBSD.org Subject: Re: ZFS pool hangs (live-locks?) after adding L2ARC Content-Language: en-US To: Dimitry Andric Cc: freebsd-fs@freebsd.org References: <88ae141a-93af-4f23-87c1-f44a99975a28@FreeBSD.org> From: Lev Serebryakov Organization: FreeBSD In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 28.12.2023 15:16, Dimitry Andric wrote: >>> I can not find any discussion about such problem on Internet. Also, "live" system doesn't have these "z_int_X_Y" threads at all. >>> I want my L2ARC, I've payed for this NVMe! >> >> Any ideas? > > Do you know 100% sure that this new NVMe device is working correctly? Are there any errors to be found in SMART information, dmesg or syslog? You never can be 100% sure, but I don't see any errors in dmesg/syslog and smartctl shows no errors too: === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 49 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 162,228 [83.0 GB] Data Units Written: 135,786 [69.5 GB] Host Read Commands: 430,232 Host Write Commands: 282,975 Controller Busy Time: 2 Power Cycles: 23 Power On Hours: 530 Unsafe Shutdowns: 14 Media and Data Integrity Errors: 0 Error Information Log Entries: 0 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Error Information (NVMe Log 0x01, 8 of 8 entries) No Errors Logged Unfortunately, AData allows to check for newer firmware only on Windows :( Do you have any advice which tests could I try? It is PCIe 4.0 device in PCIe 3.0 slot (as MoBo supports only PCIe 3.0). -- // Lev Serebryakov From nobody Thu Dec 28 15:03:10 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T1Bb62Wbbz5602q for ; Thu, 28 Dec 2023 15:03:14 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T1Bb61yYRz3DDR; Thu, 28 Dec 2023 15:03:14 +0000 (UTC) (envelope-from lev@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703775794; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JDnzxLR2vGDNtrqsE2iO+WYPiotKPzX4nu9u3TB54iA=; b=tC1YHz/E89YAUIgVzJAZIr7/F56HUZYkoL1D1B3pBxkQgtb1o++CLZSwVxW8NLlTRz7DA0 ZgNWMFDYNs5+kxOD1wIgRvRDtcwEPzs0KY3yR3oMf5t0gTP0ekDgHVwTUPTIAAsxumjEtc RwzjV/Gk9DSBLNrSmTG9j9Ja5JQlMjjZbSVMtXCgboo3PeaBJITKY7lKZKunE8V1/tX/vY YguqTtyr2Z8Ep10uVGpU0IKBHGaUYDlnzf1NWB3QKgLCOVwrhFuiPbGvdO79c5N/MSKQHu QtvSrbAvcMQ1vYP9ANv4hz8utL0vN7Y/BdeIwLz4ytBerCLx2s/FiU4mKwirMA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703775794; a=rsa-sha256; cv=none; b=WkbghL7Y6A3Z2vVhZFO+W7DZZT/r0+OSdzLp37vBNM8PR9BHAaiGJR1Y8+Isic3UUbQ7sC dR1ohKN3HSsOO8osYnfKAeVm0hPZFotp/szqzCVr5uJS/pzP5P4IRKS8F2BChP/SMR5mV1 SHCqDzYHuPxJgjykVPrhQgqS5Un3aOpPFxUzAAXtAEyMUTpkS4DkfzMf0NfcoR2TWwP+eD EYAmk6zkWijXu4OxP+l8+5Td8nHDXlcPALb4DCLEhimBQqblAysK8VftZCdQ4FMiFgjiH1 K4qq+fabfiwxxX5abCpx0UipmGCabgWUFZAN8NS7aJWNJ/XsFfOhmbb9wj7OmQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703775794; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JDnzxLR2vGDNtrqsE2iO+WYPiotKPzX4nu9u3TB54iA=; b=vysnTQ9WBBVcza9vChZaNlo0BIjaks4nM2DL/xtST4jwQbSu+lqZgMUYFJTxoLPdbfG/g8 uaMONX+9NyMVv828CuezY2/hTS7vYJdn2aoMJK+LyTMjkyijDeGuXncaFreqS2TAYakr0a W/gPl1PY/zR3e0W2mx5wpWV0LiYMt/ESRHSIRX5MJAcQbqPh7JAgFKYPDVVN0G/RHoLeLE u0ldF8bMtiLIWQ6RxDyyFb51GCiYlDSX1jv85HmYpjJk0xv4pghxD9YUPUdsYHrPwTDl04 pZCXxqUSlhlTnSSx6JfqDFGqTLT3HU93w0Tz4IsBw4pS4FjXYGW0ynxXegI/lA== Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T1Bb60P4Mz33t; Thu, 28 Dec 2023 15:03:14 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [IPV6:2001:470:923f:1:9d97:3145:f9f2:d8bf] (unknown [IPv6:2001:470:923f:1:9d97:3145:f9f2:d8bf]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 2E00616A16; Thu, 28 Dec 2023 18:03:11 +0300 (MSK) Message-ID: Date: Thu, 28 Dec 2023 16:03:10 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: lev@FreeBSD.org Subject: Re: ZFS pool hangs (live-locks?) after adding L2ARC Content-Language: en-US To: Dimitry Andric Cc: freebsd-fs@freebsd.org References: <88ae141a-93af-4f23-87c1-f44a99975a28@FreeBSD.org> From: Lev Serebryakov Organization: FreeBSD In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 28.12.2023 15:16, Dimitry Andric wrote: >>> I can not find any discussion about such problem on Internet. Also, "live" system doesn't have these "z_int_X_Y" threads at all. >>> I want my L2ARC, I've payed for this NVMe! >> >> Any ideas? > > Do you know 100% sure that this new NVMe device is working correctly? Are there any errors to be found in SMART information, dmesg or syslog? I'm running "ssd-steadystate.fio" on this SSD now (not whole SSD but only l2arc partition, which is 85% of whole SSD to have 15% over-provisioning), after running `trim` on it. Let see... If you could advice better scenario for `fio` (with some TRIM added for good measure?) I can run it too. -- // Lev Serebryakov From nobody Thu Dec 28 16:01:32 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T1CtQ4qJcz564jZ for ; Thu, 28 Dec 2023 16:01:34 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T1CtQ4MWhz3S4r; Thu, 28 Dec 2023 16:01:34 +0000 (UTC) (envelope-from dim@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703779294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rM0hdkIH81FFG4E9ek0mL8bHs3c46TNhFTDg45mXz9o=; b=NiEnAIT9C6o8kC6lGyeGsnFhk+RLPAaDQ298Pw5Np3gpHbOoxYkZQAGURuaSsZGrxxOdvE 8uCYq43Qddu+q8eZb4ThGI4RJux7aiSdHCyAWu67C15AuqVb3ePZfPPBzI9zg8fvyJV2PU O/vLdAzCSRtC5+RqbjqgFq2nipgV/CmJtjvXTsSxBi75/jaz10u6vx1BZbsnTcRrzwYI8o oa55IhARLWKA16RenD3L3QEUuW+c9mmNdL5J9zhKrTbKQ1QCIsoHb5wx9AilU3rNNyg0CU xehXatm6Fw+GxrOtZ2FiE/qGwqT7ELIcwkdG6hJfMsdrWrsWFmx4pMY6ZoN7hw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703779294; a=rsa-sha256; cv=none; b=X18zD5wECnknQHFNLn0khMBVz2cfssZCTJWHD4jSKtH59CQ2HG9202DwKA+SDSmE91U3vN +h+J3C4tJU1YINaZTgUTGux00u33JinbNxO4m44ZLRPT/HmxQfZ6MkLtzXixOJsv/u5XeI 6C4E50eXMACfvZ2mtaggalgXTKlWynOkm4OSc4P7rM1WKb8T32GirJVUV1egpyinGOoUrR se0cUFWDedSFy6M+TY8VxNNZArm8ypdCeKdLwDboqAsSZgSTfN42W4vHyOrnJAQHzoEHO4 5mEI4IAivngnZbMzd3P5JbCizUw5eMHXPbciP9+YjlNj1ErCsNuL5BRBEmjF4A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703779294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rM0hdkIH81FFG4E9ek0mL8bHs3c46TNhFTDg45mXz9o=; b=oMZlW8FctpM4fTUktjiD0+jETMiypR5FT1NUczHUGNIQZDdykV4iVTKdk2xMbtzijyPGnq pU6IFCEru4SCjM2DN6Xcd/kU6QBniVseS8BfbnVwOfXMs0RK3t+yESj4u+Yg8O7adYmY2Q d7CYYvFNarLpWg3HlyfzCoxQF2jIBw8/ckyF3m6p6oulQZkH5JDPnSKMZzUxfAIdWay+gu HqQiuHJwCm3QCCDKbTUCI8TJ2foRF05Vvmxee0aH3PEtHghAS6ReKyO0fYbHu9oRtWhF39 tJae1fD9E5SkeEM4D8ZjzTYnTgQc7yLsJYN9OurDMK1Tle2lLg/2JZN4PCTYxQ== Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "tensor.andric.com", Issuer "R3" (verified OK)) (Authenticated sender: dim) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T1CtQ2rWYz4Yr; Thu, 28 Dec 2023 16:01:34 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtpclient.apple (longrow.home.andric.com [192.168.0.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 1CD6E3DB0E; Thu, 28 Dec 2023 17:01:33 +0100 (CET) Content-Type: text/plain; charset=us-ascii List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS pool hangs (live-locks?) after adding L2ARC From: Dimitry Andric In-Reply-To: Date: Thu, 28 Dec 2023 17:01:32 +0100 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <3260F369-4F60-46E6-939A-B3668F347622@FreeBSD.org> References: <88ae141a-93af-4f23-87c1-f44a99975a28@FreeBSD.org> To: "lev@freebsd.org" X-Mailer: Apple Mail (2.3731.700.6) On 28 Dec 2023, at 16:03, Lev Serebryakov wrote: >=20 > On 28.12.2023 15:16, Dimitry Andric wrote: >=20 >>>> I can not find any discussion about such problem on Internet. = Also, "live" system doesn't have these "z_int_X_Y" threads at all. >>>> I want my L2ARC, I've payed for this NVMe! >>>=20 >>> Any ideas? >> Do you know 100% sure that this new NVMe device is working correctly? = Are there any errors to be found in SMART information, dmesg or syslog? >=20 > I'm running "ssd-steadystate.fio" on this SSD now (not whole SSD but = only l2arc partition, which is 85% of whole SSD to have 15% = over-provisioning), after running `trim` on it. >=20 > Let see... >=20 > If you could advice better scenario for `fio` (with some TRIM added = for good measure?) I can run it too. If these tests all work, I would first try making a plain new zpool, = with just this one device under it, and with the same settings you are = using for the other zpools. Then mount the pool somewhere, and exercise it with whatever workload = you can find, to see if that too "locks up". If not, you may have found = some sort of problem specific to l2arc, and I would guess it is best to = take that upstream? If it does lock up, there is probably some sort of incompatibility = between your NVMe device and the FreeBSD driver. You could then try = using another driver, IIRC there are two different ones now? -Dimitry From nobody Thu Dec 28 17:52:13 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T1GLC2PFyz56Fg6 for ; Thu, 28 Dec 2023 17:52:19 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T1GLC1p77z4Dxc; Thu, 28 Dec 2023 17:52:19 +0000 (UTC) (envelope-from lev@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703785939; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yH7hARGaUMLlCt1gPNXc2+4zAJ04SKwM9aiBa43vWYg=; b=yy4infsRQ5lyondVoYIcBBRNWaGD9Zm2rRKGww3+OfgjjJHU1XFsVfdsXWvahXfXcPol77 wFGsHxDxYhfMOKh4iTA6jR8KdWWBSYNnJ3toP42piDmb+0TpIZUo8XM3JuGdWwAm6fd/mu xn1swkyZy0klfLnE0x7KIwrqgoRnAuCLmMK8R/T0LCFX9MzejsmNu2lt7WyENB2iXDRyvW buNzv9tx74szs3hNrL2vBht+qnzU2UrdV8g1uUwgeXX00l8O57Rx3KgVa2iON2KgbF1DZj QLGSbIYoyvywnRk+pfex+D6YSVKc4nr7h94inr6DqJc03zuXP0xfQ2cwh45Sag== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703785939; a=rsa-sha256; cv=none; b=KGOVKwnEi31H4C0voVYQVes6XTffzkSomZaxOLCmFNs3buaPrqazdcY+VoINRI6gndOKMH JXr8EQI81+P5JkaMLL7CXLKQHq3QanP/ZTqfvDFSF0Rhtp+bRY7S3t445zrx36oEVxCAz2 Uj4cNrubQYAfiZY4zzHYqfTZ/sC3MoPDPj7k74HvyLpqn1XtUY96Op31fMh77cXg6lrvXV UFzHhpou9JyNDniuIJCWpcExiq2S7HkCmDatRz3DCnwS8RHnH24Zhlp84VG+djK8oo6ZBm d6elsp1Zq9xsN9ujS58aY+VMEwoO5zyxj9A1rPRFY3nSmA5IDtfwQS12ND05oQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703785939; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yH7hARGaUMLlCt1gPNXc2+4zAJ04SKwM9aiBa43vWYg=; b=LWFi+vQlOlqvMUWrWlO9zosXAZ4W+sCeiWhbeobghqBWrRTG1pHUp5XbLXEak4Xmo3s6WO iVB3HM1t3LW3ttkIzI8HDDMYB4QDeWyJeEayQoCQa0Cf/RUh/lKIMQ2b2E5ZDEj6+n3Pjs A9+xYY6yubsfxhIeOW+HzRylk2gDSB+CYiwhGG7UtieyNVi1l/e0elZneYZpw7LNPAlWVq 6iuViYLYCsw52IjA938cuMWz7WiBnd/CgwxBMPEAexwDHQo/xIJoZ6F6dN8osV2NtLL5+7 xXEC787aYtQSS8Dj24xveTbR7h+2JY/hkmjtYvJTmVjQLHDo6kpS4snQ4q5Yxg== Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T1GLC0Fdlz6Ck; Thu, 28 Dec 2023 17:52:19 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [IPV6:2001:470:923f:1:9d97:3145:f9f2:d8bf] (unknown [IPv6:2001:470:923f:1:9d97:3145:f9f2:d8bf]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 152A916A18; Thu, 28 Dec 2023 20:52:13 +0300 (MSK) Message-ID: <004090d5-cc43-4840-885a-c28297de7733@FreeBSD.org> Date: Thu, 28 Dec 2023 18:52:13 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: lev@FreeBSD.org Subject: Re: ZFS pool hangs (live-locks?) after adding L2ARC Content-Language: en-US To: Dimitry Andric Cc: freebsd-fs@freebsd.org References: <88ae141a-93af-4f23-87c1-f44a99975a28@FreeBSD.org> <3260F369-4F60-46E6-939A-B3668F347622@FreeBSD.org> From: Lev Serebryakov Organization: FreeBSD In-Reply-To: <3260F369-4F60-46E6-939A-B3668F347622@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 28.12.2023 17:01, Dimitry Andric wrote: > If it does lock up, there is probably some sort of incompatibility between your NVMe device and the FreeBSD driver. You could then try using another driver, IIRC there are two different ones now? Driver is the same (nvme(4)), there are two interfaces (with and without CAM support), and as it is not only NVMe in system I don't want to use low-level one... -- // Lev Serebryakov From nobody Fri Dec 29 09:29:57 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T1g8D5Qvbz55jqX for ; Fri, 29 Dec 2023 09:30:04 +0000 (UTC) (envelope-from peter@holm.cc) Received: from kaskal.pair.com (kaskal.pair.com [209.68.5.173]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T1g8C5vzNz4rCC for ; Fri, 29 Dec 2023 09:30:03 +0000 (UTC) (envelope-from peter@holm.cc) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of peter@holm.cc designates 209.68.5.173 as permitted sender) smtp.mailfrom=peter@holm.cc; dmarc=none Received: from kaskal.pair.com (localhost [127.0.0.1]) by kaskal.pair.com (Postfix) with ESMTP id 74BF0D656CB for ; Fri, 29 Dec 2023 04:30:01 -0500 (EST) Received: from Peters-MacBook-Air.local (unknown [94.147.82.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by kaskal.pair.com (Postfix) with ESMTPSA id CBE4DD8DA5D for ; Fri, 29 Dec 2023 04:30:00 -0500 (EST) Date: Fri, 29 Dec 2023 10:29:57 +0100 From: Peter Holm To: fs@freebsd.org Subject: panic: sbuf_clear makes no sense on sbuf 0xfffff80004234000 with drain Message-ID: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Scanned-By: mailmunge 3.11 on 209.68.5.173 X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+a:kaskal.pair.com]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[209.68.5.173:from]; FROM_HAS_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[fs@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:7859, ipnet:209.68.0.0/18, country:US]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[holm.cc]; ARC_NA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; FREEFALL_USER(0.00)[peter]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4T1g8C5vzNz4rCC X-Spamd-Bar: --- root@mercat1:/tmp # cat sysctl.sh #!/bin/sh kldstat | grep -q zfs.ko || kldload zfs.ko start=`date +%s` while [ $((`date +%s` - start)) -lt 180 ]; do for i in `jot 3`; do sysctl -a > /dev/null 2>&1 & done wait done kldstat | grep -q zfs.ko && kldunload zfs.ko exit 0 root@mercat1:/tmp # ./sysctl.sh ZFS filesystem version: 5 ZFS storage pool version: features support (5000) panic: sbuf_clear makes no sense on sbuf 0xfffff80004234000 with drain cpuid = 0 time = 1703841857 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe016a270960 vpanic() at vpanic+0x131/frame 0xfffffe016a270a90 panic() at panic+0x43/frame 0xfffffe016a270af0 sbuf_clear() at sbuf_clear+0xa8/frame 0xfffffe016a270b00 sbuf_cpy() at sbuf_cpy+0x56/frame 0xfffffe016a270b20 spa_taskq_write_param() at spa_taskq_write_param+0x85/frame 0xfffffe016a270bd0 (src/sys/contrib/openzfs/module/zfs/spa.c:1486) sysctl_root_handler_locked() at sysctl_root_handler_locked+0x9c/frame 0xfffffe016a270c20 sysctl_root() at sysctl_root+0x21e/frame 0xfffffe016a270ca0 userland_sysctl() at userland_sysctl+0x184/frame 0xfffffe016a270d50 sys___sysctl() at sys___sysctl+0x60/frame 0xfffffe016a270e00 amd64_syscall() at amd64_syscall+0x153/frame 0xfffffe016a270f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe016a270f30 --- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x116eb469819a, rsp = 0x116eb273c6e8, rbp = 0x116eb273c720 --- KDB: enter: panic [ thread pid 3697 tid 100308 ] Stopped at kdb_enter+0x33: movq $0,0xe38ff2(%rip) db> x/s version FreeBSD 15.0-CURRENT #0 main-n267275-4b817fc1f02c: Thu Dec 28 07:42:48 CET 2023 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO db> - Peter From nobody Fri Dec 29 15:43:17 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T1qQx6h3gz56KPv for ; Fri, 29 Dec 2023 15:43:21 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-oo1-xc2f.google.com (mail-oo1-xc2f.google.com [IPv6:2607:f8b0:4864:20::c2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T1qQx34Stz4Pp4 for ; Fri, 29 Dec 2023 15:43:21 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc2f.google.com with SMTP id 006d021491bc7-5945fef58caso2804451eaf.2 for ; Fri, 29 Dec 2023 07:43:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703864600; x=1704469400; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=R3mN0pq0FYXodIgdBVLc6jL0cwm12nUSILMCTZVdKeI=; b=AjrPKXu/DjntTxucYwxvYZNqmmJr/qwBggYMAJULCSCWLN3PS5x9qwdn92R+jdAHh9 1f34uV1shPmzvWhOsCYfW8Q1zarmvoYrBgrvVb7acDE/GxmpUMYFLk7/uQBSPnKUfHFg S74SVIFcPAJykDjV5IqJt+3+Gm6uqdNMHX6hmu7GfMfRZspq9ig90NXGx1aoYQPxC4gv 6hBbmkzloTdzOC3fwyLhoxUqK6OfHMN/JJCEhyKyu3unh0XxhczSc8wUzptVx65zajUt 0oGYYB9BG9XO0vryp8kDPIhJtBE+Nas8mC/94e3ro6Jh4du6Vh8z2UrQPAmRqTrik8h5 sbhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703864600; x=1704469400; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R3mN0pq0FYXodIgdBVLc6jL0cwm12nUSILMCTZVdKeI=; b=pdJEGkgxU5rfM0P3uIBFtqAlf7PwOBf4+iLYw7SlhECb8Qu7mNrPd/M8+YBLLZFssr h76s/+yCDldKfNo1gImeMwGKugsOmdRn2jDS/KT7joj/w05e8hFiS8hyD4YnYCJw2PPP KH3C3WfXrDyYPv4ksuElWr+vfh3LhES7/2JW6jiEJktIY9wn2gVZ34M5+WYIZeRGPrAZ PfQNszr4rn7Oh/b7H2qiPLcPrzv8fSbNzneRScT6URCJN+rL+3t72hjfxkfen9YiL4Tc DDzI3ioL5nhIghRvNDOqTWFC1RnrtpnSF2at/Oolhl2M1iFDbjNtpbFI9nNOdpLKVheI WXVQ== X-Gm-Message-State: AOJu0YxmFLBfLU7PhDhYEbFwM23vrLq3lRtLtiPMvOWtsnPTxFeNRxR6 XMzpUu/kFN1H40Pc1k8Z4CRYVguXuGO7Bg== X-Google-Smtp-Source: AGHT+IEkTr36Rr5Q0wyHf7VnG/gH/Xbeaqn+r/k8FjncNei2zKhOR1vunsE/U6PwBLvb2n8/roHP0w== X-Received: by 2002:a05:6358:9213:b0:174:cf3a:749 with SMTP id d19-20020a056358921300b00174cf3a0749mr13214233rwb.36.1703864600302; Fri, 29 Dec 2023 07:43:20 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id fw11-20020a05622a4a8b00b00419801b1094sm2513688qtb.13.2023.12.29.07.43.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 07:43:19 -0800 (PST) Date: Fri, 29 Dec 2023 10:43:17 -0500 From: Mark Johnston To: Peter Holm Cc: fs@freebsd.org Subject: Re: panic: sbuf_clear makes no sense on sbuf 0xfffff80004234000 with drain Message-ID: References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4T1qQx34Stz4Pp4 On Fri, Dec 29, 2023 at 10:29:57AM +0100, Peter Holm wrote: > root@mercat1:/tmp # cat sysctl.sh > #!/bin/sh > > kldstat | grep -q zfs.ko || kldload zfs.ko > start=`date +%s` > while [ $((`date +%s` - start)) -lt 180 ]; do > for i in `jot 3`; do > sysctl -a > /dev/null 2>&1 & > done > wait > done > kldstat | grep -q zfs.ko && kldunload zfs.ko > exit 0 > root@mercat1:/tmp # ./sysctl.sh > ZFS filesystem version: 5 > ZFS storage pool version: features support (5000) > panic: sbuf_clear makes no sense on sbuf 0xfffff80004234000 with drain > cpuid = 0 > time = 1703841857 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe016a270960 > vpanic() at vpanic+0x131/frame 0xfffffe016a270a90 > panic() at panic+0x43/frame 0xfffffe016a270af0 > sbuf_clear() at sbuf_clear+0xa8/frame 0xfffffe016a270b00 > sbuf_cpy() at sbuf_cpy+0x56/frame 0xfffffe016a270b20 > spa_taskq_write_param() at spa_taskq_write_param+0x85/frame 0xfffffe016a270bd0 (src/sys/contrib/openzfs/module/zfs/spa.c:1486) > sysctl_root_handler_locked() at sysctl_root_handler_locked+0x9c/frame 0xfffffe016a270c20 > sysctl_root() at sysctl_root+0x21e/frame 0xfffffe016a270ca0 > userland_sysctl() at userland_sysctl+0x184/frame 0xfffffe016a270d50 > sys___sysctl() at sys___sysctl+0x60/frame 0xfffffe016a270e00 > amd64_syscall() at amd64_syscall+0x153/frame 0xfffffe016a270f30 > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe016a270f30 > --- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x116eb469819a, rsp = 0x116eb273c6e8, rbp = 0x116eb273c720 --- > KDB: enter: panic > [ thread pid 3697 tid 100308 ] > Stopped at kdb_enter+0x33: movq $0,0xe38ff2(%rip) > db> x/s version > FreeBSD 15.0-CURRENT #0 main-n267275-4b817fc1f02c: Thu Dec 28 07:42:48 CET 2023 > pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO > db> The patch below fixes the problem for me. I submitted it upstream here: https://github.com/openzfs/zfs/pull/15719 diff --git a/sys/contrib/openzfs/module/zfs/spa.c b/sys/contrib/openzfs/module/zfs/spa.c index a21b0decf6a3..b489266e0556 100644 --- a/sys/contrib/openzfs/module/zfs/spa.c +++ b/sys/contrib/openzfs/module/zfs/spa.c @@ -1460,11 +1460,12 @@ spa_taskq_read_param(ZFS_MODULE_PARAM_ARGS) int err = 0; if (req->newptr == NULL) { + struct sbuf s; int len = spa_taskq_param_get(ZIO_TYPE_READ, buf); - struct sbuf *s = sbuf_new_for_sysctl(NULL, NULL, len+1, req); - sbuf_cpy(s, buf); - err = sbuf_finish(s); - sbuf_delete(s); + + (void) sbuf_new_for_sysctl(&s, buf, len + 1, req); + err = sbuf_finish(&s); + sbuf_delete(&s); return (err); } @@ -1481,11 +1482,12 @@ spa_taskq_write_param(ZFS_MODULE_PARAM_ARGS) int err = 0; if (req->newptr == NULL) { + struct sbuf s; int len = spa_taskq_param_get(ZIO_TYPE_WRITE, buf); - struct sbuf *s = sbuf_new_for_sysctl(NULL, NULL, len+1, req); - sbuf_cpy(s, buf); - err = sbuf_finish(s); - sbuf_delete(s); + + (void) sbuf_new_for_sysctl(&s, buf, len + 1, req); + err = sbuf_finish(&s); + sbuf_delete(&s); return (err); } From nobody Sat Dec 30 07:48:28 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2DrX3lpGz55n8R for ; Sat, 30 Dec 2023 07:48:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2DrX2JqZz4HW4 for ; Sat, 30 Dec 2023 07:48:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703922508; a=rsa-sha256; cv=none; b=pzhEU0NHlRQAmHsinnj9h9MTKa0XsjLgIxysNByL7zyne5Joz7T1+5mkdEEKs68K6yJua+ 2kgwq+kpFag9smbT1ho7wZPtDBdLk96wPfpjQ2K5YArwXb1FKeqHJ1o2JrMyNSBhqx7im2 8Lbg9EVNsdnahYKlTS8sYPrqxM/o2blU8zE/HROHV/+8CFKdjNFeUh77Bh8MVHb+SzvGR8 6JIwm/dTEKr0ut4nr5JRrwK3K3YBC5+/0P/q9OQ5QUPxF5gk3SbJR0gIc/Lc4IrleH7hw8 WYeLgeRmE9hgzOVmIq4OOqbmureir7iaFlzv9rQbJ3forvps+dZg5kjMRQ90PA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703922508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QRww610igeyLQRmT0+C91MVNJ9jQndnJtQasnlpA/ZE=; b=u2qFq+jqA5KokxhZOhIXa0qeQnwrvQV9xiVnGPu0NkFYqGkAtAtECtkcuBANjdPgC2f+gb xaHTQbwSRipMT4gvuXC0c3rS77CeNxSS8lOWofSuy8+bL+1gXRBHBbBFyhZu7Ox7RNiXII rxaM/PJK9gSn2OnwBy6GR7zVddq0IH9QMFpANQBvbdAHbHR7uaAaSiE1U6vWGYcdcMVsWo FQzMZxupDunNw3WFN0m4lAntielBqK/QsvqwKYriuKb08275r+FTOOEJ6eOzo13lWDvDWK itbI4p0Nq8TlkYxXBCBE8JOmtpy92Wch2LQg0RAs6TUIbencBSMjORdeuav6fA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2DrX1FnZzhBv for ; Sat, 30 Dec 2023 07:48:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BU7mSLR064290 for ; Sat, 30 Dec 2023 07:48:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BU7mSrp064289 for fs@FreeBSD.org; Sat, 30 Dec 2023 07:48:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 07:48:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Dec 30 13:01:38 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2MrB51hpz558dV for ; Sat, 30 Dec 2023 13:03:38 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2Mr95GYTz3LH7 for ; Sat, 30 Dec 2023 13:03:37 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=H26TomIe; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Mbu3zUi5; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.19 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id C50833200BE0 for ; Sat, 30 Dec 2023 08:03:35 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Sat, 30 Dec 2023 08:03:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1703941415; x=1704027815; bh=5YvirWGhLB pVlAHRWhPk9Gmc42GTdoFxE9WeoyYGF/I=; b=H26TomIej+9/haHF/2Fb38ZRWm /xAmhZH08cIre/ZIETzAobkAnCn0s8zw044P7fVwBvB4BR2iT34YVWAkDilnf/iF jrR3ImRFzZ2VT5L9yOMwXXBwLP1ys0IO+H3rcchABEAL+/1NdrFDbiHRKTfRfHYr r99Dy7ZuZal8eI7tAxmkSGtJ6eJJsfURBCvh78JxnWcHQPz0qQbMBhlY08aGFGP6 ouTpVbF830zsuCvcbnmgNGHSxCEWLnKMdX18KJ1bg2gkY/b/waPkzKdqSklW2Y/8 jEvqkDxQbbesorNAfN8R1FLusmC54pChPnGgHJ9uO9Vo4CAasgCTpkT/GVYQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1703941415; x=1704027815; bh=5YvirWGhLBpVlAHRWhPk9Gmc42GT doFxE9WeoyYGF/I=; b=Mbu3zUi5p2R+fyJTvlWBxjZtTBZ658VxHODx6I5RbHtR daH1/wl/N/TixkSMhBMi6j6R1o36xXFJM5oCdxE3GY5Kpt0UIWLzFPmqPXRsNAZP QjKozF96fCshi9WIL7rBExkW9iWP/fORQfvVrpdqbE22/bl7Xj39YOVIdiweM5dF fi1C5gpZJuH73W1bd1ui1PDgZGQeebjJInbes18cNvIXyIbOf96YgOcu3j8iACk0 CUg/YCh9vGzN7W1rHVyxIA8KYUZTl0l1pTSZv53xaWGz5BjykltUOZ9CuYfqn692 K1CH+mvUIcQwEGbGaQmuPNUDubYxNrixKx9oLGdoDA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdefhedggeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeeiieehuedtudeikeduieekvdffgefgfeevvdefheffgedutdehtdetvd fhveekhfenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghrufhi iigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 0BA022A2008B; Sat, 30 Dec 2023 08:03:35 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-1364-ga51d5fd3b7-fm-20231219.001-ga51d5fd3 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Message-Id: In-Reply-To: <0D486016-8806-414F-B7F9-66519E8B0EBC@yahoo.com> References: <0D486016-8806-414F-B7F9-66519E8B0EBC.ref@yahoo.com> <0D486016-8806-414F-B7F9-66519E8B0EBC@yahoo.com> Date: Sat, 30 Dec 2023 13:01:38 +0000 From: void To: freebsd-fs Subject: Re: measuring swap partition speed Content-Type: text/plain X-Spamd-Result: default: False [-4.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[64.147.123.19:from]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.19]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.19:from]; XM_UA_NO_VERSION(0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[f-m.fm]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4T2Mr95GYTz3LH7 X-Spamd-Bar: ---- On Thu, 28 Dec 2023, at 14:07, Mark Millard wrote: > Could you produce the same sort of report (gstat -spod > and such included) for the arm64 context? The direct > (well matched) comparison/contrast to amd64 could prove > of interest to folks that work on arm64 support. Differences between last lot of tests and this lot: arm64 context stable/14-n266030-e615bcd32171 boots from microsd hd is ufs swap is an entire 12gb slice I noticed a couple of things additionally 1. when the 'swap slowdown' happened, in the gstat -spod output the % busy for da0 dropped to single figures. 2. when the test starts, it's as quick as can be expected. There's not much slowdown while git is 'receiving objects'. But in the next phase, when it's 'resolving deltas', the 'swap writing speed' drops from 10-15Mb/s to less than a tenth of that. It's the *degree* of slowdown that's the problem. Maybe 'resolving deltas' and 'find / blah' are analogous in terms of disk & swap reads/writes. Observation is that it's not heavy writes causing the problem, it's more like 'looking at files and comparing $something'. It makes no noticeable difference if atime is on or off. It was off on everything apart from /var/mail until wiping the system. At the moment, atime is however it defaults to. reasonably quick at first root@generic:~ # gstat -spod dT: 1.028s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w d/s kB kBps ms/d o/s ms/o %busy Name 0 0 0 0 0 0.0 0 0 0 0.0 0 0 0 0.0 0 0.0 0.0| mmcsd0 1 1535 0 0 0 0.0 1535 11 17226 0.6 0 0 0 0.0 0 0.0 71.2| da0 root@generic:~ # dd if=/dev/urandom of=/dev/da0p2 bs=8k conv=sync status=progress ^C6489980928 bytes (6490 MB, 6189 MiB) transferred 532.008s, 12 MB/s 792389+0 records in 792388+0 records out 6491242496 bytes transferred in 532.733201 secs (12184791 bytes/sec) running the swapfile write test while 'resolving deltas' root@generic:~ # gstat -spod dT: 1.003s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w d/s kB kBps ms/d o/s ms/o %busy Name 0 0 0 0 0 0.0 0 0 0 0.0 0 0 0 0.0 0 0.0 0.0| mmcsd0 0 45 0 0 0 0.0 45 8 359 0.5 0 0 0 0.0 0 0.0 2.1| da0 root@generic:~ # dd if=/dev/urandom of=/dev/da0p2 bs=8k conv=sync status=progress ^C2108448768 bytes (2108 MB, 2011 MiB) transferred 338.007s, 6238 kB/s 257380+0 records in 257379+0 records out 2108448768 bytes transferred in 338.032562 secs (6237413 bytes/sec) root@generic:~ # dd if=/dev/urandom of=/dev/da0p2 bs=8k conv=sync status=progress you have mail ^C317849600 bytes (318 MB, 303 MiB) transferred 336.036s, 946 kB/s 38907+0 records in 38906+0 records out 318717952 bytes transferred in 336.805322 secs (946297 bytes/sec) ############### root@generic:/mnt1-fbsd-ufs # date && time -l git clone https://git.freebsd.org/src.git Thu Dec 28 17:04:27 UTC 2023 Cloning into 'src'... remote: Enumerating objects: 4440009, done. remote: Counting objects: 100% (383231/383231), done. remote: Compressing objects: 100% (28796/28796), done. remote: Total 4440009 (delta 377522), reused 354435 (delta 354435), pack-reused 4056778 Receiving objects: 100% (4440009/4440009), 1.59 GiB | 3.38 MiB/s, done. Resolving deltas: 100% (3528665/3528665), done. Checking objects: 100% (16777216/16777216), done. Updating files: 100% (97320/97320), done. 2049.63 real 2210.75 user 178.10 sys 1077200 maximum resident set size 2333 average shared memory size 211 average unshared data size 128 average unshared stack size 470060 page reclaims 186 page faults 0 swaps 2176 block input operations 2747 block output operations 426413 messages sent 1188099 messages received 1978 signals received 1244606 voluntary context switches 6331454 involuntary context switches root@generic:/mnt1-fbsd-ufs # root@generic:/mnt1-fbsd-ufs # date && time -l git clone https://git.freebsd.org/ports.git you have mail Thu Dec 28 17:04:31 UTC 2023 Cloning into 'ports'... remote: Enumerating objects: 6048197, done. remote: Counting objects: 100% (961/961), done. remote: Compressing objects: 100% (144/144), done. remote: Total 6048197 (delta 926), reused 830 (delta 817), pack-reused 6047236 Receiving objects: 100% (6048197/6048197), 1.16 GiB | 3.11 MiB/s, done. Resolving deltas: 100% (3651525/3651525), done. Checking objects: 100% (16777216/16777216), done. Updating files: 100% (157074/157074), done. 3400.64 real 3584.20 user 191.39 sys 1265592 maximum resident set size 2373 average shared memory size 216 average unshared data size 128 average unshared stack size 890112 page reclaims 162 page faults 0 swaps 7535 block input operations 3006 block output operations 283569 messages sent 868282 messages received 3293 signals received 1670707 voluntary context switches 9390334 involuntary context switches root@generic:/mnt1-fbsd-ufs # From nobody Sat Dec 30 14:23:09 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2Pbx5Cg2z55KQT for ; Sat, 30 Dec 2023 14:23:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2Pbx49fgz3T5s for ; Sat, 30 Dec 2023 14:23:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703946189; a=rsa-sha256; cv=none; b=GTZnySS6VRkuLThp+zcqNo9eJTix5l7eCHcV5ynUM9U5Q4rP0kkHEWF5BNecTDv01Q96Ou pa89X6ZEiHaS2Whnj52IGiHE+4UIFd2oC8wzSoAP9+C1UU0tTIqD4+7hM0ZO/vk8ssDlYx 0lG9Fy2hSiFvQI9xLJK2eVvt8UyBna8fBIvynsqaBFdrO69j9W2CkhfG7j9JTCexf7Txh8 gOhbt2MMvb5AwN3g1HYJTyT4103fMRYOhYIPvGI1ufrx2Vg8eeLBIHBNVulPpPafXpUsiU 2SMGo1XNBJX1yQiZ1CAK6ba8O3CrCW4rZdPFGMDCiLQ8GY2CGQlNZmFMhgG95w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703946189; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CPHZLJ8MROp+U5Ml31eBLuryVA39wVZlJPSqll5NPcQ=; b=KTzt6AHVBxegNHWz4HYx0SvGTZseq36dPLpExKLxGLdnOCY8ZdCN5tMONOAl4IyvzZPszB 1YYGS8nBdJBp9kmhK8NQirdxdo8HK2ON+EBFRr22rif/nsTh5meRk/ffPOLZ9owHqenHhE px4TdssNQAMpl6v/g3CFMKne1z+aDfVzxgQ9gO6cc6lobpL3JMYq+66801S8Z0yzoda2ng X9UmSqysIwv4Z79WBa2QpnyxH3QLfpowR9bOLKJQmqUmA0SoF3Pt761EsJdLLXQd0gWKJ8 y/VJaiZzImttekr05h181VMRn/QRMxNCWioiDfrmr/ukixyycj7UJSr39CWzog== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2Pbx37kRztmr for ; Sat, 30 Dec 2023 14:23:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BUEN9g0061659 for ; Sat, 30 Dec 2023 14:23:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BUEN9ec061658 for fs@FreeBSD.org; Sat, 30 Dec 2023 14:23:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 14:23:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 --- Comment #20 from Alan Somers --- (In reply to Konstantin Belousov from comment #19) > But if nomsyncafterwrite is set, is there an inconsistency even without copy_file_range? Yes. > For the second part, I mean that most likely this knobs works because msy= ncs > are interlined with writes, and really msync affects the next write resul= t, > not the previous. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Dec 30 16:58:00 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2T2c2mqxz55cD3 for ; Sat, 30 Dec 2023 16:58:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2T2c1FyRz4F63 for ; Sat, 30 Dec 2023 16:58:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703955480; a=rsa-sha256; cv=none; b=NNnxhf0wfP8Nvtg0nRcdFHqyKd78/FZOy5v8pInMcMkrhhCEOy+i99wruPPOzPbZ/luiCW fdesVLdnI4bIGkCBueOqb/66n+gGW22UTELQjCkPMj5KXrYcKE78f6S2L8FIRXliFqyrXO wrUNzNBWXf9c3ZFdzvt2MVGb1XLwGuby/W7xYZquO2i5ZFATOB7HUMfYLPfDzZ249mTE5I I7xTi2+rnzh3q2MdK6yi+XMKpIt8Dss0n9s9rhsC6qwF7xUFcalU6yr0bhEP1JXa9AQfv0 ufQP8J/y8lIKNCF4uPhpH8c3SI5TjWLm1YIHWqGtSgv+yK4eBeytMSCD83c9dQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703955480; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BEVq9Aun+S1uQS0kpA4VGk/7Vxn5uiKbmFfbJs6nw0s=; b=uyJ1+pUEiSSh6HobTwU4AAvr3xG4++BgVUQbhi/DqeMU4AtU2JFBY1gRXjXIt1pLE8+r3o upPULetIJDwmvejIDlVwIQVlEsoNo5qMoyktrPeWotdMogciPot1rzUkUiJLAjZ946z9lk kt+xS+XJUy0mTsbLoPtMXLew7oCyLhyYoMlUa9BpLd8wXYVOMj4lCTo+4hQXP8JqhnfzuT UR7F2HG68rZif+RjYYyytKnkfBuwlbpUG272AoifjZ4QWbAo+Q0QYO/r4PVnvQyvcIb+i6 J0IKzsZBe2Rfuq3WDt6+1Pc3lbP0wQmgR1M1pyMhkMiQGvR6YHlxMH7O14ki/Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2T2c0MKwzyWd for ; Sat, 30 Dec 2023 16:58:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BUGvxht085597 for ; Sat, 30 Dec 2023 16:57:59 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BUGvxQS085596 for fs@FreeBSD.org; Sat, 30 Dec 2023 16:57:59 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 16:58:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 --- Comment #21 from Rick Macklem --- Created attachment 247360 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D247360&action= =3Dedit Don't clear the dirty bits on pages being written through the buffer cache You could try this completely untested patch. The last time we played this game, the problem was that a call to vfs_busy_pages() cleared the dirty bits when it should not have done so. This is the only other place where the NFS client code does the same thing. Note that I think kib@'s copy_file_range patch is needed. I plan to do some testing of it, to see if I can break copy_file_range() without the patch and without using mmap'd I/O. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Dec 30 18:02:24 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2VSx0R6wz55jqv for ; Sat, 30 Dec 2023 18:02:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2VSw4Dhgz4LNf for ; Sat, 30 Dec 2023 18:02:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703959344; a=rsa-sha256; cv=none; b=YH59vkzcNC5cbzKVbY9vZcGEB+FmmP195+w5GfCIEGZmR/e07l6uCdQyXglIZDwF5NGlEW 0oIv1HY7Ci2y7BTA6BKfrUo27yOPwesUftqpWadwsV5OphRbtnJl0xqJJDbq4Oq0cgS9xD ffUZZ/05PPU0Jqj3Pf3zYj4nVR4z2bfJCZrG4s1g+62oU4sFvfgi0rfFBz3sg/Gvv3o7r0 VeeqnDCZCXgESIrjO73v2cGJDvVTuh85q1r8VM54BYSCziJ4GVcN6Nw6O7XZ/E/ed6HCRH pAW3vHPeB37lcReb1Vyub6xIFgbD4z2bFDxC1mp2lYdmSLoyZiWEZPy4x1IhXA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703959344; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SW+G7pqku1REYRsbEJ+HdsExRDnycioTBW0mqno/l7o=; b=axUu/1jNGP2MEvLXefb5mNapGedknbM6Po8yGgJZc7giM1/aEKgrlCdXHxt/9zxRqk+4cG oJzrvoItX6WLF6dLzuBgVtq+Efkyf6tXeal7W0yUwpGsKX0aS1djJ0OUInuam24y/nW3J0 BBGECqsQzI57jdRjwqb9SzSv7RAHJfGeSoMGmX6RKYA2CS4ofxW8hQSHEg5R7IQGvHnzzo uM9RqIGUIVgzGJsKSRWTh0M9cuZHbmwPgm4EVsZ24iKJ3WV2EEhcC73G4VPvESB1mtXcp1 08oVkFnsBUckHsgQbKqBqhZ+QK4EcgyD5Axt1/BLntYezuzs6FkqwARFq326ww== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2VSw3Gthz10m8 for ; Sat, 30 Dec 2023 18:02:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BUI2Olu084673 for ; Sat, 30 Dec 2023 18:02:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BUI2OkN084672 for fs@FreeBSD.org; Sat, 30 Dec 2023 18:02:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 18:02:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 --- Comment #22 from Konstantin Belousov --- (In reply to Rick Macklem from comment #21) It would be very strange if this patch helped. More, I think it is somewhat dangerous because it can cause data corruption on itself. Issue is, not clearing pages dirty bits also does not remove write perms fr= om the existing mappings, allowing writers to continue write to the file mappi= ngs. I think that this allows more mapped writes to be lost. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Dec 30 18:32:20 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2W7T0CbQz55mn5 for ; Sat, 30 Dec 2023 18:32:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2W7S6GCdz4NSK for ; Sat, 30 Dec 2023 18:32:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703961140; a=rsa-sha256; cv=none; b=hbiUUAkug9GBNk6UR679txG0EEjKUSLDlyj1/mh2EMvaoprVSbn4GmqxyEF7M4Ry72YhbK yYMj45pxRjh+ZaJfnJFskn/rM1tgYrktuF22YRt0QnzdgH3QkX9x12FCvxXKLOzKfiS29s D3MfAKBSLQZSXLueLRJXvIS972ehfgPdQwiTKh0UoROMadZ7usz3/acY8/j4c7/Z2ApbUm nQkjSCamOMYpg2PU/8FWX17o5IgHpm8McmZhMM5zGIyyVSvuM70cNvmPsNF7jAab1aaf7N 9SWE9Z7tplcZWi6NqAUWyZHULlHVmgnx6xVFLTrvwY9d4CqjwzDNEprvD+IBeA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703961140; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SZ8fYYA0l+RbzkG/vsXUA0tG/q8DavT9X+164yQo1qI=; b=RBI05Dtrp4sg4/rY0KQYZ7r+mnU7Gmor6A2ZM3FmLFriTjhwyymmyTlJmXV2PR20ERkrGu UVnnyD6V8YdrrIlM2fbSwJDQnxj+tPMyhYFK5NsrxDOXkVwdHZ13aGN95RT7dc6kEKe/m4 wjMBCsynyaqrO58Si68XPboCoIQu6JQoMai+hGIktaMBWWG8PDK8+B+GdZzqKocFI995gW e5/fWKxcWrJvNwbG7l9rVYW4sif2g7Ks/xzONX+6m5lDpJa+NfzpZ2Nq45CIGWuL/kvSde Yexh7lrJkf4EDnBOBctSdJ96/gJYQr83ddtwylq08sChpkxrmFGV6nqOHRIV8w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2W7S5MSLz10nj for ; Sat, 30 Dec 2023 18:32:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BUIWKLV033113 for ; Sat, 30 Dec 2023 18:32:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BUIWKfo033112 for fs@FreeBSD.org; Sat, 30 Dec 2023 18:32:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 18:32:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 --- Comment #23 from Konstantin Belousov --- BTW, why do we need ncl_writebp() at all? It seems that it mostly repeats the regular bufwrite() with some omissions, and added reassignbuf() call on return path. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Dec 30 19:07:21 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2Wvs73h3z55qRn for ; Sat, 30 Dec 2023 19:07:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2Wvs5RjCz4QwV for ; Sat, 30 Dec 2023 19:07:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703963241; a=rsa-sha256; cv=none; b=afUYqKxMgLf/qa3VeRXvDIdNjVZTbBp8lCW0l7kr7MzpPQT+PNrqxpmhGugVzhCLjOBpHJ +a0ljzuID5St8f+yr9Kuacgf9KRdJElRsALDDNkeCZmK7gtKuHY+BoTYRdC9fWdbl0h3+Y ehLurKnlnCO4aCHG3/bndAF7y8l9++H9oD0D24uWN9W6YIWq5rEoLHxOnqDPqefdefNCAL WNwqcBM3GPJ5OisgglnWldleyQkMpX/tKg7l78J4uuwX11IwEuYcL3C3aFqN2cMAaCb+tK Hv+1W6YbLCMKvUNFZJ2jm7LTZp30+KETwL6HVCkCHucrPrF5yS0+WGZ+qQUKHA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703963241; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FDYPw0Fye92gSZ9pkU7TuRtE7z4jcjHtiovYXDYB1x4=; b=LU6psXi0vsqUNiDH7zzAu93GNFy6+UtgYTc9jeSnrcs83vIuqcbDFlr2M3Jc9HRflwUqOD G0ZeChfS0eDcDm95EibZNNfOdoSdOu6Znx7CxbJh+E0IIsZtY7O9RchpQLrMPty0NOQl6M 7j+xkEr6Wj5iqRz0Rl/cRzCzpV5CafDQwddypO4p2uNFPtUFKpDmxYWVnFNerVZ5f7iowd hP7fVb2V3DE0IJdA29EJRdcGRZLcmNqisAoC2iN480NZSN/ucn08PdOAhhQHoq1LuxuwzF RAJNnKCN4hUI+P09kGoTOUK/RqAXAR06akVVDxdghzWJ9EL3DUysZQmTktZ5Tw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2Wvs4WNGz12Tp for ; Sat, 30 Dec 2023 19:07:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BUJ7LFm081263 for ; Sat, 30 Dec 2023 19:07:21 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BUJ7LpH081262 for fs@FreeBSD.org; Sat, 30 Dec 2023 19:07:21 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 19:07:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 --- Comment #24 from Alan Somers --- FWIW I can also reproduce the bug using exactly the same setup, config file, and seed on bfffs, a FUSE file system. I cannot reproduce it with fusefs-e= xt2, fusefs-sshfs, or fusefs-exfat, however. I will now work on adding a unit t= est to the fusefs test suite. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Dec 30 20:39:36 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2YyK2s7sz55yfw for ; Sat, 30 Dec 2023 20:39:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2YyK1pvYz4YcN for ; Sat, 30 Dec 2023 20:39:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703968777; a=rsa-sha256; cv=none; b=cDP1fuKlfU9FvPINTIpBRXXhmv3t65KqCUJAXVPEDjqtO7hFMTX9aWx1gkHUAQqARM/lPR /t6v3nriNf0UafUv3aQjhK3lC+E3sAgI2mbT1eGo5ledRFrNnFSCxkUphZggNaovN5DzdD AndrM1dQ+Zhv1+FG+oMvzkUa1/zp9LOKGGKJkMpamF58wJ/hUktTHayFxJoSD2XtQoOGP3 3DRt7+/j8fGcrIjzktD+Vx1H5qZK69KcfkfrzdTl8qHaxU91breVRIokTFmsxA2B4BCr2U UfCjYut0UJbxxIl2xqY9d+4oGAoAUfex2IQTIGDguMHAtuSVllSn74OGrLXdoA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703968777; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yuZMQDXn38VUWN4lKf2xWxsYFv6DDXeqcnw46SHyLbo=; b=p3LGVDRZPEv+kiVt8O7gcQaZljYjJnFtH7JYjSrK+H9S9Kz6tQkdgWe7EtWrGpIMHCtLsj l8npCb85taFagWi7Q4l7zg0VhzQ5vGLsXnZbepV8O3vreHy0MfzAcIeKmowtvstGgewbKW QN2g9EH67j9fem7gwiv3yXI+elGvGnKcGsUC8XLryvN9/ibUAChrwBZB/m1/QRBGXO/IbR uMl9ep4si5sMAo3qU16uWZJHLxhYcq4xvnFF8zXwJnqhP3iDedCmV/C6nGRXmcLG8NftVn fj+EzBGoy7s0+QSu2/hRyQanz4EQupjMejupk6oVIES9ZPiEXx/WXJj0Vcl+Ww== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2YyK0v46z14TX for ; Sat, 30 Dec 2023 20:39:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BUKdbXC015828 for ; Sat, 30 Dec 2023 20:39:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BUKdbWW015827 for fs@FreeBSD.org; Sat, 30 Dec 2023 20:39:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 20:39:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 --- Comment #25 from Rick Macklem --- I did not suggest the second patch as a real fix. I just wanted to see what (if any) effect it has on Alan's tests. (In the past, I've found that almost random changes can help identify a problem, like my suggestion to get rid of "nomsyncafterwrite".) If the patch happens to help with Alan's test case, then we need to figure out what needs to be done to fix it. W.r.t. ncl_writebp(), I'll take a look. This is all very old code cribbed from the old NFS client and not written by me. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Dec 30 20:47:12 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2Z751prPz560Cf for ; Sat, 30 Dec 2023 20:47:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2Z750nccz4ZVT for ; Sat, 30 Dec 2023 20:47:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703969233; a=rsa-sha256; cv=none; b=w/oi6MSved9h/aDM3Xg2NbfrJyb79UuspLATxAvwuBihF7iKncIQ92Xy1CNxjjN30MiKWt 5mXkFitrKCsCUAbSx4zd5E3SfLhgcgrHqklHsbmBHgkMGHT65n/X5J5At5YHZUeGAYHPTT KaGyVcNGpQNKJfiIGkuJh3w5WMLgIR94ps++my0Xl3i44BLhSzph3pcYvw07kx5EPaAPpl dGUDh797gB99/7TaVeFziCRTUWl2EOKGgLS6B6XCxWlkdR00WRDUSkZVtdUV8EGyigU2i2 j1eaX4qzMny2/UzV6+4dP4gJMglBkTLEWEryh9gaMzWhbxzM7G3IZtxdF6kPVg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703969233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G71tTwNgWSp9NYj5P9Oll6GhfNTZKQ4+nP8u7m8sMWA=; b=cA5k40mtYzZwX/PP/qEerUe3H64xlks6j/qNMjcLxe3RuLxdhGgf7JwGYfXfv5/rqTdlxx kfD/QfnzuCJfnQ5JOP8ZtWFXBZk+h3qVEjiI2T2iy/N5chzrfgxjNtNZp2Pc7nbwL6+jHV Yr7MhqlXhW40GFIuh9BRGmt+OiAi40hxXQIhwy88QKOMJeVSelqUtsJteqDZY/3XAbt7G5 enJeCV7k6ev+H5tzug/qpXWLHFvFJmCaxQzUjDiXsb5D2xJAJLcKcPfGvbxo/t0Lp1FEtW q1OJeFoGYufU/fqso+K7BQoBWnOOcKhmTw9WkpQ0vo2z1SNm3CCXJ/hbLmpuJg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2Z746zf9z14vm for ; Sat, 30 Dec 2023 20:47:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BUKlCac029539 for ; Sat, 30 Dec 2023 20:47:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BUKlCCL029538 for fs@FreeBSD.org; Sat, 30 Dec 2023 20:47:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 20:47:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 --- Comment #26 from Alan Somers --- I just tested rmacklem's latest patch along with kib's. It does not fix the bug. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Dec 30 22:23:27 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T2cG76vZBz55NxW for ; Sat, 30 Dec 2023 22:23:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T2cG75pm4z4jYG for ; Sat, 30 Dec 2023 22:23:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703975007; a=rsa-sha256; cv=none; b=OByrgT3Z3G2mVhXoIv7FARzxEbG9uZfgIDkDE6P8kfhZ1q25Uw82q+8fQGdMDAOSomRtVq 7fdDYT69dLLP3sdawa329lG7/XnF23eIyfdDEJqbEm5C7+ZjrCZ0izizZNdvDrxV0ej+qu wnpRQwVfC/uaXCjddjje61kyMA7sG7vTB3scA8JV2HT4vOq4IyRKERmTjHEcT4uIU9wf8l p4RkjzlRPsuGQC075JkSpHnnB65PZ964HaUGcv+wB/CYuLQidxm+17LOXN8I2e84LNZzgA YdkteJBktKnjmsnTffmYpt+MizCczwoEppcJKQCOUb9De+fZ7DZxxqe/tcudkQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703975007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/pNL417bPdyExs0F8qDJg3Uj6Oct/N24EMNhXECBElk=; b=s4rAmKJ8ptUw2TD4gNDNBPIJzVyAsk/P4GC2I+0auXF4QAg/JJ8rt8NudDZpBiZAJsKm33 45UtgHH1t/cKCm1lFZtzmTw4+Iv/WxZjLIvVEaCFj5hdHM/iakWIzJS2EJz4f670Uwn8CN HCvSucRkHmt/jld9QrUHUNtxfGlKKW7gOert9fb/9CdLWUohm75CItAoEvquZuDLD4EvRm w6XLX/NYvX9N4Zbof4z/61ZUHT9WeC2JEu+oCB5NlDHbYlJNcOKa43aE/qeIJmFnt9Q1jq GcIFu30b4nKyFQcvEmWoTrXfJW8AAnfj1c+H9NwVmziZAZm0gbHms9f5JNNW5w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T2cG74s7gz17bw for ; Sat, 30 Dec 2023 22:23:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3BUMNRjj075398 for ; Sat, 30 Dec 2023 22:23:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3BUMNRlM075397 for fs@FreeBSD.org; Sat, 30 Dec 2023 22:23:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276002] nfscl: data corruption using both copy_file_range and mmap'd I/O Date: Sat, 30 Dec 2023 22:23:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276002 --- Comment #27 from Konstantin Belousov --- So we have an nfscl bug reproducible without copy_file_range, right? Could you try to determine a minimal set of syscalls types needed to reproduce it? E.g. is it just read/write/mapped write or something more is required? --=20 You are receiving this mail because: You are the assignee for the bug.=