From nobody Mon Feb 5 20:49:37 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TTJQn5h2Qz5903w for ; Mon, 5 Feb 2024 20:49:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TTJQn2FTgz4yhC for ; Mon, 5 Feb 2024 20:49:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707166177; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=twICTko+pcNA9p0alonldFIo5Tn7sKAuyuW3+xBpNQM=; b=hiHT9yV3Dawt+TY2PYt+FS0xKKyIW0WBRoaDVa5qPGinFsNgU5vOfLZHW8n1QM3SyrBU1U SMasDcTHLr19/TJHYu5zQbmChMBXsBzjNlsMJHwKWHtkBtp6R8UUhYZXIYY/pFa2OBFOV3 KFyKT8w48hMHyqgkVgrP8f3e+vK2Ne7WlITMpTqEKuviCTor57qoSB1ZX8YxrVymBNL/lc 7c/hyI2sxLg6aOWtCYoh7eEx1/8VqcJ8ZmWjeFyuBfAfwee28xFVnuWoDk3PNUHSrqLx10 6tg9W+XF3kNkE1+LhZfHecaNyOvAa6lHmE5RHLTJVZAJ6WR8Wuj8QF2c47ccUg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707166177; a=rsa-sha256; cv=none; b=q/P8kp2UNrNX7/haO1J8Io6iKaxgzCIxOR5XUpg+6DUHJa0J4Gf8rwblYNoK3guRq0kD/o NGQ1lOf33i7nVt9RqRPQTHBDTmXCiAg16EA38gUwJ8dG51FkVA1yTqXvP9b9ot7ycskSXx /eiiUghYMeuvuw/KXujLYj97SGyAx3nsAHf9LG2M6zbWwf1CncoeXO1kdgG0OA+hxz2Gzm Fojn6+2xAERf6lXhbt6n5JzDVZ8gzZzjR9/TkRf/9BH2jWAmjiqA7mUHp1wIcon4gGwV9z b+a/+fPtdjDoqoF+xzfBhIuaNF/AdVj25UzTT4nya94NUJDqXdkRIHgjUfxLSA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TTJQn1JWYztsl for ; Mon, 5 Feb 2024 20:49:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 415Knb5g011122 for ; Mon, 5 Feb 2024 20:49:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 415Knbl0011121 for fs@FreeBSD.org; Mon, 5 Feb 2024 20:49:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 268005] rsync to FAT32 flash drive gets "Freeing unused sector" errors Date: Mon, 05 Feb 2024 20:49:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: se@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: se@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D268005 Stefan E=C3=9Fer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |se@FreeBSD.org Assignee|fs@FreeBSD.org |se@FreeBSD.org Status|New |Open --- Comment #5 from Stefan E=C3=9Fer --- I have been able to reproduce the issue with alloc/free tracing added to msdosfs: usemap_free: cluster=3D185185 usemap_alloc: cluster=3D361562 usemap_free: cluster=3D185186 usemap_alloc: cluster=3D361563 /mnt/FAT32: Freeing unused sector 185186 2 fffffff8 /dev/md0: remounting read-only due to corruption remounted /dev/md0 read-only In this test run block 185186 has been freed twice in a row. I'll ltry to identify the code path that causes usemap_free() to be called twice for a single vnode operation. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 6 21:35:29 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TTxPG243Mz5B1MN for ; Tue, 6 Feb 2024 21:35:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TTxPG101Rz4fG0 for ; Tue, 6 Feb 2024 21:35:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707255330; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vHin+xsoGebFpu9Fgs4ua2s0Jv0HlyebnVMXJ0s5L74=; b=CPjX8cno0XcTFgctWgA8+4+pP54yvXMAoY6CRpftsd5kEVyPUz634qgZytkmo9RLvLa/eu z4ULylOWibPTzRRouSgF81aojGrBu538Ls7K/b2aqZ9y4YDtCq1bpsSaGd0yINfAcIbco7 V6tzZgdxn64bqn/882Co6sbXy47LyrpQTLfKFJ5AHv3Vrx7NcqnsJEvt9/I4YT4cEf9Lt2 rdizlfbHr54QobrzBP8nToChV4NbqwuwqJE9DHHehdPmAg6YswspmXzIM0Ei3lYoKR+Sv7 SCfgYAiD200O0Diluh/WNV9zXyWXEZLNJAi9NAZaDCXbeYLXSJEAawYKWs60FQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707255330; a=rsa-sha256; cv=none; b=qUrk+TDCzjrxLYE0ljmeXjEWNRUDRIlIG4XtD7hm9EE2rHNWQ20ynMGX77/oEQRMnqW6qM cxIuuQ30YaYQ6V0CRCQoRQd80X2pbwZEgU6uxJm+Zpt37V1SZIHIFSpTTGbPLwp8b1A8Yd dlBX/itLUZsJo8L7YRcWqxywdhDK6JkFhcLIDvAVlFemDTEYQ2MnlqTP0XyIWSRROcI+HM eMI9enwJaNO0/xI4H3qewAocJuC1dFIR44htsIgwv1JVkqQ0kCIX2i2Yxc8PJw4qdzPPoU zTZW82PHjUzQ/rV9gAjvoM+ZiBwMk+A6OGSVHJFU+vuKFYhAWclQiInDPFH/hA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TTxPG06Chzgbn for ; Tue, 6 Feb 2024 21:35:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 416LZT86073959 for ; Tue, 6 Feb 2024 21:35:29 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 416LZTPa073958 for fs@FreeBSD.org; Tue, 6 Feb 2024 21:35:29 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Tue, 06 Feb 2024 21:35:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pmc@citylink.dinoex.sub.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 Peter Much changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pmc@citylink.dinoex.sub.org --- Comment #43 from Peter Much --- Just installed 13.3-BETA1, machine is doing effectively nothing: System Memory: 11.05% 864.11 MiB Active, 33.27% 2.54 GiB Inact 28.36% 2.16 GiB Wired, 0.00% 0 Bytes Cache 21.25% 1.62 GiB Free, 6.06% 474.05 MiB Gap Real Installed: 8.00 GiB Real Available: 98.15% 7.85 GiB Real Managed: 97.22% 7.63 GiB Logical Total: 8.00 GiB Logical Used: 47.97% 3.84 GiB Logical Free: 52.03% 4.16 GiB ARC Size: 3.49% 244.88 MiB Target Size: (Adaptive) 4.04% 283.59 MiB Min Size (Hard Limit): 3.58% 251.26 MiB Max Size (High Water): 27:1 6.85 GiB Compressed Data Size: 160.01 MiB Decompressed Data Size: 347.37 MiB Compression Factor: 2.17 PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -8 - 0B 10M CPU2 2 406:21 100.00% kernel{arc_prune} This is not useable. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Feb 7 10:55:54 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVH9568VSz5977K for ; Wed, 7 Feb 2024 10:56:09 +0000 (UTC) (envelope-from andrea@brancatelli.it) Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVH9509HHz53H2 for ; Wed, 7 Feb 2024 10:56:08 +0000 (UTC) (envelope-from andrea@brancatelli.it) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=brancatelli-it.20230601.gappssmtp.com header.s=20230601 header.b=FNQ6sGm1; dmarc=none; spf=none (mx1.freebsd.org: domain of andrea@brancatelli.it has no SPF policy when checking 2a00:1450:4864:20::62c) smtp.mailfrom=andrea@brancatelli.it Received: by mail-ej1-x62c.google.com with SMTP id a640c23a62f3a-a3850ce741bso54364966b.3 for ; Wed, 07 Feb 2024 02:56:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=brancatelli-it.20230601.gappssmtp.com; s=20230601; t=1707303365; x=1707908165; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=zDtRoYGWs1GS4QKErvwNoLfIoclYmqG7s+YkAmpp8qI=; b=FNQ6sGm1U1jNqqCLY6ImtYLVGqj4VTZcUn7AFRO6S4gPvJtvvOYbXo0KFqRSuEpP1T W3sPtMg9EEDOZE97OlkBA2oxQExCDc9P5gxas70/G2/UZz4Eu1EBJ2OHM4bf70MhhVLY Up2Ua4voAe4ezrBv7+R6cB4uBr5OfUle/5aBNi1qMSCbyHBUdP0tW6DBGodqcSDkOOaw RcmrueHHhRC0bN4e0zQ1TywDFaioc6JtVxiln0muMDzTRUo5iBvhreKFtl4UoxLcGgZ7 r8NRysUaCtbd3rg7oaR9kC3n3FLzZXq8Bb+2pQWzBlfPCp3Ni9zTFykBly/SnkXyDnFb zmtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707303365; x=1707908165; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zDtRoYGWs1GS4QKErvwNoLfIoclYmqG7s+YkAmpp8qI=; b=KSmPHW+ODiF5FUuR4hOARNO4zSHETtQvBaz5WT6q2dZAIr5dhIwIu8jzQtEkV6W7Xr e8nKPz+JYpIic3GmaYlZ58lBH9CK+R7QFaUP8h5nA7JEQ0sy9C2WsgPzQFX3c4xcMuxM lhjGo/KDy0vtPFf0BGeA16ANoydOhXDO3jHkIjt+erfjmHKRiYRYiRpWDJbrHP1McDJ6 uiTKfsqkoUEtfI0KNrXHMS1JiJreNZsffR58uxXAc7QYcp0O7S/9fECpmOKpINvJv/KK VZSQQNvCAAN3+bItMDGVu5wXGGNPKhpqEenM0FBngEGpGANUfJo2t5gtT0eMneLbWLqo /zOA== X-Gm-Message-State: AOJu0YzlmIkJpFem9Fy4b7VaK+QQoicCgzv70TwTY0FF24lyCbl+XKEY Ukrq6bVRJYYmDYAP0ZUiURVGEVLfRBrvsTnlSI7jPHFP98FMgPux7P0Zd3ioUUz6EUBx0s93c3Z Pgjir3jhnO71PbKamKTeuTNqtHQKkV0/plYjrcMcZRlZImxdE X-Google-Smtp-Source: AGHT+IEX7H+xOFqDleakDq2Zlfb8f6Meg41IkXi5gKEemcgEfSWL3oVU5zlay/2qXUsH83zNPQwbjKDFWwOGGdewx0I= X-Received: by 2002:a17:906:2929:b0:a38:58ef:c14d with SMTP id v9-20020a170906292900b00a3858efc14dmr1362562ejd.67.1707303365487; Wed, 07 Feb 2024 02:56:05 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 From: Andrea Brancatelli Date: Wed, 7 Feb 2024 11:55:54 +0100 Message-ID: Subject: ZFS on a shared iSCSI To: fs@freebsd.org Content-Type: multipart/alternative; boundary="0000000000009fc66e0610c88a16" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.28 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.982]; R_DKIM_ALLOW(-0.20)[brancatelli-it.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DKIM_TRACE(0.00)[brancatelli-it.20230601.gappssmtp.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_SPF_NA(0.00)[no SPF record]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[brancatelli.it]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[fs@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62c:from] X-Rspamd-Queue-Id: 4TVH9509HHz53H2 --0000000000009fc66e0610c88a16 Content-Type: text/plain; charset="UTF-8" Hello guys, I'm not 100% this is the correct list to ask this question, if not please feel free to point me in the right direction. I was wondering what could be the best recipe to have an HA cluster sharing an external ZFS storage. Let's say I have two servers running a bunch of Jails and, thus, I'd like to use ZFS as the underlying storage layer and I have an external (iSCSI) storage connected. Would it be "easily possible" to have some (2?) iSCSI LUN exposed to both servers and then activate the pool on one or the other server? The idea would be to reactivate the filesystem from server A on server B if the server A fails. Would it be "easier" to replicate everything and zfs send datas back and forth? Clearly that would mean doubling datas and havin a scheduled replica with a possible delay in data replication, so I'd like to avoid this. Any thoughts? Thanks a lot. -- *Andrea Brancatelli* --0000000000009fc66e0610c88a16 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello guys, I'm not 100% this is the correct=C2=A0list= to ask this question,=C2=A0if not please feel=C2=A0free to point me in the= right=C2=A0direction.

I was wondering what could be the= best recipe to have an HA cluster sharing an external ZFS storage.

Let's say I have two servers running a bunch of Jails= and, thus, I'd like to use ZFS as the underlying storage layer and I h= ave an external (iSCSI) storage connected.

Would i= t be "easily possible" to have some (2?) iSCSI LUN exposed to bot= h servers and then activate=C2=A0the pool on one or the other server?=C2=A0=

The idea would=C2=A0be to reactivate the filesyst= em=C2=A0from server A on server B if the server A fails.

Would it be "easier" to replicate everything and zfs send = datas back and forth? Clearly that would mean doubling datas and havin a sc= heduled replica with a possible delay in data replication, so I'd like = to avoid this.

Any=C2=A0thoughts?

Thanks a lot.

--
Andrea Brancatelli
--0000000000009fc66e0610c88a16-- From nobody Wed Feb 7 11:25:27 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVHpz4lncz599kl for ; Wed, 7 Feb 2024 11:25:31 +0000 (UTC) (envelope-from ben.rubson@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "Telekom Security ServerID OV Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVHpy2TyBz55dy for ; Wed, 7 Feb 2024 11:25:30 +0000 (UTC) (envelope-from ben.rubson@gmx.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmx.com header.s=s31663417 header.b=XAcZM8pl; dmarc=pass (policy=quarantine) header.from=gmx.com; spf=pass (mx1.freebsd.org: domain of ben.rubson@gmx.com designates 212.227.17.21 as permitted sender) smtp.mailfrom=ben.rubson@gmx.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.com; s=s31663417; t=1707305128; x=1707909928; i=ben.rubson@gmx.com; bh=ki6iHIENX+JOF5chFeTB4iOft2jRga38XFZ2S+B7k4E=; h=X-UI-Sender-Class:From:Subject:Date:References:To:In-Reply-To; b=XAcZM8plOxJDM8J8b2gR+dGhl3+d9i4qk/q3qyRBqxVKZ6X0xudUhGxQOPDBQ1Ys S7XPnpXB9aHW9zfPs16BJQwWwhqhU5QEov0Cq2DXg7YVG5Xy0PgwAGxE6iBt0ao36 BmIBF1xbtPxBbyyuzOpxMD6ZC/dqR+dckBx7w0xcb+H2WgdjPo/urXrf5YdgC6iI2 qEp9Pjmh/nrE7bQudnP8zrHEzTH+gcb7rhvLUZUXNSAo6QDSsuPKSN2YRoi700ryn MIoiB/1recud6vo8HVvbcd3EV4nYGXyaBztK9i/AWbPcUDp8zO2yq/HqBoLQtdrip sEQn7K1+o35efVQuRQ== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([45.13.49.129]) by mail.gmx.net (mrgmx104 [212.227.17.174]) with ESMTPSA (Nemesis) id 1MKKYx-1rIU3L0kdb-00Lkp8 for ; Wed, 07 Feb 2024 12:25:28 +0100 From: Ben RUBSON Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS on a shared iSCSI Date: Wed, 7 Feb 2024 12:25:27 +0100 References: To: fs@freebsd.org In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3731.700.6) X-Provags-ID: V03:K1:wcaWF15g9IpPBoqDpJmd/XjbUYPbrAQlkBIApfBaY6VLKBTO2Ku CqnvTKo20uADgumSJeJZZ3BpOT6enqvLd/d210ST84QCI0+SWxO/9R7p2ytVppiDX5KTC0s 84N01CPUTPUGMlq9PzSDFSn4LaBw/MccbkOtE2dNRo6mIHyCCjf3mmPRGYIENVYfRj1cXoI TAI7BxM3wLFkadkB57KOg== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:yquS8lh6dF0=;aIEdA9RbiSctOtf8/71AlA9OKCn WRCFjAASdfiyqJHG0GNTGOWhHvbOAMai4IByGtIOcaO991dsGq8eDPDUgT+J4//tScj3lD8rQ 6MC/hpX+wPqCvmH9TEBXOMhQY6AqL0+vZDmteQw9LlCKmAf9xMOZO6QOPzeEicEknP3AvaK1l AggKdRO50cQbMhMVCCbn1Xmqz8Pxvn2jMN6MnOieL3IztSsxOwHYfKpzfb2GPBZ9+VHBRUInO rkakJLv/nH+A+wIhrLhkOwqlIcy7bgoGPm4CQ4+e3dDiHkkX4o+aW8EUQPp2wh5vVRJ2mGEYi mq+W3gOm/oCk9mn3s29xUuAKBB0wojQyvxrRQrTpiSNJFBxt9kwwNUVOZSckBAe/IGh8LZH8D l6OxFSFz6ZgTTZrHTgPyOSrsvo81RogHVr6U+TXN3mGR66AXwHZJmcz7aMQ1d3Zj4Iwj5Vv3f 07olakSVNAGZReXwVTAAPVwYK+KeWs5H26ppXk7bY+FgYrmDq5pn1dosfvhwcRXgA7nJOrZy0 ZeIO+fu5sm5jbX0rvpXZH8zZPnSQDRJ/1PkqORAsJGUFuCqQ8aTHQXhairpLSLDWsd30IHeeG vSRe18lqP+6vtSFhvMgM7/u8iJoJUpTofj49xpnGL3tBHoWqzWHdbMFv4Y5j50QWiSrKSNpVl rb+G/9BDdLK63Xwr0ZEbl7C2Jw4fDBtpezeL/r5QIDIQ7hom5KEDGgX2HSrG3wZkT5xKliOj4 jBM+GE/f2d//Xu0ozZ19O8V6z2HuYR+1dePvzKHIN01oYMSL/gQdfo01b+uicAC9Pxx0ohRrR aCnhH0qD4b2h0/brTfiqCUnXdxChiNjQq0DoN7dnCSBu8= X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.39 / 15.00]; DWL_DNSWL_LOW(-1.00)[gmx.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmx.com,quarantine]; NEURAL_HAM_SHORT(-0.29)[-0.292]; R_SPF_ALLOW(-0.20)[+a:mout.gmx.net]; R_DKIM_ALLOW(-0.20)[gmx.com:s=s31663417]; ONCE_RECEIVED(0.10)[]; RCVD_IN_DNSWL_LOW(-0.10)[212.227.17.21:from]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[212.227.17.21:from]; DKIM_TRACE(0.00)[gmx.com:+]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_FROM(0.00)[gmx.com]; MLMMJ_DEST(0.00)[fs@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmx.com] X-Rspamd-Queue-Id: 4TVHpy2TyBz55dy > On 7 Feb 2024, at 11:55, Andrea Brancatelli = wrote: >=20 > Would it be "easily possible" to have some (2?) iSCSI LUN exposed to = both servers and then activate the pool on one or the other server?=20 Hi Andrea, Yes, I have such configurations : 2 servers with same JBODs configuration. When zpool is mounted on server A, mirroring is done on both local disks = and iSCSI disks from server B. When zpool is mounted on server B, mirroring is done on both local disks = and iSCSI disks from server A. Works flawlessly. And you can have one of the 2 servers offline for some time, when it = will be back online, sync will restart. What you can't have is zpool online on both servers at the same time. So you can do the same with iSCSI disks only. Disks will have to be mounted on one server at a time only. You'll have to have a rather low network / iSCSI latency, or it will = fail / zpool will be unusable. If you have high latency, you'll have to go for ZFS send/receive. Ben From nobody Wed Feb 7 12:24:29 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVK7B03QMz59Gw1 for ; Wed, 7 Feb 2024 12:24:38 +0000 (UTC) (envelope-from SRS0=uefI=JQ=perdition.city=julien@bebif.be) Received: from orval.bbpf.belspo.be (orval.bbpf.belspo.be [193.191.208.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4TVK793tsgz59Tr for ; Wed, 7 Feb 2024 12:24:37 +0000 (UTC) (envelope-from SRS0=uefI=JQ=perdition.city=julien@bebif.be) Authentication-Results: mx1.freebsd.org; none Received: from belspo (unknown [10.209.1.137]) by orval.bbpf.belspo.be (Postfix) with ESMTPS id 30A861DB40; Wed, 7 Feb 2024 13:24:30 +0100 (CET) Date: Wed, 7 Feb 2024 13:24:29 +0100 From: Julien Cigar To: Andrea Brancatelli Cc: fs@freebsd.org Subject: Re: ZFS on a shared iSCSI Message-ID: References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ppdebwkw7x3q32sg" Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4TVK793tsgz59Tr X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:2611, ipnet:193.191.192.0/19, country:BE] --ppdebwkw7x3q32sg Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 07, 2024 at 11:55:54AM +0100, Andrea Brancatelli wrote: > Hello guys, I'm not 100% this is the correct list to ask this question, if > not please feel free to point me in the right direction. >=20 > I was wondering what could be the best recipe to have an HA cluster shari= ng > an external ZFS storage. >=20 > Let's say I have two servers running a bunch of Jails and, thus, I'd like > to use ZFS as the underlying storage layer and I have an external (iSCSI) > storage connected. >=20 > Would it be "easily possible" to have some (2?) iSCSI LUN exposed to both > servers and then activate the pool on one or the other server? >=20 > The idea would be to reactivate the filesystem from server A on server B = if > the server A fails. >=20 > Would it be "easier" to replicate everything and zfs send datas back and > forth? Clearly that would mean doubling datas and havin a scheduled repli= ca > with a possible delay in data replication, so I'd like to avoid this. >=20 > Any thoughts? I asked something similar a few years ago, the whole thread may be an inter= esting read: https://lists.freebsd.org/pipermail/freebsd-fs/2016-June/023456.html Just to say that I tried a lot of different things, and came up to the conclusion that for critical production usage all those things are fragile and with hidden dragons everywhere. I love FreeBSD (I use it exclusively everywhere, also at work), but there aren't any supported open-source solution for something like an "highly available ZFS cluster". On the commercial side there is RSF-1 and beasts like Pure storage ($$$) but ... I would really like to see CEPH ported to FreeBSD, that would be extremely useful Julien >=20 > Thanks a lot. >=20 > --=20 > *Andrea Brancatelli* --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --ppdebwkw7x3q32sg Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEnF27CBNtOraRNmgqCLYqJMpBHmkFAmXDdnoACgkQCLYqJMpB HmkWCRAAxpLKKakPNDKjgqsrCUtD7T3wN5dxzzQrL/2ipRiG1Azt7Q179aFYqf5L H4n57+CZ8vfdE9xO7qV3v8HRN3M3JhtfF53JdzRh+s4VsIqLr1mLwvtiP9hsG3aL 3vfl50VDJamvR2qIdrdnwqNrSl9BrCK7ea6qGzRiyUB2yiLxXiOteOp7CT6GyDlV 9qFI2HGNXYNAytm3fBkmR71xsi2UcFYXtkSjkrFhD9k47SAu3VdK5APqGX1oj7QE a0v2rburG4BvbqyNuiy3U7LKFH18xY5l8HAXudiTLVzbNfPMRhgSsGYcqodIgFfH C3uNRG0+u1NDlt6Vh2MY+xvWC7CzAhZX8PkMBhPrq2PJo9keisd4qjhLXq1vBFaV MUFLZAnKz/H2WAh0bQ9PQsHMZhHWizvVdRXSy3hdIGj6A6B/xedR9j+ISzPcsthU s+6iA3rqIR4vbHDUTkbv85H7/yzmY9l6u9czYeVZX/ve1VwkYAh/XYKhZNsSgJjr BBG6eTX7QoONgnLYQSHzWneu3ainihxMk4c7PrQLUUD9oGMwsd1oR/APec5fEKtK vcpLeipmN45Rhy8nj4EHnuzy6/lQPfEs1MenbGMYBU5zNfPRqUR6ULbiMEksYWyV r8hJADa16cV5De4WRpg8Emo5zNWp5tdDBliKhwqHhyGdxpKsC2w= =dOEi -----END PGP SIGNATURE----- --ppdebwkw7x3q32sg-- From nobody Wed Feb 7 13:36:18 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVLk8284hz59P9P for ; Wed, 7 Feb 2024 13:36:32 +0000 (UTC) (envelope-from andrea@brancatelli.it) Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVLk75Dclz43sr for ; Wed, 7 Feb 2024 13:36:31 +0000 (UTC) (envelope-from andrea@brancatelli.it) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-5116588189aso1080715e87.1 for ; Wed, 07 Feb 2024 05:36:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=brancatelli-it.20230601.gappssmtp.com; s=20230601; t=1707312990; x=1707917790; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=2ePsxSa/pYAJsd5P/NU0v+6bps8wWGq9Cw+0VNFFOb8=; b=JB2M4T2MOhsPNr5HsDFXyRrOymApWRiCdQDqu9sXST56GsYG6oUZttz43kncG5UnsL Vi0ZbqkIvltH7+zRBFRg+3J6GPD6jvcDwLVv+RfZOK/UAmqaXcsA1c03QyrOauGFPNqI Ii9XKCKKD+oWbtFaZe8d57SiKTkQsQt1fpDVWzu+05AX3PXjPnjOARAiZ9dN5QEFC3LJ 6RWmeknCTIZqKxH6nO1/X6QM8ecfc9gbg0e1QXl2tIBOlBMz9HjRkSeizIZFbr8U6xAX CljnewO2HfM3LbxMyeRUoJjZGsi8kSpeO/DXHZTfHPwK+PJKXvAIAlVobDfP1XBwhmE4 CNFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707312990; x=1707917790; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=2ePsxSa/pYAJsd5P/NU0v+6bps8wWGq9Cw+0VNFFOb8=; b=dF3GPKUknpQdOwpMPOdD5y/N9VZ9vSX/rM8yVVaz7+Wxqjf5ChDJDf+ilGB5AD4Qvm OPR07M7VMjKv8IXASe3mjiLEjvyNh/JJju6AfJjEZ5JerkU2UgVTp+Hi7yA21duj+weq oeAlvziJ29HoAg+htDS48Lwq7knWU8bVNyeogNI6nydTJ+nn3Axb0te6EVGm0zntMCWu YyKtFFGnqRQNgpDq6pyViJ9AigdHX93391G21NFsBbARPFRbsRakddVrVPUHXYKmUbJh WUj9EoU/iMbZ0k93nOk16rdl+zJb1QLaxsCztS3YD23B2OXcDmLa3VzASmfoSSR4Jh85 onwg== X-Gm-Message-State: AOJu0YxfxYEp8Lben6foyN9cIoXyzDN3ra72PhQZ938xzBNGyTeyZE1J fDwlr2aZjSHZPaxT9+iBL3YLKvAfy/89UfneKJB+LaNk1xH67XRn/jmZRCH/5/9yjw5zinYH87/ fq1X+FNT9HhWvD3hjjNZX/ouulaatN/mwDAB6YtdCv+zUpfcU X-Google-Smtp-Source: AGHT+IGv7Q5VPBfJYoMnqISjIwK0mOY9qCDRg3OxTM+voHsh2uBrOoKsp6BK51/j40bOlxouLAeuy0HdRZZnSmiGG9Q= X-Received: by 2002:ac2:593a:0:b0:511:4edb:5acb with SMTP id v26-20020ac2593a000000b005114edb5acbmr3893791lfi.64.1707312989760; Wed, 07 Feb 2024 05:36:29 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Andrea Brancatelli Date: Wed, 7 Feb 2024 14:36:18 +0100 Message-ID: Subject: Re: ZFS on a shared iSCSI To: Ben RUBSON Cc: fs@freebsd.org Content-Type: multipart/alternative; boundary="0000000000004681040610cac869" X-Rspamd-Queue-Id: 4TVLk75Dclz43sr X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] --0000000000004681040610cac869 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 7, 2024 at 12:25=E2=80=AFPM Ben RUBSON wro= te: > > On 7 Feb 2024, at 11:55, Andrea Brancatelli > wrote: > > > > Would it be "easily possible" to have some (2?) iSCSI LUN exposed to > both servers and then activate the pool on one or the other server? > > Hi Andrea, > > Yes, I have such configurations : > 2 servers with same JBODs configuration. > When zpool is mounted on server A, mirroring is done on both local disks > and iSCSI disks from server B. > When zpool is mounted on server B, mirroring is done on both local disks > and iSCSI disks from server A. > Works flawlessly. > And you can have one of the 2 servers offline for some time, when it will > be back online, sync will restart. > What you can't have is zpool online on both servers at the same time. > > So you can do the same with iSCSI disks only. > Disks will have to be mounted on one server at a time only. > > You'll have to have a rather low network / iSCSI latency, or it will fail > / zpool will be unusable. > If you have high latency, you'll have to go for ZFS send/receive. > That sounds like an interesting setup, although we have an external iSCSI "SAN" so the iSCSI approach would be totally offloaded, without any local storage. Imagine Server A having a certain LUN on the iSCSI device and Server B having another one. If a server fails, the idea would be to activate the pool "on the fly" on the other one. Otherwise I totally take iSCSI out of the picture (less is better), have local disks and zfs-snapshot stuff back and forth. --=20 *Andrea Brancatelli* --0000000000004681040610cac869 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Wed, Feb 7, 2024 at 12:25=E2=80=AFPM B= en RUBSON <ben.r= ubson@gmx.com> wrote:
> On 7 Feb 2024, = at 11:55, Andrea Brancatelli <andrea@brancatelli.it> wrote:
>
> Would it be "easily possible" to have some (2?) iSCSI LUN ex= posed to both servers and then activate the pool on one or the other server= ?

Hi Andrea,

Yes, I have such configurations :
2 servers with same JBODs configuration.
When zpool is mounted on server A, mirroring is done on both local disks an= d iSCSI disks from server B.
When zpool is mounted on server B, mirroring is done on both local disks an= d iSCSI disks from server A.
Works flawlessly.
And you can have one of the 2 servers offline for some time, when it will b= e back online, sync will restart.
What you can't have is zpool online on both servers at the same time.
So you can do the same with iSCSI disks only.
Disks will have to be mounted on one server at a time only.

You'll have to have a rather low network / iSCSI latency, or it will fa= il / zpool will be unusable.
If you have high latency, you'll have to go for ZFS send/receive.

That sounds like an interesting setup, altho= ugh=C2=A0we have an external iSCSI "SAN" so the iSCSI approach wo= uld be totally offloaded, without any local storage.

Imagine Server A having a certain LUN on the iSCSI device and Server B h= aving another one.

If a server fails, the idea wou= ld be to activate the=C2=A0pool "on the fly" on the other one.

Otherwise I totally take iSCSI out of the picture (l= ess is better), have local disks and zfs-snapshot stuff back and forth.


=C2=A0


--
Andrea Brancatelli

--0000000000004681040610cac869-- From nobody Wed Feb 7 13:52:40 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVM4p4kq6z59QBd for ; Wed, 7 Feb 2024 13:52:42 +0000 (UTC) (envelope-from SRS0=uefI=JQ=perdition.city=julien@bebif.be) Received: from orval.bbpf.belspo.be (orval.bbpf.belspo.be [193.191.208.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4TVM4p2Kw8z45mR for ; Wed, 7 Feb 2024 13:52:42 +0000 (UTC) (envelope-from SRS0=uefI=JQ=perdition.city=julien@bebif.be) Authentication-Results: mx1.freebsd.org; none Received: from belspo (unknown [10.209.1.137]) by orval.bbpf.belspo.be (Postfix) with ESMTPS id 696F01D9DF; Wed, 7 Feb 2024 14:52:40 +0100 (CET) Date: Wed, 7 Feb 2024 14:52:40 +0100 From: Julien Cigar To: Andrea Brancatelli Cc: Ben RUBSON , fs@freebsd.org Subject: Re: Re: ZFS on a shared iSCSI Message-ID: References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="3h47p2q2wl5agv2k" Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4TVM4p2Kw8z45mR X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:2611, ipnet:193.191.192.0/19, country:BE] --3h47p2q2wl5agv2k Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 07, 2024 at 02:36:18PM +0100, Andrea Brancatelli wrote: > On Wed, Feb 7, 2024 at 12:25=E2=80=AFPM Ben RUBSON w= rote: >=20 > > > On 7 Feb 2024, at 11:55, Andrea Brancatelli > > wrote: > > > > > > Would it be "easily possible" to have some (2?) iSCSI LUN exposed to > > both servers and then activate the pool on one or the other server? > > > > Hi Andrea, > > > > Yes, I have such configurations : > > 2 servers with same JBODs configuration. > > When zpool is mounted on server A, mirroring is done on both local disks > > and iSCSI disks from server B. > > When zpool is mounted on server B, mirroring is done on both local disks > > and iSCSI disks from server A. > > Works flawlessly. > > And you can have one of the 2 servers offline for some time, when it wi= ll > > be back online, sync will restart. > > What you can't have is zpool online on both servers at the same time. > > > > So you can do the same with iSCSI disks only. > > Disks will have to be mounted on one server at a time only. > > > > You'll have to have a rather low network / iSCSI latency, or it will fa= il > > / zpool will be unusable. > > If you have high latency, you'll have to go for ZFS send/receive. > > >=20 > That sounds like an interesting setup, although we have an external iSCSI > "SAN" so the iSCSI approach would be totally offloaded, without any local > storage. >=20 > Imagine Server A having a certain LUN on the iSCSI device and Server B > having another one. >=20 > If a server fails, the idea would be to activate the pool "on the fly" on > the other one. >=20 > Otherwise I totally take iSCSI out of the picture (less is better), have > local disks and zfs-snapshot stuff back and forth. If you can afford to lose x minutes of data then this is the way. FYI we use zrepl here, with snapshots taken every 5 minutes (sent both locally to another bare metal server, an offsite to a remote Hetzner instance over a wireguard link) >=20 >=20 >=20 >=20 >=20 > --=20 > *Andrea Brancatelli* --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --3h47p2q2wl5agv2k Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEnF27CBNtOraRNmgqCLYqJMpBHmkFAmXDiygACgkQCLYqJMpB HmnWqA//c+1mrGmfxzd9TQ/RsClPGkfH/l/zSnjqOEn1gJEpUslcf6tTYGV69Ewn zHay7ZrDtWZA4xMru9pmZOBpEMMeN14b0i5Pt9WDHxKlC0RwiIuZpdeFvIDbGUQN 5O1EJLUz5ypp7NAaXoJR/bCsMv/cgqgxTnJy18bVTLcp5MmNkdmtAZHM9EVHhLpH HQxYTAQFyozMRkvB8uBkCWcAY3vKv/kkS/ACzXyfX20lbK3rhU2chvQnMob4xz6o Chjp9rClXvLU5LLOT4Sl2qNuRaTaJBqJuCu9Iqo10HBxUraqJFPgZ1lB3KWRb68P R7rykr5BO+9DdRG4VW03u4zsv7MaWilNvGIdii9qXi1RDuxsrdzva9SU0JWTDg9Q noZEwwpGiBGOSlqhkpO1L75Q2jJl45agcyGtRMbOkSLlIvh/3qjNHQGkBQnm5d/N k3fjtgW+A6DgqpkD4ZVJvfR3uEvS+9vH3q9qviIyt1OT2Hg8yFRIuQr6E7YHTiUQ kT5YrIBWXHZmhw5FhWzgvLC1Of1JG6Wbhrg55ALYFQXXfULHcYNe8TsmMtPSEFuw 090T7cs4Pq99SOABgFwAaAFATbo52O1mT2fKWEB9ktTB2jxiT/mJ3hK/rcZx4tvP tr3b0l3YkPHBg67NP9YiZwbaADIeWIMxg3ZKmPvy3Cm6Dh7k2jA= =M148 -----END PGP SIGNATURE----- --3h47p2q2wl5agv2k-- From nobody Wed Feb 7 14:12:21 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVMWn5hpLz59S4K for ; Wed, 7 Feb 2024 14:12:37 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVMWm4Z9Zz47X1 for ; Wed, 7 Feb 2024 14:12:36 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of shuriku@shurik.kiev.ua designates 193.239.74.7 as permitted sender) smtp.mailfrom=shuriku@shurik.kiev.ua Received: from [188.231.181.61] (helo=[10.2.1.104]) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.97.1 (FreeBSD)) (envelope-from ) id 1rXif5-000000008sO-3VSq for freebsd-fs@freebsd.org; Wed, 07 Feb 2024 16:12:27 +0200 Message-ID: <761c8030-5de9-4114-9313-22ec860fb257@shurik.kiev.ua> Date: Wed, 7 Feb 2024 16:12:21 +0200 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ZFS on a shared iSCSI Content-Language: uk-UA, en-US To: freebsd-fs@freebsd.org References: From: Oleksandr Kryvulia In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ACL-Warn: SPF failed. 188.231.181.61 is not allowed to send mail from shurik.kiev.ua. X-Spamd-Bar: - X-Spamd-Result: default: False [-1.74 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.55)[0.549]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[shurik.kiev.ua]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4TVMWm4Z9Zz47X1 07.02.24 14:24, Julien Cigar: > On Wed, Feb 07, 2024 at 11:55:54AM +0100, Andrea Brancatelli wrote: >> Hello guys, I'm not 100% this is the correct list to ask this question, if >> not please feel free to point me in the right direction. >> >> I was wondering what could be the best recipe to have an HA cluster sharing >> an external ZFS storage. >> >> Let's say I have two servers running a bunch of Jails and, thus, I'd like >> to use ZFS as the underlying storage layer and I have an external (iSCSI) >> storage connected. >> >> Would it be "easily possible" to have some (2?) iSCSI LUN exposed to both >> servers and then activate the pool on one or the other server? >> >> The idea would be to reactivate the filesystem from server A on server B if >> the server A fails. >> >> Would it be "easier" to replicate everything and zfs send datas back and >> forth? Clearly that would mean doubling datas and havin a scheduled replica >> with a possible delay in data replication, so I'd like to avoid this. >> >> Any thoughts? > I asked something similar a few years ago, the whole thread may be an interesting > read: https://lists.freebsd.org/pipermail/freebsd-fs/2016-June/023456.html > > Just to say that I tried a lot of different things, and came up to the > conclusion that for critical production usage all those things are > fragile and with hidden dragons everywhere. > > I love FreeBSD (I use it exclusively everywhere, also at work), but > there aren't any supported open-source solution for something like an > "highly available ZFS cluster". On the commercial side there is RSF-1 > and beasts like Pure storage ($$$) but ... > > I would really like to see CEPH ported to FreeBSD, that would be > extremely useful What about www/minio? Does anyone have any experience of production use? We have a lot of bhyve vm's backened by local zfs storage, but I am interesting in some kind of hyperconverged solution. From nobody Wed Feb 7 14:20:59 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVMjZ0bqqz59StL for ; Wed, 7 Feb 2024 14:21:06 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from fc.opsec.eu (fc.opsec.eu [IPv6:2001:14f8:200:4::4]) by mx1.freebsd.org (Postfix) with ESMTP id 4TVMjY3gBqz49Cf for ; Wed, 7 Feb 2024 14:21:05 +0000 (UTC) (envelope-from pi@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from pi (uid 104) (envelope-from pi@freebsd.org) id 5cf7d by fc.opsec.eu (DragonFly Mail Agent v0.13+ on fc.opsec.eu); Wed, 07 Feb 2024 15:20:59 +0100 Date: Wed, 7 Feb 2024 15:20:59 +0100 From: Kurt Jaeger To: Julien Cigar Cc: Andrea Brancatelli , fs@freebsd.org Subject: Re: ZFS on a shared iSCSI Message-ID: References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4TVMjY3gBqz49Cf X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:12502, ipnet:2001:14f8::/32, country:DE] Hi! > I would really like to see CEPH ported to FreeBSD, that would be > extremely useful net/ceph14 was removed at 2023-10-31 -- current upstream has 19.0. I'm not sure what type of blockers are with porting it. -- pi@FreeBSD.org +49 171 3101372 Now what ? From nobody Wed Feb 7 14:24:32 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVMnY3jBjz59T6Q for ; Wed, 7 Feb 2024 14:24:33 +0000 (UTC) (envelope-from SRS0=uefI=JQ=perdition.city=julien@bebif.be) Received: from orval.bbpf.belspo.be (orval.bbpf.belspo.be [193.191.208.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4TVMnY2yRCz4C8m for ; Wed, 7 Feb 2024 14:24:33 +0000 (UTC) (envelope-from SRS0=uefI=JQ=perdition.city=julien@bebif.be) Authentication-Results: mx1.freebsd.org; none Received: from belspo (unknown [10.209.1.137]) by orval.bbpf.belspo.be (Postfix) with ESMTPS id F1A4C1DA6C; Wed, 7 Feb 2024 15:24:32 +0100 (CET) Date: Wed, 7 Feb 2024 15:24:32 +0100 From: Julien Cigar To: Oleksandr Kryvulia Cc: freebsd-fs@freebsd.org Subject: Re: Re: ZFS on a shared iSCSI Message-ID: References: <761c8030-5de9-4114-9313-22ec860fb257@shurik.kiev.ua> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="75g6gu3zemzf4ret" Content-Disposition: inline In-Reply-To: <761c8030-5de9-4114-9313-22ec860fb257@shurik.kiev.ua> X-Rspamd-Queue-Id: 4TVMnY2yRCz4C8m X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:2611, ipnet:193.191.192.0/19, country:BE] --75g6gu3zemzf4ret Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 07, 2024 at 04:12:21PM +0200, Oleksandr Kryvulia wrote: > 07.02.24 14:24, Julien Cigar: > > On Wed, Feb 07, 2024 at 11:55:54AM +0100, Andrea Brancatelli wrote: > > > Hello guys, I'm not 100% this is the correct list to ask this questio= n, if > > > not please feel free to point me in the right direction. > > >=20 > > > I was wondering what could be the best recipe to have an HA cluster s= haring > > > an external ZFS storage. > > >=20 > > > Let's say I have two servers running a bunch of Jails and, thus, I'd = like > > > to use ZFS as the underlying storage layer and I have an external (iS= CSI) > > > storage connected. > > >=20 > > > Would it be "easily possible" to have some (2?) iSCSI LUN exposed to = both > > > servers and then activate the pool on one or the other server? > > >=20 > > > The idea would be to reactivate the filesystem from server A on serve= r B if > > > the server A fails. > > >=20 > > > Would it be "easier" to replicate everything and zfs send datas back = and > > > forth? Clearly that would mean doubling datas and havin a scheduled r= eplica > > > with a possible delay in data replication, so I'd like to avoid this. > > >=20 > > > Any thoughts? > > I asked something similar a few years ago, the whole thread may be an i= nteresting > > read: https://lists.freebsd.org/pipermail/freebsd-fs/2016-June/023456.h= tml > >=20 > > Just to say that I tried a lot of different things, and came up to the > > conclusion that for critical production usage all those things are > > fragile and with hidden dragons everywhere. > >=20 > > I love FreeBSD (I use it exclusively everywhere, also at work), but > > there aren't any supported open-source solution for something like an > > "highly available ZFS cluster". On the commercial side there is RSF-1 > > and beasts like Pure storage ($$$) but ... > >=20 > > I would really like to see CEPH ported to FreeBSD, that would be > > extremely useful >=20 > What about www/minio? Does anyone have any experience of production use? = We > have a lot of bhyve vm's backened by local zfs storage, but I am interest= ing > in some kind of hyperconverged solution. I'm not sure Minio offers block storage..? >=20 --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --75g6gu3zemzf4ret Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEnF27CBNtOraRNmgqCLYqJMpBHmkFAmXDkqAACgkQCLYqJMpB HmljMhAAjYVueY5uYKSpYJC45RAT9Vj9pLesXAxmrqaxbhoCCgCYz6I6q2FkqueM CxQJ6slC9BJrLuFfRo97HE1fauG9NriMF4AZGrar+Ud8s1sggLMk03zn3J13yPze bd5GyRXiKH11FgiVI9bedKzJo/gCmGp+M3Vqoc7dDbwx0kQt6OetBB04VRpgYCAo C1e0d7DhODIQmDSL/yR+lEvKFPbe0wFIAS8MBnQhIhMmK4KCvftE01tyP9Q0CRed vzviTfH4gti72ytyzF1/w1Uvow4V30P8wnA6+PtybLhELpqEkmnH/vDZkaIM81J9 kptzf/QnHBt334o+VOaY+sJlc6x5hlyD78h+sbAKBGVy0tHiLnOFf75ghw/6Kvt1 151TJ2S4e7Yn5xInrcEvoYCtj77otC/z9+kmT5P3UKz//7otSigULedEbzQvktSO 7GcIE8iZRmr1O7QOj3lCqx/4AhoGO63SHCBaY2QczedICcrmI5VVbKm/w7gTQfd/ iN8yI2wOtpjGuwh6ho11VmBYSwPsjVQ4WGuyXqQBaKqqyI8gppWh+dX46gCTYg0X MJGc2Jflof+g9bhgry5DZ4z10iIxDC5abt95nuEu/NQe0my3IePF9lvl+4aPSQub Lb1W83IBIDw4sRlYshgJb59R7LmteZtRn6p+QbJ591+w07MGxew= =mX+1 -----END PGP SIGNATURE----- --75g6gu3zemzf4ret-- From nobody Wed Feb 7 15:19:44 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVP1P3Y52z59YCq for ; Wed, 7 Feb 2024 15:19:53 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVP1P1ghhz4K7n for ; Wed, 7 Feb 2024 15:19:53 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Authentication-Results: mx1.freebsd.org; none Received: from [188.231.181.61] (helo=[10.2.1.104]) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.97.1 (FreeBSD)) (envelope-from ) id 1rXjiH-00000000K7O-2Ihr; Wed, 07 Feb 2024 17:19:49 +0200 Message-ID: Date: Wed, 7 Feb 2024 17:19:44 +0200 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ZFS on a shared iSCSI Content-Language: uk-UA To: Julien Cigar Cc: freebsd-fs@freebsd.org References: <761c8030-5de9-4114-9313-22ec860fb257@shurik.kiev.ua> From: Oleksandr Kryvulia In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ACL-Warn: SPF failed. 188.231.181.61 is not allowed to send mail from shurik.kiev.ua. X-Rspamd-Queue-Id: 4TVP1P1ghhz4K7n X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA] 07.02.24 16:24, Julien Cigar: > On Wed, Feb 07, 2024 at 04:12:21PM +0200, Oleksandr Kryvulia wrote: >> 07.02.24 14:24, Julien Cigar: >>> On Wed, Feb 07, 2024 at 11:55:54AM +0100, Andrea Brancatelli wrote: >>>> Hello guys, I'm not 100% this is the correct list to ask this question, if >>>> not please feel free to point me in the right direction. >>>> >>>> I was wondering what could be the best recipe to have an HA cluster sharing >>>> an external ZFS storage. >>>> >>>> Let's say I have two servers running a bunch of Jails and, thus, I'd like >>>> to use ZFS as the underlying storage layer and I have an external (iSCSI) >>>> storage connected. >>>> >>>> Would it be "easily possible" to have some (2?) iSCSI LUN exposed to both >>>> servers and then activate the pool on one or the other server? >>>> >>>> The idea would be to reactivate the filesystem from server A on server B if >>>> the server A fails. >>>> >>>> Would it be "easier" to replicate everything and zfs send datas back and >>>> forth? Clearly that would mean doubling datas and havin a scheduled replica >>>> with a possible delay in data replication, so I'd like to avoid this. >>>> >>>> Any thoughts? >>> I asked something similar a few years ago, the whole thread may be an interesting >>> read: https://lists.freebsd.org/pipermail/freebsd-fs/2016-June/023456.html >>> >>> Just to say that I tried a lot of different things, and came up to the >>> conclusion that for critical production usage all those things are >>> fragile and with hidden dragons everywhere. >>> >>> I love FreeBSD (I use it exclusively everywhere, also at work), but >>> there aren't any supported open-source solution for something like an >>> "highly available ZFS cluster". On the commercial side there is RSF-1 >>> and beasts like Pure storage ($$$) but ... >>> >>> I would really like to see CEPH ported to FreeBSD, that would be >>> extremely useful >> What about www/minio? Does anyone have any experience of production use? We >> have a lot of bhyve vm's backened by local zfs storage, but I am interesting >> in some kind of hyperconverged solution. > I'm not sure Minio offers block storage..? No, but maybe md(4) can be used if needed. For bhyve vm's in my case we can move from zvol to img files. From nobody Wed Feb 7 20:35:23 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVX1l1WSrz5B2Z7 for ; Wed, 7 Feb 2024 20:35:39 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVX1j350Sz49NV for ; Wed, 7 Feb 2024 20:35:37 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nomadlogic.org header.s=04242021 header.b=U8LLqdVp; dmarc=pass (policy=quarantine) header.from=nomadlogic.org; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 66.165.241.226 as permitted sender) smtp.mailfrom=pete@nomadlogic.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1707338120; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kFNZQvyfImG0Tljd8CXfcLkPStlmfCo4ZhpzRMr0HE8=; b=U8LLqdVp05sqPrriOTwwHleKV2gs8TsP46L4k77Db5wNeBoDljwq2bNHyzsCr503NvG8uM Fq9x3wosOKcg8MHUJtQE+j3Mf+N22wgCd+EOWno8u9RxVxUqT/Gg/Ou0D6lO+O+1r0Ugyy DxjmxtdFCz9rm4rb495rO2H1zAULEK4= Received: from [192.168.1.160] ( [47.150.83.63]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 34c5fc18 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 7 Feb 2024 20:35:19 +0000 (UTC) Message-ID: <61dbd87f-2be5-4515-8a93-8656b114cd8e@nomadlogic.org> Date: Wed, 7 Feb 2024 12:35:23 -0800 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ZFS on a shared iSCSI Content-Language: en-US To: freebsd-fs@freebsd.org References: From: Pete Wright In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.98 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.99)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[nomadlogic.org,quarantine]; R_DKIM_ALLOW(-0.20)[nomadlogic.org:s=04242021]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[nomadlogic.org:+] X-Rspamd-Queue-Id: 4TVX1j350Sz49NV On 2/7/24 02:55, Andrea Brancatelli wrote: > Hello guys, I'm not 100% this is the correct list to ask this > question, if not please feel free to point me in the right direction. > > I was wondering what could be the best recipe to have an HA cluster > sharing an external ZFS storage. > > Let's say I have two servers running a bunch of Jails and, thus, I'd > like to use ZFS as the underlying storage layer and I have an external > (iSCSI) storage connected. > > Would it be "easily possible" to have some (2?) iSCSI LUN exposed to > both servers and then activate the pool on one or the other server? > > The idea would be to reactivate the filesystem from server A on server > B if the server A fails. > > Would it be "easier" to replicate everything and zfs send datas back > and forth? Clearly that would mean doubling datas and havin a > scheduled replica with a possible delay in data replication, so I'd > like to avoid this. > You could probably roll your own solution using corosync and pacemaker, possibly in addition to using HAST to replicate blocks between your LUNs.  i would avoid trying to do ZFS replication in this scenario. the tl;dr could look like: - HAST replicates blocks between iSCSI LUNs (assuming your vendor doesn't already support this on the target side, many of the enterprise vendors should provide this for you IMHO). - corosync/pacemaker are used to detect health of each of your freebsd systems.  if a  heartbeat fails between nodes it can trigger a failover event automatically. - the failover event would mount the LUN on the healthy box and do other housekeeping (failing over IPs maybe?, restarting jails?) i've actually build a system using corosync to do failover in AWS, and one of the nice things with it is when a failover event is triggered you can run arbitrary scripts.  so in my use case i was able to interact with the AWS EC2 API via some scripts to migrate network devices from one instance to another.  it seems pretty reliable, and handles some critical infrastructure for us. but to get this all right is pretty complicated...but so is distributed computing in general and i'd be suspicious of any vendor who says they can make this simple :) regardless of your approach you'd need to do a lot of testing and monitoring for critical production use.  it all comes down to what amount of resources you want to put into this. -pete -- Pete Wright pete@nomadlogic.org From nobody Thu Feb 8 03:32:05 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVjGH3rWxz58l6s for ; Thu, 8 Feb 2024 03:32:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVjGH0l7Yz4r9v for ; Thu, 8 Feb 2024 03:32:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707363127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qh3cmnFUrtr4ow/oVpcvkCqkpgaePdAYs8Aa3D14CQw=; b=Bd9/8J9CXrtJ46DeCkXIqCG26Vt4txJfs6XaQg8uFGdhUQYoeCXNlMDcJRoYqw7zwpOVTd ucV+G+6UcF9MDjEO4YsdrWVEphPGQRSbiiTIdcMSp9YvSHvHQX612FgwDbxe39AmdvWrBZ qbiqB5hu1YSL0FsddALBFZ1twIBznvjg+qM/CCR3taIscRu9DB8RD4ZHguFVmi8BsOxTzU 4i8iY+2XXzxqjEklRO7Xt4Dt0jQfeSed7QpYy4LO83A4hJXkJWs61sPsptKbOeCj0GEYRt DHr26uVcgfxYMPY+vYFzCDh4Nfksqp5Lp4hF3wB+ywpbxkbAtI6+7cJ5qSCmSg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707363127; a=rsa-sha256; cv=none; b=plmfCDKp9Bu8QkV+yjvpe1l2k7/ud/Je5Y19htKhY543PqWknB+Q5hKWutm8VzTAnAOJw5 Avs5dNZLk/lANV0YaycPctsH9YFRDllYlXvOqSISSlCkzAxKeB7FyfPjq/brOcnMAdW33U cPUmNbPzx62hstX0h1rXK795Ct3O56D/zBc5/rboPwuDq5779HIXAM8JQh2VEiNuS5te/8 Tf+QigwPeCNke2G1kuUtQ9Fefs4oY5fcmQNFT/YniaRv5mOJfaUoilihgFHLADn5GdvpAq 0tnwhnaNqgxwEvolLq3QhX9fu1Y9Z1L9u5BFfT1enEvQGXLDK3BNv52KZQOa/g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TVjGG6wW7zZLY for ; Thu, 8 Feb 2024 03:32:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4183W6g6007743 for ; Thu, 8 Feb 2024 03:32:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4183W6xW007742 for fs@FreeBSD.org; Thu, 8 Feb 2024 03:32:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Thu, 08 Feb 2024 03:32:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #44 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #42) * Fix Status - Backport to releng/13.3. Done locally. - Stall upon low memory When the vm_lowmem kernel event happens in the situation where the ARC prun= ing cannot evict the sufficient ZFS vnodes, a pagedaemon thread may wait for the ARC eviction indefinitely. This causes the partial system stall. Accelerate the ARC pruning in such the case. The fix has been tested locally. - Nullfs(5) node recycling This is the fix targeting at poudriere-bulk(8). Recycle the nullfs(5) vnodes not in use in the same way as the znodes, so t= hat the lower ZFS vnodes can be recycled as well. The implementation is partly shared with the accounting of the ZFS in-use znodes. This has made a drastic improvement on the ZFS behaviour, including: - The ARC dnode size has reduced greatly; it no longer grows monotonically during poudriere-bulk(8). - The ARC metadata and data now always have some evictable sizes. At lea= st, they no longer fall to zero. - There are always some number of the prunable ZFS vnodes. I believe this is how ZFS is supposed to work. The fix has been tested locally. - In-use counter overshoot and undershoot An overshoot on the nullfs(5) in-use node counter (introduced for the nullf= s(5) node recycling) has been found. This may cause a wraparound on the vnlru_free_vfsops() argument and hence make an out-of-control behaviour. The fix has been applied to nullfs(5) and ZFS. The local test is in progress. I will publish the updated git repo once the local test above completes. Hope there are no more blockers... --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Feb 9 13:03:26 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TWYv3235jz59gJN for ; Fri, 9 Feb 2024 13:03:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TWYv26fLSz42Hx for ; Fri, 9 Feb 2024 13:03:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707483806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r/cdxLl4APmWZ/FW8pcy2IcPFyCiJNA4A4BmfrX84XQ=; b=o8naSxUalk9BQ5Vb7n1vtL1ifTQsEapnxMUr2WAyrNLYt2X1l4xMUBww1I50JmIlxS1GCB E2YQOFncghtkqxpOo9S43DBwuHHQbOJDYpczMInnWfF2eb6EcMjBUAiOCLelPntmI6nQ8P Xu2sHtHwcO5Dyh7tPCktp2I047hJDYJb7NzFhhVdKikl5boGPEr1KnJbttjqWcKeUzHU5p +axOKByu430VsYRHzhBh8DBRavery45T11p98IBjIqHaP738bcaaxRKqqOsfCpvxpOHukh 2jepYP0k1rQCa5WpbgezXebM/MEqpKEp/kTg1Ef5U5PeA3jbvcRivWBngUK7Pg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707483806; a=rsa-sha256; cv=none; b=dvzxds/ZLyKjivwUIHr+4li1FIhx4UaHFq1qUKW1dFLFwTgjvmyPFe2DA0FgLoo3aAHKEu trHBL/2MiOlvHxfuYWMPY/Gyw4lH5sPI+i6dmjDfykG1QUiR/+ocie1OZ89wQpSlDydItI TlyM6NO9OYBF7Ttd+z95/ITxdoEdRGxyu6x7fTviHJZnxCvSY2VCTwuMx82Ot2gUB9/lN8 Ked3BiNhoRZZp6gPLWOMR5Lfibyslr2WOt2Bo6ZtLNKFbdf4b+mdUgGl034jOU7RJaayCT 7FjL3jCKkaR6Yvw9klsKmWZDvIbRjCm08RRp2rZUwi/HwIQjqFGsoMs5KH47MQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TWYv25lQ2zZ7d for ; Fri, 9 Feb 2024 13:03:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 419D3QPK016835 for ; Fri, 9 Feb 2024 13:03:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 419D3QcT016834 for fs@FreeBSD.org; Fri, 9 Feb 2024 13:03:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276918] "zpool status -t": output is inconsistent Date: Fri, 09 Feb 2024 13:03:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276918 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Feb 9 13:18:45 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TWZDj5XBxz59hKZ for ; Fri, 9 Feb 2024 13:18:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TWZDj47HMz43Ff for ; Fri, 9 Feb 2024 13:18:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707484725; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d/EAIdUdYJb0EvHXUoz0d1za5mAGRztOaH3weaxhmoo=; b=c+/CQ3Do0KtlLKInJ9AT0GpVs7O1xry39wmzOXes+DZVpoSF96/97V3e0l6csP4n8Asgjd BBViHpVEdieV6gpq+FMzI7i74WJdxhDAWt8HY5VkVebP8EzoIpw7b8z5A+ZCF8EVuZAWFi ttUMs+uKt8TssW4JT4iPvqLRKysDOe8TlCKOuOXWTeA0rkg/s8fp9acI/ugLd0OJ1s2ORg VqKWgrk6oeL++EQsiDcvKIc13d66B6ouB/x3xT7BpEL+bk61vvUnJNHOG6Gnt/aSTv85a6 GEJcj3+rLHIJDcyP8GD026ybHGHk2IulNBwrgpS3i3wRDNk2kKdqBAO2bcU27Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707484725; a=rsa-sha256; cv=none; b=w7TdpkFjc4k14s2dGkIxU4sduPBw1Hhx65uErY9vHeYPbqZvbWhumyjz9YiucAG8cTM11S zwlJrgAoj4Aa/ydlE79f9ObpBeaD9amwrKbDt58PxX0f5vIIiESos4QAVoYn7bXUTvNjdq BC3TW34ZDoOY3PPqpR78MAZlChO1NAIOt3SCqsKroqO4pP04XBur+3IujPiiMvfVChRSwf GE6kNruw18DqXfayyBI7hmawOvOY1o6CVtWVVpyujCaJwL5yVJ7cWwvx0G9V+J/DprDK7K Ilo0KypSlJBfAm/yyvDN73N/z8CBEHlX39Jz5JT/ksZ9UVIobp848XQdb+AgDA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TWZDj3BjkzZBs for ; Fri, 9 Feb 2024 13:18:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 419DIjNo084164 for ; Fri, 9 Feb 2024 13:18:45 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 419DIjHu084163 for fs@FreeBSD.org; Fri, 9 Feb 2024 13:18:45 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276918] "zpool status -t": output is inconsistent Date: Fri, 09 Feb 2024 13:18:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: vvd@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276918 Vladimir Druzenko changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |vvd@FreeBSD.org --- Comment #1 from Vladimir Druzenko --- 1st part: src/sys/contrib/openzfs/cmd/zpool/zpool_main.c (print_scan_scrub_resilver_status): > end =3D ps->pss_end_time; =E2=80=A6 > (void) printf(gettext("scrub repaired %s " > "in %s with %llu errors on %s"), processed_buf, > time_buf, (u_longlong_t)ps->pss_errors, > ctime(&end)); 2nd part: src/sys/contrib/openzfs/cmd/zpool/zpool_main.c (print_status_trim): > (void) localtime_r(&t, &zaction_ts); > (void) strftime(tbuf, sizeof (tbuf), "%c", &zaction_ts); =E2=80=A6 > (void) snprintf(zbuf, sizeof (zbuf), ", %s %s", > gettext("completed at"), tbuf); I think localtime_r + strftime do this. IMHO, it's better to ask OpenZFS upstream to change this behavior. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Feb 9 13:56:59 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TWb4r4tCwz59kyd for ; Fri, 9 Feb 2024 13:57:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TWb4r3rZjz45pd for ; Fri, 9 Feb 2024 13:57:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707487020; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lDrJbOkGxiocD2xW+wW/EWWXudR8gd+EKz9EvSq5E68=; b=Ok1UodyQ8KICjvJGUjgCIQWogsEOE+etLbMzU5hQJO+G7GlBhX1KDUURWMvRa2R6C58/d7 5tadkOGAsQ/2zPosFwry8QodQcFFFz03I14bB9M/pK9SSOEiVbrgx9Jmei4EijW72vBvGX HEjZtjtu8yqVvUnhepsl2rGEOSpjWrL/LOYi853I+SmxFYWYse/oNRM+LKat9s8NRL4ctO HsD6+wKlJjuc33TNd3rWC88Iss+OOpYpUd6DGVZFbkPrHbHYOS1Zhc/tIcfN8iGibSFvNo rYFbvBhHd/3J/aa6g812GoiZG/hhkF7j3O3yEnILVnbv4PSrajZuGIhLFrzbRA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707487020; a=rsa-sha256; cv=none; b=hpTf10Szri5skuqgc33gQtjBc/YICDfLWgV2VGYgs7skmZIi4wPvByMsxxTKWt1+G1VSoB PKMYpJoVLpaUrdeAH21hQUvYkpGXOFLxRwesj+DMpxSuIW+FWr7P2yROppqfR8mg4rQLw8 I1PZsGQUZFzOQSo4od/ILEtuwk+NoWiaPDGawP2e12Q1CQgbmvBdUuke2B6BVhX0sUegM9 2aJuCPd0GcGM16H00vMqaoKUXW3u+YZx5+ZlPo/+JzkhIkIVaONwwpUdyQlM0eNQEdgtgc sBkp1R9z7A+uVqdgME2N35kg4877svtyxad23A0C5KXnoHllWnmuq0SdwphdRg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TWb4r2ww4zbh1 for ; Fri, 9 Feb 2024 13:57:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 419Dv0ff073909 for ; Fri, 9 Feb 2024 13:57:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 419Dv0ej073906 for fs@FreeBSD.org; Fri, 9 Feb 2024 13:57:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 263885] fsck fails with UFS journaled soft-updates filesystem under ESXI 7.0u2a due to 13.0 increased kern.maxphys Date: Fri, 09 Feb 2024 13:56:59 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: chris@xaerolimit.net X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: mav@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D263885 Chris Brennan changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |chris@xaerolimit.net --- Comment #8 from Chris Brennan --- Created attachment 248282 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D248282&action= =3Dedit VMWare Screenshot of error My FreeBSD 13.1 installation (in VMWare) started experiencing this today. A= ll of my VMWare guests had been offline for 2 months due to a host-system disk issue, I've since replaced the faulty disk and rebuilt/repaired the storage pool. today was the first time I had started to spin these images up again,= my Debian image came back up fine, but FreeBSD dumped me into single-user mode. fsck during boot failed w/ this error and then when I ran rsck in single-us= er mode, it marked the system clean but fails w/ this error and if I reboot, it just recycles and does this again. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Fri Feb 9 14:54:37 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TWcML23Gnz59qG4 for ; Fri, 9 Feb 2024 14:54:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TWcML10m4z4DRP for ; Fri, 9 Feb 2024 14:54:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707490478; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t6/6aCrWWEQhoSX4V/W86zgz2OC2HpB7dVIBluiWV6g=; b=DbbvNDkFRSHjMIY6tGitt0o6iQJ04PRahPXg8yrVUxTmlfT5X+yP0M+7BZ8veAJ1i/33If qtbjemIFqF7+YcuZ/E71iyfv6vxoBvo8S1bJ9YxHMmavLxdwQgFPqHWZ9P2puMnguwfQLr NTOfDr7QTFFcfAWmCkLAX2cDiiHwUQ9k34Ul65lTJTw+Nbk50/kVtXlAZgDid+rOY+pW6e Z11TQTY4zaPloC5IQylvf5AeSV5WPpUT6BNT5UhWVyq/49tNMl9imPNiwnz/yNEmfEP22D Mh0qG0G+nli98w0qXnKQ4NOUtzYqsAhRqIi8TFH4hWmW8kgp9dNy4kkL8YJ+cw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707490478; a=rsa-sha256; cv=none; b=vXEXREHFB1twbo+YY6tLDWIfqTmg1CvZtYjcGgDe/ey/Ydf56qfqY94xrfsmJ8KbDBl1bC lO2ZjhJqo5dqnAxH9JcbkEQy0PjVgEoZCbBES8irUnvjOvTfE4ck5T+bjCQSQ+jnjMuRwM fRWU27pdsJWEc81CwzHWL/9XdVNwJjnCUdulvQow3EZEllWTEo9V2YFduAduL6B0gyqbj2 1SNhM8+DNk2qsBRhCvJfgQaIEWF7V4gF/SQl9KGISxPnFGFHvPdPVtxXKj+bMz69TJowmE gfM7AJybknjGUtMSdvwMuwuiKGnzoFhNQJlC9XZqNVsx+TGiRBYP/PxwHfgrCg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TWcML06GLzd9V for ; Fri, 9 Feb 2024 14:54:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 419EsbLH068891 for ; Fri, 9 Feb 2024 14:54:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 419Esbtd068890 for fs@FreeBSD.org; Fri, 9 Feb 2024 14:54:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 263885] fsck fails with UFS journaled soft-updates filesystem under ESXI 7.0u2a due to 13.0 increased kern.maxphys Date: Fri, 09 Feb 2024 14:54:37 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ronald@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: mav@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D263885 Ronald Klop changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ronald@FreeBSD.org --- Comment #9 from Ronald Klop --- (In reply to Christopher Brennan from comment #8) You don't really pose a question so I don't know if you want a reply. But the fix for your problem is in FreeBSD 13.2. A workaround for 13.1 and 13.0 is mentioned in comment #0. Set kern.maxphys=3D"131072" while booting. And put this setting in /boot/loader.conf to persist it for reboots. You can remove the setting from /boot/loader.conf after upgrading to 13.2. According to the commit mentioned in this PR kern.maxphys=3D"262144" might = also work. This is the number that 13.2 uses for pvscsi. I hope this helps. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Fri Feb 9 19:29:11 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TWkS81QYmz5BCH7 for ; Fri, 9 Feb 2024 19:29:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TWkS80Ctgz4mXp for ; Fri, 9 Feb 2024 19:29:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707506952; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oxvFqnsaFb7Z01Wv6mrCA+k/s9hGa24zTtNIcF2YxR0=; b=Eg2KawuX0IDaPiCP8e6Q51+kQXj+v3LZJvSpcTpb6UGyBTRTMK+X+6H7PMXWjf5ubzmnvT RG543mTPO/cqfJvrfBzkzAKftFgLqPsCEl9TCpZ9wM5nKPHXm+aTaQwo4NTaWWi0mgi0tM h/BFRYXDmYAgRIqyDJnDY0OPQjcIYGig/TLVrgvuvj1PhZvI/0xPrsAcS/LXYUz40E4FPM feGSKr+LdWj2hvkbVL1k9OKlE+YpwAGuZf31qh2jwZz0wjBVGezjksOBh13SLJbOyd7jT6 X2/SYvod58vjEvPCFS8AW7c0ibWE0/RHyPey3xl2WbQbFz2LSOSYxb1S98grFg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707506952; a=rsa-sha256; cv=none; b=yGHUnyIWSjprNHaygdW6oEFX4K3wFnxh2F5/0NmgNkYCo9JhjfvNnJ+EdhngllOQniliVp C1qmQ0+l0zZLJ+ag7QNhIgA64DW74Y2Iz1IN1OBkUU4yTIXZT/MESW5o1OL6oCn67qGNBc 3mFDiMK3UMklCeFF4aoqUcC8WsLWCEWUy6/t7UVjwUk7kODKZPPDdq3WexQjfyC4chE044 YQHopdjIzspx1vbWZs9yHbe6blrcXUujY+D56ZcCm+bWcbLUqlIXn8BMP4OL5sXYLjvoar L6n7qo2NQJHNsi88OoMz4GYoiCRO0T3d1xzL+szAcaPCJKSTuJ9nE5IDVNXwgA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TWkS76PRRzm12 for ; Fri, 9 Feb 2024 19:29:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 419JTBC7034361 for ; Fri, 9 Feb 2024 19:29:11 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 419JTBIL034360 for fs@FreeBSD.org; Fri, 9 Feb 2024 19:29:11 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276918] "zpool status -t": output is inconsistent Date: Fri, 09 Feb 2024 19:29:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: emaste@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276918 Ed Maste changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emaste@freebsd.org --- Comment #2 from Ed Maste --- > IMHO, it's better to ask OpenZFS upstream to change this behavior. Yes, that is the best course of action. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Feb 9 20:05:29 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TWlG24nfkz5BFbt for ; Fri, 9 Feb 2024 20:05:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TWlG23jM9z3xJS for ; Fri, 9 Feb 2024 20:05:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707509130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O/+YzfxoMLQ28HXT/6sHotz6YbEkD36iKpFIoBBY1oM=; b=A/QGUYn79rdtPqCHfdRN3fPH5dK0HRESabdNaDvN9kOXaWQ08GMaM726zRw5GSjzdzxxd8 3HArm003PpFRNFPA+9Dg2ZZniBHh9x1esobDvnEK++TfZ6aSL5Jv3LfU9QfL4Xg11P15EZ nt9fUdlDhxR6e9LvT8kbZw7JjOt0lPew9+KV1jCPh7DJVuzVAG94+rzrfgUZ2yfyqVC0G+ db6czrwPGrpJPazuWDBQNUB/c+1dDYCQrBPfXPHCDeLjYImkGHZAr0kj5ve4Sm1XEzI+K0 9C9ykXf5FOJr7y5Jleu31DHVeCEJyRIfeTPfrXWEfGSIgx+RhEoXO2ZGm4c1mw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707509130; a=rsa-sha256; cv=none; b=OdL6QhCqLE7o3cJ0ohnuu+puhrV3vEOqN3OK2ergMtKDwCxX7ijgDtDH8gaQx2KtUsPb9C +6IOZKYJP9nFPs7JFIyoG1HZ8GBWKUog5OoOqfwXkLKXiE4QpsV3RPbTybTBTU3rmGJPhW AQqQgYQCBE8ORAdZHpf5QPWK4cZXbUiBoip4FGA5xQ4VMUzU1xY3jsdZqP0lvRDKb+xMvB YIY2MYrAgjBbtXQBue5kGwDgY+ogK8UiSIeA5KRMiDp6Ol2n/o8buypCz8Q/JeB8dnP1Tu CZdipKIsJUX0bbH8FHbqRkkaKr+0BNn+TSrosawp7tE2Ur6gTeDfTSEJXgry0g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TWlG22n1kzmjx for ; Fri, 9 Feb 2024 20:05:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 419K5UrB022937 for ; Fri, 9 Feb 2024 20:05:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 419K5UgE022935 for fs@FreeBSD.org; Fri, 9 Feb 2024 20:05:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276646] failmode=wait bad idea for zroot Date: Fri, 09 Feb 2024 20:05:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276646 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |markj@FreeBSD.org Status|New |Open --- Comment #3 from Mark Johnston --- This seems sensible as a bsdinstall enhancement. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Feb 10 15:08:31 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TXDcv1zGFz59qTQ for ; Sat, 10 Feb 2024 15:08:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TXDct6jZVz52t9 for ; Sat, 10 Feb 2024 15:08:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707577710; a=rsa-sha256; cv=none; b=BCnHj/vBEgs0BZHhAAG4x6dtop7LFk9AjnJjLcMEat8j7bqaDODSsj3ig9AJ2wcvQMOyOJ DpjFUdQO5d7JyPOnYtdSBSlBiguNegy0UfNinGdPwNRb+ioXsMIyz+/Px0S0OMt5lOWUiS GXXvWjTMxcNbSEXJny17gm0p4XIbelaOCRSylRODzjRqrFkbXtzHIojUrQmuAwhH8uQ8Wr nAodbdB6RzUdKwmBqCrk6tkOOB1Kvf6MYfxcSHjlg6KE8kGuTQ/JIRBM4kIJZfVQHv3uRl BtFC20g0zvYLZ3LKih2lSDZSNOgxXMNiYLyw9SS2Q15fumfLKriPTpybY5iImA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707577710; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EVFnmA5KBbLNP0F8032wWU+B6uO7RD9cwXgBoPTpXgU=; b=xdBh3DAjHp7bMU51UoL6qFmYS6Am1BHpv5cTRKJR2IZceETl3PdiXYPs7kvY8TgN7bKYiW 4PP7cCK4XiPBmw4GESgyhYwwHNVhCMUCkhtkXAoPy0em6MbR/LsER8YKiWunz1EDvvsN1Z IUh5E5mgvs6xHbOAjnuuFDiSImk1AtK7p3yy0PnGe4R/wJpy1ZHhowQY1m1fa+tUZycLCv +1kIqlHh3UvRDsvyH46Sl4xEbeec0uePR128aixohIOTLZqTjUzhPrnZYmQbZFfG6yPp7x bdp2+9BbpGHOuJRNPDIUNDIBPfcG1Dn2yDX6JJd41Pb1W76sVUhEzccPLDl+ww== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TXDct5nT1zLPq for ; Sat, 10 Feb 2024 15:08:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41AF8Umf084338 for ; Sat, 10 Feb 2024 15:08:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41AF8UsR084336 for fs@FreeBSD.org; Sat, 10 Feb 2024 15:08:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed Date: Sat, 10 Feb 2024 15:08:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: warlock@phouka.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341 --- Comment #10 from John Kennedy --- I just had this happen again, during system shutdown: 15.0-CURRENT #26 main-n268150-8ecb7494250. [via kgdb] ... Unread portion of the kernel message buffer: <118>Stopping devd. <118>Waiting for PIDS: 757. <118>Writing entropy file: . <118>Writing early boot entropy file: . <118>. <118>Terminated <118>Feb 10 04:55:50 bsd15 syslogd: exiting on signal 15 Waiting (max 60 seconds) for system process `vnlru' to stop... done Waiting (max 60 seconds) for system process `syncer' to stop...=20 Syncing disks, vnodes remaining... 0 0 0 0 0 0 0 0 done All buffers synced. Uptime: 9h56m17s GEOM_ELI: Device vtbd0p4.eli destroyed. GEOM_ELI: Detached vtbd0p4.eli on last close. panic: VERIFY3(rc->rc_count =3D=3D number) failed (29696 =3D=3D 0) cpuid =3D 0 time =3D 1707569759 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c737e= af0 vpanic() at vpanic+0x135/frame 0xfffffe00c737ec20 spl_panic() at spl_panic+0x3a/frame 0xfffffe00c737ec80 zfs_refcount_destroy_many() at zfs_refcount_destroy_many+0xd1/frame 0xfffffe00c737ecb0 arc_fini() at arc_fini+0x281/frame 0xfffffe00c737ece0 dmu_fini() at dmu_fini+0x9/frame 0xfffffe00c737ecf0 spa_fini() at spa_fini+0x36/frame 0xfffffe00c737ed30 zfs_kmod_fini() at zfs_kmod_fini+0x7a/frame 0xfffffe00c737ed60 zfs_shutdown() at zfs_shutdown+0x2b/frame 0xfffffe00c737ed70 kern_reboot() at kern_reboot+0x4f3/frame 0xfffffe00c737edb0 sys_reboot() at sys_reboot+0x3a9/frame 0xfffffe00c737ee00 amd64_syscall() at amd64_syscall+0x153/frame 0xfffffe00c737ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c737ef30 --- syscall (55, FreeBSD ELF64, reboot), rip =3D 0x28d86a, rsp =3D 0x820f08= bf8, rbp =3D 0x820f08ce0 --- KDB: enter: panic Uptime: 9h56m17s Dumping 3585 out of 16340 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..= 91% ... (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=3Dtextdump@entry=3D1) at /usr/src/sys/kern/kern_shutdown.c:403 #2 0xffffffff80b52da0 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:521 #3 0xffffffff80b532a2 in vpanic (fmt=3D0xffffffff8225ea82 "VERIFY3(rc->rc_= count =3D=3D number) failed (%llu =3D=3D %llu)\n", ap=3Dap@entry=3D0xfffffe00c737= ec60) at /usr/src/sys/kern/kern_shutdown.c:973 #4 0xffffffff81fbc92a in spl_panic (file=3D, func=3D, line=3D, fmt=3D) at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c:100 #5 0xffffffff820b2681 in zfs_refcount_destroy_many (rc=3D, number=3D) at /usr/src/sys/contrib/openzfs/module/zfs/refcount.c:98 #6 0xffffffff820b269c in zfs_refcount_destroy (rc=3D) at /usr/src/sys/contrib/openzfs/module/zfs/refcount.c:112 #7 0xffffffff8200d561 in arc_state_fini () at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:7421 #8 arc_fini () at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:7753 #9 0xffffffff8203b449 in dmu_fini () at /usr/src/sys/contrib/openzfs/module/zfs/dmu.c:2549 #10 0xffffffff820db0b6 in spa_fini () at /usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c:2549 #11 0xffffffff8219985a in zfs_kmod_fini () at /usr/src/sys/contrib/openzfs/module/zfs/zfs_ioctl.c:7933 #12 0xffffffff81fc32eb in zfs__fini () at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/kmod_core.c:284 #13 zfs_shutdown (arg=3D, howto=3D) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/kmod_core.c:297 #14 0xffffffff80b52d13 in kern_reboot (howto=3D16392) at /usr/src/sys/kern/kern_shutdown.c:517 #15 0xffffffff80b527c9 in sys_reboot (td=3D0xfffff80001a39000, uap=3D0xfffff80001a39400) at /usr/src/sys/kern/kern_shutdown.c:312 #16 0xffffffff81059473 in syscallenter (td=3D0xfffff80001a39000) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:186 #17 amd64_syscall (td=3D0xfffff80001a39000, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1192 #18 #19 0x000000000028d86a in ?? () Backtrace stopped: Cannot access memory at address 0x820f08bf8 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Feb 10 16:14:13 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TXG4j2yNxz59wSX for ; Sat, 10 Feb 2024 16:14:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TXG4j1tjBz589p for ; Sat, 10 Feb 2024 16:14:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707581653; a=rsa-sha256; cv=none; b=pbq5BrdfN8y8X7SGFyJ5WUD++2IfugydgT02VKyOM/qZCYCgo64s4UEjS0fxgv4SxdQPDU Q4sOE0K4Tyzi7QM3/l2dsiKktHl3eb8BqkdvN81qGb9WmbRrJjeoPMVUUPIxB/hu2C1JQ5 aRt1WlCpb6iOjiussJtR2QAFR8YlI76eLt4PnZsy3ZljTx4RTzTh64h4U/wbldv3N0V2wM 2q8p9R7meuDidBeMP/DYJNtzfPSVZH7w433Q1JJhOowBZPKNkSBPo38q66fNDl/QoM0Ezr QvU83pvFMApy7Up8+hr+80mY6TNBsaAdzi5aCxYJfrwKMhiUl+5yFHxw+qf3OA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707581653; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jhqZZDAmoDHKw/PxOUORgWuvoIV4xfV2gi0C3eSx4ek=; b=MOqF+nDgWoUhuY7XnNkQfg16WE7dKi53yCqSGQXmM7w5Qf3OqiYnRStYe0YwoAix/nHJJx JuYsN6p8wLaCxjasVMY45JTW7iHDJGL3Uh38RBz/84GCqZW8Kxiz6jn9B88jo1s/605oYU peFcRQcZRs9EtjQSd/I0FpTdilHIsZhwIBa1pjPVyAN/ejP+HrzA5gFy+HVe2GQSjJh9XM MUHc/e0XxDNJHzNcuFQXf+YmPrR9Sq5X7NBiTSf2jhXFvsQEOR7JZ002eVgQWdXTW2E76c BzBrh0RTBaIj9CRl2yWo+RGRtq8OZwDJx+Wth0o4Dexry5iViDz5aAuFf/p5YA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TXG4j0vKJzNMP for ; Sat, 10 Feb 2024 16:14:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41AGEDeg025226 for ; Sat, 10 Feb 2024 16:14:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41AGEDnZ025222 for fs@FreeBSD.org; Sat, 10 Feb 2024 16:14:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276918] "zpool status -t": output is inconsistent Date: Sat, 10 Feb 2024 16:14:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: che@bein.link X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276918 Maxim Filimonov changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |Overcome By Events --- Comment #3 from Maxim Filimonov --- Here's the issue with OpenZFS GitHub: https://github.com/openzfs/zfs/issues/15878 Here's the pull request: https://github.com/openzfs/zfs/pull/15879 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Feb 10 16:21:31 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TXGF81ntgz59wYp for ; Sat, 10 Feb 2024 16:21:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TXGF80pfXz58SL for ; Sat, 10 Feb 2024 16:21:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707582092; a=rsa-sha256; cv=none; b=R52UR1uY8cziPOT6mq3yIG2u7VjVu4sTofdVZBBfiJLnzkkV3l7f1s5Rrf2lvx2WEHijQN YxvTchRrx9hG5XZC/diP1FwUSq4mveePxtuHA0vFLcipfjBmHiGosahNWFAWQB/b2em6XI OIOI63bWIQuJcO75+ZEWHxDA4IEAvxLRBuaOM+h6MN4wfzuEIMnUCXPs0PPEivW85aHl79 NtRSToPxeNOersVtYS572WZkOgSfOpDPx2G3+JZ3P+tKAQGV2rlhYjDVzL9hGBpAwen4uV 3DRBBzxhu5kCm1cVLboF2Cg4zuVSLW4MuRiorxusU0C6Bo4kyTAlrKK3WIJ6fw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707582092; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zVCL5NuAW1kQX0TwWsyWxGdmNXvNqbRYCPpdO84P61M=; b=Ydj1wFnDzPqAR/W0zCb6JzXQULkC/3wpELBQH6i5Kp0ujR1YerSed83Nc6V6KvO2C4OKL2 58xLBRUMjT8iT+DwjcpmL+En3pl0RJrRPdhG5leV8c9F11wirBldMo3j+EG/7Y5w42TfAF +iRBWFiZnLEbMQVRCGiy8EHBf0ve+iiaZ1lyYSJJ9empAUzDzBz9ut8Q4KTkELPY/7Mq8d qFgiMVIqXS3yEd6bpiyXYpdD/XP8bNuYiR22wjA91OphX7et4zLhA+h68yx7nCVG6yOdVL a6/WEhpAKNIl9ZFtq27ohJIOgibeoqFYWBpw6izLHHd5fJAJ84I3C+20SurznQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TXGF76rgszNC7 for ; Sat, 10 Feb 2024 16:21:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41AGLVDh057414 for ; Sat, 10 Feb 2024 16:21:31 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41AGLVLj057413 for fs@FreeBSD.org; Sat, 10 Feb 2024 16:21:31 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed Date: Sat, 10 Feb 2024 16:21:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: Rincebrain@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341 Rich Ercolani changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Rincebrain@gmail.com --- Comment #11 from Rich Ercolani --- I wouldn't necessarily expect using openzfs-2.0 as a compat to help here, s= ince refcount.c is about ordinary reference counting, not the reflink/BRT/... feature, and none of the stacktraces in here seem to be pointing to that. If the crash started in rebooting from late December's CURRENT, then the question, I suppose, becomes what you were running before you booted into t= hat, as a data point for when the change that caused this breakage went in. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Feb 10 16:41:19 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TXGh00j6bz59y6w for ; Sat, 10 Feb 2024 16:41:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TXGgz3MHmz3yKY for ; Sat, 10 Feb 2024 16:41:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707583279; a=rsa-sha256; cv=none; b=TIGEco56v2EeOAI7/vPU+5iSX+zLRD7ORopSb7u+S1SAZOP2UHHysWrKQqMbN6U4guwC16 Cgs7zHjxqEvykg2nI3tD31zp/+aBQk7TIjPw+qNXUcFOEkoZRofK2Y++XwOffynWvnACnZ PAqQAfgLvIKzr3rTBHNqVTGC+svO/rePwrMKOn7zpN8YOnJLvXek6oy/TdTSyxjABOsIPI lG2vJa3F33a5WcdJa4cObxLnuWVddOCoEzDVlUUY9qC7Fd7TvGvg+CE4S71RQ7IfGgLRkp tIwrrE1VMt7CKNMnaOdthqkFhnoAI+K/58ZoHwOcUTXVxbCvHDwVG0HXD5b5yw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707583279; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lfraRnOgaXLSsPftqUlTkRE/Ke25B4EfqHP38scY8Ek=; b=wvAhmIN358oP5r0Er3ZyCSfKWJ2Yw8bTfObHwRJ4uJ/4Zz/EKDLMoX8RxINSeQbVf3R74j n01UmCoJRqb+hUu0xarTo3z0K2LfsqDiygARclMt7Kr//XbwH9mXCadd9J4AAT9b/Jr2Mv 9tO1kKX5gLYEtbFQtkML2aYuI4euF98h6LcT+mMW3AwhQ/7l9ZN8Guu5rGQw81h6BkQ4V1 QWc0xibHPu7WQ0wpdbScY/lIXbDfOY1PWAI7Zcu+Nxfl0YZuDpOpsf2+5sz0NMu7+ksFyL 6yH8jNBoV0XtuUncWgFDF7642epyThMavN8lt3OV95n0mnL6ZIT/V3Blyyj7Uw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TXGgz2RCkzNbL for ; Sat, 10 Feb 2024 16:41:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41AGfJJ7056925 for ; Sat, 10 Feb 2024 16:41:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41AGfJof056924 for fs@FreeBSD.org; Sat, 10 Feb 2024 16:41:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed Date: Sat, 10 Feb 2024 16:41:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341 --- Comment #12 from John F. Carr --- My first panic was on December 12. The kernel was based off c711af7727824da79d87f375f3d6829feec3799a with some local changes that should not affect this bug. An earlier kernel based off of 21fce617d1de231a30833cdd9819ef61277b08d8 did not crash the one time it was rebooted. In between the compiler was upgraded to llvm 17 and the following changes were made to sys/contrib/openzfs: 3494f7c019fc Notable upstream pull request merges: #15539 687e4d7f9 Extend import_progress kstat with a notes field #15544 c7b611926 Allow block clon= ing across encrypted datasets #15553 adcea23cb ZIO: Add overflow checks for li= near buffers #15593 5f2700eee zpool: flush output before sleeping #15609 3e4be= f52b Only provide execvpe(3) when needed #15610 735ba3a7b Use uint64_t instead = of u_int64_t #15612 bcd83ccd2 ZIL: Remove TX_CLONE_RANGE replay for ZVOLs #1= 5617 55b764e06 ZIL: Do not clone blocks from the future #15623 727497ccd module/icp/asm-arm/sha2: enable non-SIMD asm kernels on armv5/6 #15625 9743d0963 BRT: Limit brt_vdev_dump() to only one vdev #156= 29 f9765b182 zdb: Dump encrypted write and clone ZIL records #15634 2aa3a482a ZIL: Remove 128K into 2x68K LWB split optimization #15639 11656234b FreeBS= D: Ensure that zfs_getattr() initializes the va_rdev field=20 #15647 4836d293c zfs_refcount_remove: explictly ignore returns #15649 f0cb6482e setproctitle: fix ununitialised variable #15650 450f2d0b0 import: ignore return on hostid lookups 525fe93dc748 zfs: merge openzfs/zfs@a03ebd9be db0a4f2d7a65 openzfs: unbreak 32-bit builds. 2276e53940c2 zfs: merge openzfs/zfs@688514e47 c5405d1c8507 vn_copy_file_range(): provide ENOSYS fallback to vn_generic_copy_file_range() --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Feb 10 16:58:58 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TXH4L4dwTz5B0ck for ; Sat, 10 Feb 2024 16:58:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TXH4L3b5Bz425W for ; Sat, 10 Feb 2024 16:58:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707584338; a=rsa-sha256; cv=none; b=JxACsM6zSyHxDwG2uOoaKhJoLFf1TGIp8hcaehcfdvetJZ3K//yAIDwCn9PB48plB7+DnL +o3QFDYeIYnih50bzmxwgDBnEq8Vdux2WAHj+ly9pC9vNw4kBZ8O+qj4VbigGdaS+SWkXs 8u6WwvyNsMs6QkBnPkshvxJ2drXL1jZDvYLAfdmgNKjGjNvnWf1K+58Jwtos2vmuB/OwTW WSFa1cup81TRKfZKgu02Kx/BN9TqbBAWjw3ClQqMzRoWbshNfBav5tLAhCyXLhRp7IUVlT m1UPAV5I6194xXvKsQgSILtHvjHs0yf7MpTgnAbiI/nxAnf/NGHtZEjBjkQo6g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707584338; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TvuaJiIHpSVNSEIhaSy0Q4zm1cogR3Cx+FIIHqi/M68=; b=ZDDRrLnKskoX14Yu3M7aef3oHBPMf8hrOrWGqekoNshzJMpkDiXi0N0UxP0dHB5+nhKmll nTV5UIw0H+GHv5V0lqRN3G5lNtgi2ZoIyTm5mcEZ93tMktdVASU6pgzE44JJAIOU1Qzzyl SdxDIcG7+I3O1mwDl1KpOVLVzEVl1rAyBCKV4UiXKVkrPg4yMRLoLyENqKOJtK3JeWgjKb AcBNSoSXVIzF10+rqHVklkFaYlaNP0PRuRmQgGgi76RrQCdfWOU4i/nCAH7cQJxQyIiNw0 FVlNQTxqvNds201/WthTA26K6Bnn9CsONvNgUz2dmFYTyABwmG/5F7fp6dq1pA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TXH4L2XFszPQN for ; Sat, 10 Feb 2024 16:58:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41AGwwxq039680 for ; Sat, 10 Feb 2024 16:58:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41AGwwuR039679 for fs@FreeBSD.org; Sat, 10 Feb 2024 16:58:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed Date: Sat, 10 Feb 2024 16:58:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: Rincebrain@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341 --- Comment #13 from Rich Ercolani --- At least 21fce617d1de231a30833cdd9819ef61277b08d8 ... 3494f7c019fc6558a99f63b7f647373b89bcde92 is a small window. None of the changes in that window seem super obviously fraught or risky he= re, skimming them. Half wondering if 21fce617d1de231a30833cdd9819ef61277b08d8=20 built with LLVM 17 would break the same way and it's some weird optimization going awry, but I haven't remotely tried testing this enough to say that. Anything you can tell us about the datasets on the pool? I'm going to go try reproing this in a 15 VM, but if it doesn't trivially reproduce for me (tho= ugh if syzkaller is hitting it, maybe it will), then knowing things about the s= etup hitting it would be useful. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Feb 10 17:11:55 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TXHMJ0Rrdz5B1rQ for ; Sat, 10 Feb 2024 17:11:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TXHMH6QSJz4368 for ; Sat, 10 Feb 2024 17:11:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707585115; a=rsa-sha256; cv=none; b=MDdzh/AAJ5Ar/r3Fe8gaWII0Q68cdzXJZK/w7bfAWcEn+O1KDwNg3vDjJmwcDdMjYQR7ZW YNt11U0otuNHw6QhdqkRGnZUcEctSjLVWWhCvkLNVati0agfekhn28AC8hx1Lfeg6yT7Qh 9bVdNUKCTJL5wvaSyM6fP9xvV4gZVzYk0cKzhwRoxj9QZ/FpAfWAnUn+olCXFxYq7yAqc3 UiKhGOzWpxEsnBATEhjJyzfX2Y4nYOT5XThSxA1j+1RpjPGaYOn8VmowzVQqG5yZbjAnIC LT7+oThoIQ0hCxcPQ5pc1dPGDYqNHcoN/U1jWcDK/LzGUKLwSh3PV4eyZyL8cg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707585115; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V8kXVk4uDXxWcY3xMLFq2YJxiw2dvOmSzr9cHGg7y7Y=; b=f2lcDHV3ZH3qU2qf4Vf5WLWMeWOATxIPkeRnytWJLJMKkdnCctaxqJgs7G3Jdkjcj0I/ev X0KRt3qaJjQCT6gjQsx//DD9WaSP/g0QRZpHRvZVUIxydtQygqPfDR3r3LaBnF43qJn91R 3n65XA/4Fl0P9wHyu0B3OsLEV55zxBGUajr1ehOOnJ/62GnI9OSkO9FjkcvbE1jMTypY0g YdQCtePBOkBXDk1ZzUkt7wylHRQHuIs2Sv6adFQD5OA+AZS3Ca6VuA9Y+5ljkR9Xin7WTT hntE3MzU334dK41X0taSm6JBcVvyDXIZXLGzj20KzE/2mSUqdpwl1iWvyTntHw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TXHMH5TbwzPyR for ; Sat, 10 Feb 2024 17:11:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41AHBtUB016326 for ; Sat, 10 Feb 2024 17:11:55 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41AHBtga016320 for fs@FreeBSD.org; Sat, 10 Feb 2024 17:11:55 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed Date: Sat, 10 Feb 2024 17:11:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: allanjude@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341 Allan Jude changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |allanjude@FreeBSD.org --- Comment #14 from Allan Jude --- Based on Mark's analysis, this error is that not all of the memory used by = the ARC was freed properly during module unload. This would not seem to be putt= ing anyones data at risk. Why the refcount (bytes of memory used) is not getting to 0 is as yet unanswered. It seems to be different amounts of memory for different people, so it is n= ot likely some small static item like it might have originally appeared (initi= al report was about 256kb) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Feb 10 20:14:29 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TXMPx2R6sz5BHpg for ; Sat, 10 Feb 2024 20:14:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TXMPx0gH5z4SJ1 for ; Sat, 10 Feb 2024 20:14:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707596069; a=rsa-sha256; cv=none; b=jL1WmLnLiuq67qL2nruzOEnHpe5I8jhWzt2VHn8B6jtwg+QzR0Vd3HSnMgy8JOzM4vpDNT gHcNIo/QwgOxR3BxkpxkTHYgBzMjfAjgYm25maR9ZdO4pxvWXPCtVS741EZaJ9W71m3o18 q14tkJUWRrvSHQ9ZGJRmMI3HY67NTqyFBY8gBZiIo4tK2er1ha3c9K6ssixrXnARIptPc7 Iyu/ydHk2TBnSzQs0/1KpBLTHQaGUnQkxY2dV4b5nwR5XM0hUOAutC8Xwhz0xm3WSKTyQG PbfXakaFGrl+SygNIRQnxS80dzIAgo9NZgpNQLiWREOyHYD1mN6q54YGXi1TLQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707596069; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S2bJixb+t0/UVk6WHGE1rpFZt3qWhax8zrbGfN+0B0E=; b=kZzuDVP+Xae5Jm3WYc4PRgJexToA285Af8gf0XZJzIq7I3AAWbEfxQFfaeOcHV5na023UA rcr+LL5nD8Z2Ug2+X0AfOI61YfbLomOe2awRpqopdmR+mIg40Yns4hfgL9I1tgXDAeT9Ht Ygluazu1USBY+mHXrZMuiPP7ngUB2ZSTv4flyl85NFNhGKUGDXwu8t4EC1Sd+SrIOWtKtY VHW1gHwphawr65b/5TsVrevhds11Wr/+ZybV5wyPfxJpJpiIXpwJ79gC+XS08XbAf5/n1G m+zHu5tmUYU2Nl48KJFGNT2SpvuUfvp2FIbcyVQhQkDJtXWyADyKLC6eRvvfsg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TXMPw6pz4zW4H for ; Sat, 10 Feb 2024 20:14:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41AKESwA056028 for ; Sat, 10 Feb 2024 20:14:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41AKESVK056026 for fs@FreeBSD.org; Sat, 10 Feb 2024 20:14:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed Date: Sat, 10 Feb 2024 20:14:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341 --- Comment #15 from John F. Carr --- A correction. My panic on December 12, 2023 was a different ZFS panic (bug #275731). I don't know if that kernel would have crashed on normal reboot.= =20 The kernel that crashed with the rc_count message was built on December 26 = and has these additional commits: commit 188408da9f7c19f476c1afe9becb0d373088da31 Merge: 45835894bd98 dbda45160ffa Author: Martin Matuska Date: Tue Dec 19 23:17:48 2023 +0100 zfs: merge openzfs/zfs@dbda45160 Notable upstream pull request merges: #15665 9b1677fb5 dmu: Allow buffer fills to fail Obtained from: OpenZFS OpenZFS commit: dbda45160ffa43e5ecf0498a609230f1afee7b3f commit 5fb307d29b364982acbde82cbf77db3cae486f8c Merge: c2e340452c14 86e115e21e5b Author: Martin Matuska Date: Fri Dec 15 14:17:23 2023 +0100 zfs: merge openzfs/zfs@86e115e21 Notable upstream pull request merges: #15643 a9b937e06 For db_marker inherit the db pointer for AVL comparis= ion #15644 e53e60c0b DMU: Fix lock leak on dbuf_hold() error #15653 86063d903 dbuf: Handle arcbuf assignment after block cloning #15656 86e115e21 dbuf: Set dr_data when unoverriding after clone Obtained from: OpenZFS OpenZFS commit: 86e115e21e5bdeee73c88c5a1a73a29d9637380a --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Feb 11 21:00:48 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TY0Nw4YvXz5BQ8V for ; Sun, 11 Feb 2024 21:00:48 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TY0Nw2B6bz4NQs for ; Sun, 11 Feb 2024 21:00:48 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707685248; a=rsa-sha256; cv=none; b=wvQwvpMhY+bG4Nc0A8pZCNN8pkq6IeUuNDQsSmawdQ4v9ydTqeO1xTS7zbZzwSn8tlqMzG ptnYVRbAn1mQua9hir4QoWHnM8OrQyR1eg5QGNYQncK+MMQUjHzSHknWbhe4dUrg6tjKmE t0nXyiGBV51LCcY5HFiBHQetRT8rvZCGkoR4RciONcJ12P8g2vJNaWgAB6tbscsLy33naY PFstsYzRBPgCocTJPKdk1rRDZ05RoxT61XjrnUlezChdUa1F+zQhD/gjD1nB5uUjylE0oK mo752/6MAxOGdCOOofjAxRU2CzKlp7OetPx9/0jnoezYcGR+i2E2dyteq2RlxQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707685248; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=yikNEO0AfW8qzIc1+3AgiAIrDqhIDho8sv08xY35U/o=; b=vuXU2Gk7uyjm0Q9YL1uL3tGlaufB8Esoi0M906P+VjePU0vgwRclxsmsxZlHQ1OvMJ47Am R60VxNShdR5p+z36DGul6NM0KYdz6OnjygceXtOBl0jl+WC1Y/g0uVuvxHkQ8O035NXqK7 oj9hGmvFKzE0v+J+Ji6wHQrUFMktRij3xAJqC/cy9Vva1GsRs/OLxjELf7E1ChJnilbJ6O hooPTy/mFVyHNKJUNgu+NgE574WskHzhQAkT7MiaUgfub50nm87yDRSsLaBVmoGjMaMT26 7pS4nTOyK5FbvPNBghTBqGI1jLdC9wW7svR+vLLOa6ZCM8H0wKpXAvH6pyPerA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TY0Nw1Fs1z1Dgh for ; Sun, 11 Feb 2024 21:00:48 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41BL0mL2085212 for ; Sun, 11 Feb 2024 21:00:48 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41BL0mEp085206 for fs@FreeBSD.org; Sun, 11 Feb 2024 21:00:48 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202402112100.41BL0mEp085206@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 11 Feb 2024 21:00:48 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17076852481.DAE1.75515" Content-Transfer-Encoding: 7bit --17076852481.DAE1.75515 Date: Sun, 11 Feb 2024 21:00:48 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 231794 | zfs: Panic due to ARC related KVA memory exhausti Open | 251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W Open | 264139 | ata: NCQ_DSM_TRIM trim method for Seagate IronWol Open | 266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy Open | 268162 | zfskeys_enable: each successful load of a key is Open | 269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit Open | 271384 | zfs_load is not suitably documented Open | 226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 = 8 problems total for which you should take action. --17076852481.DAE1.75515 Date: Sun, 11 Feb 2024 21:00:48 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    231794 | zfs: Panic due to ARC related KVA memory exhausti
Open        |    251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W
Open        |    264139 | ata: NCQ_DSM_TRIM trim method for Seagate IronWol
Open        |    266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy 
Open        |    268162 | zfskeys_enable: each successful load of a key is 
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit
Open        |    271384 | zfs_load is not suitably documented
Open        |    226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =

8 problems total for which you should take action.
--17076852481.DAE1.75515--