From nobody Mon Nov 1 20:09:49 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7D59A18136A3 for ; Mon, 1 Nov 2021 20:09:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hjkf532TBz4Vn9 for ; Mon, 1 Nov 2021 20:09:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4500F177DF for ; Mon, 1 Nov 2021 20:09:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A1K9nIW086819 for ; Mon, 1 Nov 2021 20:09:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A1K9nW5086818 for fs@FreeBSD.org; Mon, 1 Nov 2021 20:09:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: =?UTF-8?B?W0J1ZyAyNTkwOTBdIFVGUzogYmFkIGZpbGUgZGVzY3JpcHRvcjog?= =?UTF-8?B?c29mdCB1cGRhdGUgam91cm5hbGluZyBjYW4gbm90IGJlIGVuYWJsZWQgb24g?= =?UTF-8?B?RnJlZUJTRC1wcm92aWRlZCBkaXNrIGltYWdlcyBmb3IgMTMuMC1SRUxFQVNF?= =?UTF-8?B?IGFuZCAxMy4wLVNUQUJMRSDigJMgZmFpbGVkIHRvIHdyaXRlIHVwZGF0ZWQg?= =?UTF-8?B?Y2c=?= Date: Mon, 01 Nov 2021 20:09:49 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: mckusick@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D259090 Kirk McKusick changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Open CC| |mckusick@FreeBSD.org --- Comment #4 from Kirk McKusick --- I have been able to reproduce the problem and will look into it. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Tue Nov 2 06:36:24 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 327331824313 for ; Tue, 2 Nov 2021 06:36:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hk0Y50rrMz4RFS for ; Tue, 2 Nov 2021 06:36:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EE57D20410 for ; Tue, 2 Nov 2021 06:36:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A26aOGx024454 for ; Tue, 2 Nov 2021 06:36:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A26aO58024453 for fs@FreeBSD.org; Tue, 2 Nov 2021 06:36:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 259607] sysutils/node_exporter / OpenZFS: need better handling for ZFS statistic numbers Date: Tue, 02 Nov 2021 06:36:24 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: delphij@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: dor.bsd@xm0.uk X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter cc flagtypes.name Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D259607 Bug ID: 259607 Summary: sysutils/node_exporter / OpenZFS: need better handling for ZFS statistic numbers Product: Ports & Packages Version: Latest Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: Individual Port(s) Assignee: dor.bsd@xm0.uk Reporter: delphij@FreeBSD.org CC: dor.bsd@xm0.uk, fs@FreeBSD.org CC: dor.bsd@xm0.uk Flags: maintainer-feedback?(dor.bsd@xm0.uk) node_exporter is spamming syslog with: level=3Derror ts=3D2021-11-02T06:18:08.285Z caller=3Dstdlib.go:105 msg=3D"e= rror gathering metrics: 11 error(s) occurred:\n* [from Gatherer #2] collected me= tric \"sysctl_vfs_zfs_l2arc_write_boost\" { untyped: } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"sysctl_vfs_zfs_arc_max\" { untyped: } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"sysctl_vfs_zfs_l2arc_feed_min_ms\" { untyped: } was collected before with the same name and label values\n* [from Gathe= rer #2] collected metric \"sysctl_vm_uma_tcp_log_bucket_size\" { untyped: } was collected before with the same name and label values\n* [from Gathe= rer #2] collected metric \"sysctl_vfs_zfs_l2arc_write_max\" { untyped: } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"sysctl_vfs_zfs_l2arc_feed_again\" { untyped: } was collected be= fore with the same name and label values\n* [from Gatherer #2] collected metric \"sysctl_vfs_zfs_l2arc_norw\" { untyped: } was collected before w= ith the same name and label values\n* [from Gatherer #2] collected metric \"sysctl_vfs_zfs_l2arc_noprefetch\" { untyped: } was collected be= fore with the same name and label values\n* [from Gatherer #2] collected metric \"sysctl_vfs_zfs_arc_min\" { untyped: } was collected before with= the same name and label values\n* [from Gatherer #2] collected metric \"sysctl_vfs_zfs_l2arc_feed_secs\" { untyped: } was collected bef= ore with the same name and label values\n* [from Gatherer #2] collected metric \"sysctl_vfs_zfs_l2arc_headroom\" { untyped: } was collected befo= re with the same name and label values" The problem is that some ZFS values are exported from two OIDs, for example: vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc.feed_min_ms: 200 But node_exporter is aliasing "." to "_" unconditionally. To get a list of all affected sysctl OIDs, one can use: sysctl -da | grep -E \("$(sysctl -Na | sed -e s,\\.,_,g | sort | uniq -c | = sort -n | awk '{ if ($1 >1) print $2; }' | sed -e s,_,.,g | paste -sd \| -)"\): and on FreeBSD 13.0-RELEASE-p4, I got: vm.uma.tcp_log_bucket.size: Allocation size vm.uma.tcp_log.bucket_size: Desired per-cpu cache size vfs.zfs.arc_max: max arc size (LEGACY) vfs.zfs.arc_min: min arc size (LEGACY) vfs.zfs.l2arc_norw: no reads during writes (LEGACY) vfs.zfs.l2arc_feed_again: turbo warmup (LEGACY) vfs.zfs.l2arc_noprefetch: don't cache prefetch bufs (LEGACY) vfs.zfs.l2arc_feed_min_ms: min interval milliseconds (LEGACY) vfs.zfs.l2arc_feed_secs: interval seconds (LEGACY) vfs.zfs.l2arc_headroom: number of dev writes (LEGACY) vfs.zfs.l2arc_write_boost: extra write during warmup (LEGACY) vfs.zfs.l2arc_write_max: max write size (LEGACY) vfs.zfs.l2arc.norw: No reads during writes vfs.zfs.l2arc.feed_again: Turbo L2ARC warmup vfs.zfs.l2arc.noprefetch: Skip caching prefetched buffers vfs.zfs.l2arc.feed_min_ms: Min feed interval in milliseconds vfs.zfs.l2arc.feed_secs: Seconds between L2ARC writing vfs.zfs.l2arc.headroom: Number of max device writes to precache vfs.zfs.l2arc.write_boost: Extra write bytes during device warmup vfs.zfs.l2arc.write_max: Max write bytes per interval vfs.zfs.arc.max: Max arc size vfs.zfs.arc.min: Min arc size So it's not just ZFS. I *think* the proper fix should be to change the translation code to first replace all _ with __, then replace . with _. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Tue Nov 2 11:56:34 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 43F261847520 for ; Tue, 2 Nov 2021 11:56:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hk7fV0fJNz3J5F for ; Tue, 2 Nov 2021 11:56:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EC352244B4 for ; Tue, 2 Nov 2021 11:56:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A2BuXhm087446 for ; Tue, 2 Nov 2021 11:56:33 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A2BuXwt087445 for fs@FreeBSD.org; Tue, 2 Nov 2021 11:56:33 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 259607] sysutils/node_exporter / OpenZFS: need better handling for ZFS statistic numbers Date: Tue, 02 Nov 2021 11:56:34 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: dor.bsd@xm0.uk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: dor.bsd@xm0.uk X-Bugzilla-Flags: maintainer-feedback+ X-Bugzilla-Changed-Fields: flagtypes.name Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D259607 David O'Rourke changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|maintainer-feedback?(dor.bs |maintainer-feedback+ |d@xm0.uk) | --- Comment #1 from David O'Rourke --- Hi, Are these errors actually from the FreeBSD `prometheus_sysctl_exporter`? The `sysctl_` prefix suggests so. Node Exporter doesn't appear to export all of the sysctls anywhere and the metrics it does expose for ZFS and other sysctl based things are very specifically selected and named to avoid collisions. prometheus_sysctl_exporter does appear to show this unfortunate problem: $ prometheus_sysctl_exporter | grep l2arc | sort sysctl_vfs_zfs_l2arc_feed_again 1 sysctl_vfs_zfs_l2arc_feed_again 1 sysctl_vfs_zfs_l2arc_feed_min_ms 200 sysctl_vfs_zfs_l2arc_feed_min_ms 200 Are you using the Node Exporter to export these `prometheus_sysctl_exporter` metrics via the --collector.textfile.directory feature, perhaps? -David --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Tue Nov 2 18:24:37 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D78E3182EF9D for ; Tue, 2 Nov 2021 18:24:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HkJGF5Zrjz3R24 for ; Tue, 2 Nov 2021 18:24:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9FA361AB3 for ; Tue, 2 Nov 2021 18:24:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A2IObUC001004 for ; Tue, 2 Nov 2021 18:24:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A2IObt9001003 for fs@FreeBSD.org; Tue, 2 Nov 2021 18:24:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 256868] lock order reversal in vfs_mount when mounting ZFS based jail Date: Tue, 02 Nov 2021 18:24:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mjg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256868 Mateusz Guzik changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mjg@FreeBSD.org Status|New |Closed Resolution|--- |FIXED --- Comment #1 from Mateusz Guzik --- fixed in https://cgit.freebsd.org/src/commit?id=3Ddbc689cdef0cc8ff11171642cdcf107dfb= c3fb41 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Nov 2 18:25:20 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4EFDD182F354 for ; Tue, 2 Nov 2021 18:25:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HkJH416d3z3h69 for ; Tue, 2 Nov 2021 18:25:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 078CB1D23 for ; Tue, 2 Nov 2021 18:25:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A2IPJhU001159 for ; Tue, 2 Nov 2021 18:25:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A2IPJO6001158 for fs@FreeBSD.org; Tue, 2 Nov 2021 18:25:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 228426] [lor] isofs devfs when rebooting the system after successful installation Date: Tue, 02 Nov 2021 18:25:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mjg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228426 Mateusz Guzik changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |FIXED CC| |mjg@FreeBSD.org --- Comment #4 from Mateusz Guzik --- fixed in https://cgit.freebsd.org/src/commit?id=3Ddbc689cdef0cc8ff11171642cdcf107dfb= c3fb41 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Nov 2 18:26:57 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 75CC61831656 for ; Tue, 2 Nov 2021 18:26:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HkJJx2ktFz3jXd for ; Tue, 2 Nov 2021 18:26:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3F67F1C30 for ; Tue, 2 Nov 2021 18:26:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A2IQvfW001513 for ; Tue, 2 Nov 2021 18:26:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A2IQvcW001512 for fs@FreeBSD.org; Tue, 2 Nov 2021 18:26:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 191782] [ufs] [lor] ufs/devfs on reboot r264318 Date: Tue, 02 Nov 2021 18:26:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mjg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D191782 Mateusz Guzik changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mjg@FreeBSD.org Resolution|--- |FIXED Status|Open |Closed --- Comment #2 from Mateusz Guzik --- fixed in https://cgit.freebsd.org/src/commit?id=3Ddbc689cdef0cc8ff11171642cdcf107dfb= c3fb41 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Nov 2 18:28:23 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3DA6C1833394 for ; Tue, 2 Nov 2021 18:28:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HkJLb17kTz3kpG for ; Tue, 2 Nov 2021 18:28:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 064C61D26 for ; Tue, 2 Nov 2021 18:28:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A2ISMnB001753 for ; Tue, 2 Nov 2021 18:28:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A2ISMHw001752 for fs@FreeBSD.org; Tue, 2 Nov 2021 18:28:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 221570] lock order reversal fs_mount.c:1271, msdosfs_vfsops.c:928 Date: Tue, 02 Nov 2021 18:28:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mjg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221570 Mateusz Guzik changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |mjg@FreeBSD.org Status|New |Closed --- Comment #1 from Mateusz Guzik --- fixed in https://cgit.freebsd.org/src/commit?id=3Ddbc689cdef0cc8ff11171642cdcf107dfb= c3fb41 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Nov 2 19:48:40 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 79014182F773 for ; Tue, 2 Nov 2021 19:48:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HkL7F0BJ0z4gjX for ; Tue, 2 Nov 2021 19:48:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9CF2F3193 for ; Tue, 2 Nov 2021 19:48:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A2JmeLt040915 for ; Tue, 2 Nov 2021 19:48:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A2JmeYc040914 for fs@FreeBSD.org; Tue, 2 Nov 2021 19:48:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 228750] panic on zfs mirror removal Date: Tue, 02 Nov 2021 19:48:40 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ahmedsayeed1982@yahoo.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: mav@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228750 Ahmed changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ahmedsayeed1982@yahoo.com --- Comment #22 from Ahmed --- ------8<------- 1 size_t fwrite(const void * __restrict ptr, size_t size, http://www-look-4.com/category/travel/ 2 size_t nmemb, register FILE * __restrict stream) 3 { 4 size_t retval; https://komiya-dental.com/category/technology/ 5 __STDIO_AUTO_THREADLOCK_VAR; 6 http://www.iu-bloomington.com/category/technology/ 7 > __STDIO_AUTO_THREADLOCK(stream); 8=20 9 retval =3D fwrite_unlocked(ptr, size, nmemb, stream); 10 https://waytowhatsnext.com/category/technology/ 11 __STDIO_AUTO_THREADUNLOCK(stream); 12 http://www.wearelondonmade.com/category/travel/ 13 return retval; 14 } ------>8------- http://www.jopspeech.com/category/travel/ Here, we are at line 7. Using the "next" command leads no where. However, setting a breakpoint on line 9 and issuing "continue" works. http://joerg.li/category/travel/ Looking at the assembly instructions reveals that we're dealing with the critical section entry code http://the-hunters.org/technology/meta-symbol/ = [1] that should never be interrupted, in this case by the debugger's implicit breakpoints: http://connstr.net/category/travel/ ------8<------- ... http://embermanchester.uk/category/travel/ 1 add_s r0,r13,0x38 2 mov_s r3,1 3 llock r2,[r0] <-. 4 brne.nt r2,0,14 --. | http://www.slipstone.co.uk/category/travel/ 5 scond r3,[r0] | | 6 bne -10 --|--' 7 brne_s r2,0,84 <-' http://www.logoarts.co.uk/category/travel/ ... ------>8------- http://www.acpirateradio.co.uk/category/travel/ Lines 3 until 5 (inclusive) are supposed to be executed atomically. Therefo= re, http://fishingnewsletters.co.uk/crypto/kelleci-bay/ GDB should never (implicitly) insert a breakpoint on lines 4 and 5, else the http://www.compilatori.com/category/travel/=20 program will try to acquire the lock again by jumping back to line 3 and gets stuck in an infinite loop. https://www.webb-dev.co.uk/category/technol= ogy/ The solution is to make GDB aware of these patterns so it inserts breakpoin= ts after the sequence -- line 6 in this example. The solution is to make GDB a= ware of these patterns so it inserts breakpoints http://www.go-mk-websites.co.uk/services/kyoto/ after the sequence -- line 6 in this example. The solution is to make GDB aware of these patterns so it inserts breakpoi= nts after the sequence -- line 6 in this example. http://www.mconstantine.co.uk/services/new-zealand/ The solution is to make GDB aware of these patterns so it inserts breakpoin= ts after the sequence -- line 6 in this example. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Wed Nov 3 00:52:24 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 211471827101 for ; Wed, 3 Nov 2021 00:52:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HkSsh0Gymz3DHP for ; Wed, 3 Nov 2021 00:52:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DF0937347 for ; Wed, 3 Nov 2021 00:52:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A30qNkL004660 for ; Wed, 3 Nov 2021 00:52:23 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A30qNER004659 for fs@FreeBSD.org; Wed, 3 Nov 2021 00:52:23 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: =?UTF-8?B?W0J1ZyAyNTkwOTBdIFVGUzogYmFkIGZpbGUgZGVzY3JpcHRvcjog?= =?UTF-8?B?c29mdCB1cGRhdGUgam91cm5hbGluZyBjYW4gbm90IGJlIGVuYWJsZWQgb24g?= =?UTF-8?B?RnJlZUJTRC1wcm92aWRlZCBkaXNrIGltYWdlcyBmb3IgMTMuMC1SRUxFQVNF?= =?UTF-8?B?IGFuZCAxMy4wLVNUQUJMRSDigJMgZmFpbGVkIHRvIHdyaXRlIHVwZGF0ZWQg?= =?UTF-8?B?Y2c=?= Date: Wed, 03 Nov 2021 00:52:24 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: grahamperrin@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D259090 --- Comment #5 from Graham Perrin --- Created attachment 229232 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D229232&action= =3Dedit Screenshot: FreeBSD-12.3-PRERELEASE-amd64.vhd OK (In reply to Graham Perrin from comment #0) > Not reproducible with FreeBSD-12.2-RELEASE-amd64.vhd.xz Similarly, not reproducible with recent FreeBSD-12.3-PRERELEASE-amd64.vhd.x= z =E2=80=93 screenshot attached.=20 (In reply to Kirk McKusick from comment #4) Thanks. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Wed Nov 3 22:10:43 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 377911842690 for ; Wed, 3 Nov 2021 22:10:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hl1Dg10Gwz4pq6 for ; Wed, 3 Nov 2021 22:10:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 02CBF20461 for ; Wed, 3 Nov 2021 22:10:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A3MAgpX049916 for ; Wed, 3 Nov 2021 22:10:42 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A3MAg53049915 for fs@FreeBSD.org; Wed, 3 Nov 2021 22:10:42 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254282] 13.0-RC2: NFS export from nullfs mount doesn't work as of 13.0 Date: Wed, 03 Nov 2021 22:10:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: marek@mky.waw.pl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254282 marek@mky.waw.pl changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marek@mky.waw.pl --- Comment #7 from marek@mky.waw.pl --- For me, the mountd does not see a nullfs as a separate filesystem and tries= to set the attributes for the upper filesystem.=20 The simple workaround that worked for me is to create an empty ZFS dataset mounted at the same point as nullfs, but before nullfs, eg.: zfs create -o mountpoint=3D/usr/src zpool/nfs-usr-src=20 mount_nullfs /var/jail/builder/src /usr/src After that, the NFS exports work fine for me. Tested on 13-RELEASE-p4. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Nov 4 23:09:29 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 545D21806D4E for ; Thu, 4 Nov 2021 23:09:42 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:13b:39f::9f:25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HlfVF0G0qz4dPN for ; Thu, 4 Nov 2021 23:09:41 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id D26ED8D4A215 for ; Thu, 4 Nov 2021 23:09:32 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 30079E708BA for ; Thu, 4 Nov 2021 23:09:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id G0-vuqo4S4Wg for ; Thu, 4 Nov 2021 23:09:30 +0000 (UTC) Received: from nv.sbone.de (nv.sbone.de [IPv6:fde9:577b:c1a9:31::2013:138]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 4B0D6E707AC for ; Thu, 4 Nov 2021 23:09:30 +0000 (UTC) Date: Thu, 4 Nov 2021 23:09:29 +0000 (UTC) From: "Bjoern A. Zeeb" To: freebsd-fs@freebsd.org Subject: panic: ufs_dirbad: /: bad dir ino 160292 at offset 512: mangled entry Message-ID: X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 4HlfVF0G0qz4dPN X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of bzeeb-lists@lists.zabbadoz.net designates 2a01:4f8:13b:39f::9f:25 as permitted sender) smtp.mailfrom=bzeeb-lists@lists.zabbadoz.net X-Spamd-Result: default: False [-3.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:4f8:13b:39f::9f:25]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_NA(0.00)[zabbadoz.net]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE]; MIME_TRACE(0.00)[0:+] X-ThisMailContainsUnwantedMimeParts: N Just got this on a VM disk image. I couldn't dump as I needed the crashdump which was waiting for me already (means the system had previously paniced). A full fsck w/o journal seems to have fixed it on the next round. Just posting in case this is seen more often so people can find it. Lots of health (or more than that fs that moment) to all of you, /bz 2021-11-04T22:59:47.564904+00:00 - init 1 - - /bin/sh on /etc/rc terminated abnormally, going to single user mode Enter full pathname of shell or RETURN for /bin/sh: root@:/ # fsck / ** /dev/ada0p4 ** SU+J Recovering /dev/ada0p4 USE JOURNAL? [yn] y ** Reading 33554432 byte journal from inode 4. RECOVER? [yn] y ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. WRITE CHANGES? [yn] y ***** FILE SYSTEM IS CLEAN ***** ** 13 journal records in 2048 bytes for 20.31% utilization ** Freed 2 inodes (0 dirs) 1 blocks, and 3 frags. ***** FILE SYSTEM MARKED CLEAN ***** root@:/ # ^D Setting hostuuid: e0738d77-da8d-11eb-8339-000db9559b74. Setting hostid: 0x789c8996. Fast boot: skipping disk checks. panic: ufs_dirbad: /: bad dir ino 160292 at offset 512: mangled entry cpuid = 0 time = 1636066794 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0051f1e6a0 vpanic() at vpanic+0x187/frame 0xfffffe0051f1e700 panic() at panic+0x43/frame 0xfffffe0051f1e760 ufs_lookup_ino() at ufs_lookup_ino+0xd9a/frame 0xfffffe0051f1e860 vfs_cache_lookup() at vfs_cache_lookup+0xa6/frame 0xfffffe0051f1e8b0 cache_fplookup_final_modifying() at cache_fplookup_final_modifying+0x2f4/frame 0xfffffe0051f1e930 cache_fplookup_noentry() at cache_fplookup_noentry+0xb2/frame 0xfffffe0051f1e9a0 cache_fplookup() at cache_fplookup+0x5f5/frame 0xfffffe0051f1ea90 namei() at namei+0x1e6/frame 0xfffffe0051f1eb50 vn_open_cred() at vn_open_cred+0x12c/frame 0xfffffe0051f1ecb0 kern_openat() at kern_openat+0x285/frame 0xfffffe0051f1ee00 amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe0051f1ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0051f1ef30 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x8011e3c2a, rsp = 0x7fffffffe248, rbp = 0x7fffffffe330 --- KDB: enter: panic [ thread pid 89 tid 100099 ] Stopped at kdb_enter+0x37: movq $0,0x127094e(%rip) -- Bjoern A. Zeeb r15:7 From nobody Fri Nov 5 10:13:21 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 45929183FF79 for ; Fri, 5 Nov 2021 10:13:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HlxD11Rvvz3Hhq for ; Fri, 5 Nov 2021 10:13:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 12E6515D30 for ; Fri, 5 Nov 2021 10:13:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A5ADLQm096148 for ; Fri, 5 Nov 2021 10:13:21 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A5ADLjl096147 for fs@FreeBSD.org; Fri, 5 Nov 2021 10:13:21 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 221909] [ZFS] Add a sysctl to toggle send_corrupt_data Date: Fri, 05 Nov 2021 10:13:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: fk@fabiankeil.de X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: mfc-stable11? X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221909 Fabian Keil changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Open |Closed Resolution|--- |Overcome By Events --- Comment #4 from Fabian Keil --- I've recently updated to STABLE/12 and confirmed that the vfs.zfs.send_corrupt_data sysctl works as advertised. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Nov 5 15:40:22 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7A341183F8EA for ; Fri, 5 Nov 2021 15:40:34 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm4TY2RpSz4sGn for ; Fri, 5 Nov 2021 15:40:33 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 5 Nov 2021 15:40:25 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1005::15]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id dbcde58d (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Fri, 5 Nov 2021 11:40:24 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: ZFS operations hanging, but no visible errors? Message-Id: Date: Fri, 5 Nov 2021 11:40:22 -0400 To: freebsd-fs X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4Hm4TY2RpSz4sGn X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of cross@distal.com designates 168.235.95.80 as permitted sender) smtp.mailfrom=cross@distal.com X-Spamd-Result: default: False [0.20 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.999]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[distal.com]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+a:relay.dynu.com]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]; MIME_TRACE(0.00)[0:+]; TAGGED_FROM(0.00)[freebsd]; RECEIVED_SPAMHAUS_PBL(0.00)[108.48.165.176:received] X-ThisMailContainsUnwantedMimeParts: N Hey there. I have a server running FreeBSD 13.0-RELEASE, with a large = ZFS zpool. I have a UFS mirror on hardware raid, then a bunch of JBOD = disks in a pool. I recently added a new vdev to this pool, which may or = may not be related. Today, I started an rsync of a large (100GB) file from the pool to = another host. After a while (7%), it seemed no progres was being made. = I tried to kill the rsync, which didn=E2=80=99t exit, or suspend. Now = anything that touches the pool seems to hang. But, the system is = otherwise functional, console shows no issues, the controller (via = out-of-band management interface) shows all disks as having no errors or = issues. Any idea what I should be looking for, and if there=E2=80=99s any way to = recover it without reboot? Another operation I had running was in the = middle of writing data I=E2=80=99d rather not lose progress on, which is = why I haven=E2=80=99t rebooted yet. - Chris= From nobody Fri Nov 5 15:43:09 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2B32E1840497 for ; Fri, 5 Nov 2021 15:43:14 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm4Xd6lYyz4t29 for ; Fri, 5 Nov 2021 15:43:13 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 5 Nov 2021 15:43:12 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1005::15]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 231a5f73 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Fri, 5 Nov 2021 11:43:11 -0400 (EDT) From: Chris Ross Content-Type: multipart/alternative; boundary="Apple-Mail=_14D70F2F-1C8D-44AC-866F-3816F3AFB9B2" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: ZFS operations hanging, but no visible errors? Date: Fri, 5 Nov 2021 11:43:09 -0400 References: To: freebsd-fs In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4Hm4Xd6lYyz4t29 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: Y --Apple-Mail=_14D70F2F-1C8D-44AC-866F-3816F3AFB9B2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On Nov 5, 2021, at 11:40, Chris Ross wrote: Hey there. I have a server running FreeBSD 13.0-RELEASE, with a large = ZFS zpool. I have a UFS mirror on hardware raid, then a bunch of JBOD = disks in a pool. I recently added a new vdev to this pool, which may or = may not be related. Today, I started an rsync of a large (100GB) file from the pool to = another host. After a while (7%), it seemed no progres was being made. = I tried to kill the rsync, which didn=E2=80=99t exit, or suspend. Now = anything that touches the pool seems to hang. But, the system is = otherwise functional, console shows no issues, the controller (via = out-of-band management interface) shows all disks as having no errors or = issues. Any idea what I should be looking for, and if there=E2=80=99s any way to = recover it without reboot? Another operation I had running was in the = middle of writing data I=E2=80=99d rather not lose progress on, which is = why I haven=E2=80=99t rebooted yet. - Chris Ps, apologies for the rapid secondary email, but I forgot to mention = something important. A =E2=80=9Czpool status tank=E2=80=9D is also = hanging. So, it=E2=80=99s not just FS operations that are stuck. --Apple-Mail=_14D70F2F-1C8D-44AC-866F-3816F3AFB9B2-- From nobody Fri Nov 5 15:59:26 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C24ED184A792 for ; Fri, 5 Nov 2021 15:59:32 +0000 (UTC) (envelope-from wfc@mintsol.com) Received: from scully.mintsol.com (scully.mintsol.com [199.182.77.206]) by mx1.freebsd.org (Postfix) with ESMTP id 4Hm4vS49d6z3HdZ for ; Fri, 5 Nov 2021 15:59:32 +0000 (UTC) (envelope-from wfc@mintsol.com) Received: from mintsol.com (officecc.mintsol.com [96.85.114.33]) by scully.mintsol.com with esmtp; Fri, 05 Nov 2021 11:59:26 -0400 id 00BA88D1.00000000618554DE.0000149F Received: from localhost (localhost [127.0.0.1]) (IDENT: uid 1002) by mintsol.com with esmtp; Fri, 05 Nov 2021 11:59:26 -0400 id 000001FC.618554DE.000177CA Date: Fri, 5 Nov 2021 11:59:26 -0400 (EDT) From: Walter Cramer To: Chris Ross cc: freebsd-fs Subject: Re: ZFS operations hanging, but no visible errors? In-Reply-To: Message-ID: <20211105115234.H95319@mulder.mintsol.com> References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-2138392286-1636127895=:95319" Content-ID: <20211105115818.U95319@mulder.mintsol.com> X-Rspamd-Queue-Id: 4Hm4vS49d6z3HdZ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[freebsd]; REPLY(-4.00)[] X-Spam: Yes X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-2138392286-1636127895=:95319 Content-Type: TEXT/PLAIN; CHARSET=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Content-ID: <20211105115818.E95319@mulder.mintsol.com> On Fri, 5 Nov 2021, Chris Ross wrote: > > > On Nov 5, 2021, at 11:40, Chris Ross wrote: > > Hey there. I have a server running FreeBSD 13.0-RELEASE, with a large ZF= S zpool. I have a UFS mirror on hardware raid, then a bunch of JBOD disks = in a pool. I recently added a new vdev to this pool, which may or may not = be related. > > Today, I started an rsync of a large (100GB) file from the pool to anothe= r host. After a while (7%), it seemed no progres was being made. I tried = to kill the rsync, which didn=E2=80=99t exit, or suspend. Now anything tha= t touches the pool seems to hang. But, the system is otherwise functional,= console shows no issues, the controller (via out-of-band management interf= ace) shows all disks as having no errors or issues. > > Any idea what I should be looking for, and if there=E2=80=99s any way to = recover it without reboot? Another operation I had running was in the midd= le of writing data I=E2=80=99d rather not lose progress on, which is why I = haven=E2=80=99t rebooted yet. > > - Chris > > Ps, apologies for the rapid secondary email, but I forgot to mention some= thing important. A =E2=80=9Czpool status tank=E2=80=9D is also hanging. S= o, it=E2=80=99s not just FS operations that are stuck. > Any chance that you have output from an earlier `zpool status`, or even=20 `zpool list`, to give us a better sense of your situation? Might `dmesg -a` show any relevant errors? (FWIW, I usually have a daily cron job log a few things like `zpool=20 status` to /var/backups/zfs, to help with situations like this.) --0-2138392286-1636127895=:95319-- From nobody Fri Nov 5 16:19:08 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 102331803F9F for ; Fri, 5 Nov 2021 16:19:15 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm5LB4srWz3QKc for ; Fri, 5 Nov 2021 16:19:14 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 5 Nov 2021 16:19:11 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1005::15]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id e4f60858 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 5 Nov 2021 12:19:10 -0400 (EDT) Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: ZFS operations hanging, but no visible errors? From: Chris Ross In-Reply-To: <20211105115234.H95319@mulder.mintsol.com> Date: Fri, 5 Nov 2021 12:19:08 -0400 Cc: freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: <0CB9C85C-867C-4501-A9DC-9B730303E388@distal.com> References: <20211105115234.H95319@mulder.mintsol.com> To: Walter Cramer X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4Hm5LB4srWz3QKc X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: N > On Nov 5, 2021, at 11:59, Walter Cramer wrote: >=20 >=20 > Any chance that you have output from an earlier `zpool status`, or = even `zpool list`, to give us a better sense of your situation? Sadly, no. I looked through my buffers and don=E2=80=99t. Have a few = df=E2=80=99s, but other than a showing a system with it=E2=80=99s few = filesystem including the 40T zfs one, it doesn=E2=80=99t help. Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 3.5T 4.2G 3.2T 0% / devfs 1.0K 1.0K 0B 100% /dev /dev/da0p1 256M 1.7M 254M 1% /boot/efi /dev/ada1p1 445G 36K 423G 0% /ssd/scratch tank 49T 9.5T 40T 19% /tank I can say that the pool has three raidz vdevs in it, each with three = devices. 3 4TB disks, 3 6TB disks, and 3 18TB disks. > Might `dmesg -a` show any relevant errors? Nope, that shows just a clean boot from when I rebooted the system = yesterday. > (FWIW, I usually have a daily cron job log a few things like `zpool = status` to /var/backups/zfs, to help with situations like this.) Yeah. Maybe that should be in the normal daily job if the system is = using any ZFS volumes=E2=80=A6. - Chris= From nobody Fri Nov 5 16:39:35 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6BCEF1831543 for ; Fri, 5 Nov 2021 16:40:12 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay03.ispgateway.de (smtprelay03.ispgateway.de [80.67.18.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm5pN2fKxz3p09 for ; Fri, 5 Nov 2021 16:40:12 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [91.20.68.89] (helo=fabiankeil.de) by smtprelay03.ispgateway.de with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mj2GO-0004hu-Kj for freebsd-fs@freebsd.org; Fri, 05 Nov 2021 17:40:24 +0100 Date: Fri, 5 Nov 2021 17:39:35 +0100 From: Fabian Keil To: freebsd-fs@freebsd.org Subject: Re: ZFS operations hanging, but no visible errors? Message-ID: <20211105173935.7aa53269@fabiankeil.de> In-Reply-To: References: Reply-To: freebsd-fs@freebsd.org List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/tBj+BYLZ16DGO2OLxlmvMTu"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Df-Sender: Nzc1MDY3 X-Rspamd-Queue-Id: 4Hm5pN2fKxz3p09 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N --Sig_/tBj+BYLZ16DGO2OLxlmvMTu Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Chris Ross wrote on 2021-11-05: > Hey there. I have a server running FreeBSD 13.0-RELEASE, > with a large ZFS zpool. I have a UFS mirror on hardware > raid, then a bunch of JBOD disks in a pool. I recently > added a new vdev to this pool, which may or may not be related. >=20 > Today, I started an rsync of a large (100GB) file from > the pool to another host. After a while (7%), it seemed > no progres was being made. I tried to kill the rsync, > which didn=E2=80=99t exit, or suspend. Now anything that touches > the pool seems to hang. But, the system is otherwise > functional, console shows no issues, the controller > (via out-of-band management interface) shows all disks > as having no errors or issues. >=20 > Any idea what I should be looking for, and if there=E2=80=99s > any way to recover it without reboot? You can check with "procstat -kk" where the hanging processes are stuck. The output of: dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, stringof(arg0))}' could be useful as well. Fabian --Sig_/tBj+BYLZ16DGO2OLxlmvMTu Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQTKUNd6H/m3+ByGULIFiohV/3dUnQUCYYVeSAAKCRAFiohV/3dU nR5+AKDKNghhEaIH8ksYjE82yz/+NpyQ9wCgiAEkOv6otYFHdrbBgNkZlDx2/j8= =Krpn -----END PGP SIGNATURE----- --Sig_/tBj+BYLZ16DGO2OLxlmvMTu-- From nobody Fri Nov 5 16:59:47 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 076E9183CE24 for ; Fri, 5 Nov 2021 16:59:52 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm6F35MHwz3vYY for ; Fri, 5 Nov 2021 16:59:51 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 5 Nov 2021 16:59:50 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1005::15]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 351d76a9 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Fri, 5 Nov 2021 12:59:48 -0400 (EDT) From: Chris Ross Content-Type: multipart/alternative; boundary="Apple-Mail=_9D4F7EDC-2107-491F-8DD4-205632C97D87" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: ZFS operations hanging, but no visible errors? Date: Fri, 5 Nov 2021 12:59:47 -0400 References: <20211105173935.7aa53269@fabiankeil.de> To: freebsd-fs In-Reply-To: <20211105173935.7aa53269@fabiankeil.de> Message-Id: <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4Hm6F35MHwz3vYY X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: Y --Apple-Mail=_9D4F7EDC-2107-491F-8DD4-205632C97D87 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 5, 2021, at 12:39, Fabian Keil = wrote: > You can check with "procstat -kk" where the hanging > processes are stuck. Running prostate -kk on the rsync that was hung, then killed, then = SIGKILL=E2=80=99d shows: procstat -kk 35220 PID TID COMM TDNAME KSTACK 35220 102499 rsync - mi_switch+0xc1 = _sleep+0x1cb vm_wait_doms+0xe2 vm_wait_domain+0x51 = vm_domain_alloc_fail+0x86 vm_page_alloc_domain_after+0x7e = uma_small_alloc+0x58 keg_alloc_slab+0xba zone_import+0xee = zone_alloc_item+0x6f abd_alloc_chunks+0x61 abd_alloc+0x102 = arc_hdr_alloc_abd+0xb0 arc_hdr_alloc+0x11e arc_read+0x4f4 = dbuf_issue_final_prefetch+0x108 dbuf_prefetch_impl+0x3d0 = dmu_zfetch+0x558 >=20 > The output of: > dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, = stringof(arg0))}' > could be useful as well. I=E2=80=99ve run=20 % sudo dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, = stringof(arg0))}=E2=80=99 But, it=E2=80=99s yielding no output. Is that normal because it outputs = information about operations (which aren=E2=80=99t happening), or should = it=E2=80=99s output have been immediate? - Chris= --Apple-Mail=_9D4F7EDC-2107-491F-8DD4-205632C97D87-- From nobody Fri Nov 5 17:03:35 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 00AFE183FAEF for ; Fri, 5 Nov 2021 17:03:41 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-vk1-xa30.google.com (mail-vk1-xa30.google.com [IPv6:2607:f8b0:4864:20::a30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm6KS4zgkz4Rcg for ; Fri, 5 Nov 2021 17:03:40 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: by mail-vk1-xa30.google.com with SMTP id bc10so4935559vkb.1 for ; Fri, 05 Nov 2021 10:03:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=YElTuuUyZQ9lhs/B6o8hdLBMTagP6HXunYVrb6L1aZc=; b=ZPKCiVBevoLD8psDCk1krobLlqsWHqAM2IFJP/M+u1xiQzuclvpnDKv78N/Gns9cP6 LNLlYnl32rdPSXH2VDc5INHaADL7HAb+m4vJGj6GrhATDIo5aW197xkzjaL/u4dgHnkU ninYR9/O+gLORddd1pzjFo3tiLYnchvlCmm9r2fCA8POVDTyICdZbSqbGcbrg38oU6W6 4gdBCZdo9dvo0RnZx6O5Bw7V3oxENm57lDOMeaZ8/fEXW9MVA8Rv2pPBWucAJFugxiHj 4hXH/Nkx/uaCqSxS0Ispxa8uPOfLYqjQsZ62M8tl2+duFurTn7nOjG5BrJDhujEwXTn3 15+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=YElTuuUyZQ9lhs/B6o8hdLBMTagP6HXunYVrb6L1aZc=; b=qoLtJsk15NuVszlq2yQM30QjTY3+q4r16Fl6I0t0hoOProGYlwgifirW7fplnEYreS iNkWHi3W2jWNR3C6KCS6NMTiEYjl6jQazk6aa/YXTLtlTJlZiOhRJ13mNW0bKI95FKXT b4rOlgTyYcDgcyepfyQU9HY43wUgIKI+FjsSAN5hTrjqCf9S768mldTixNy3kvRMghpR OY6jOuJk7tlubR2KCtfc9uDnnmrPxgozOcytQ4toUS2qaa6vZ91UMxMidfCh6IQ4PlqA KPAfwdcPDM8qAwK6ZRFGCmBcgEGqcXA9+wsHO1Ze073C4RmoIW3EHgR4ioQDJn3E2Xpi 3TGw== X-Gm-Message-State: AOAM533A+LNNoYRf4Uus82ZBPpQIp6btbXOBuOQyAI8Mfwvin808NB+m geMX10LguwhkK2PxiTuuNHWu73pXzuAojvT5BlnIJMGfLko= X-Google-Smtp-Source: ABdhPJz0Gj6ImZAf6E5uTQtbKJWzqcGT+l7gulaOFJzhm3gIGDx9zWgMfVI6LqmvCtlGoAFqnDILZUcVrsPCSCw+xPs= X-Received: by 2002:a1f:cf47:: with SMTP id f68mr45968837vkg.10.1636131820205; Fri, 05 Nov 2021 10:03:40 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <20211105173935.7aa53269@fabiankeil.de> <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> In-Reply-To: <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> From: Rich Date: Fri, 5 Nov 2021 13:03:35 -0400 Message-ID: Subject: Re: ZFS operations hanging, but no visible errors? To: Chris Ross Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4Hm6KS4zgkz4Rcg X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[freebsd]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N That would, indeed, print things as they happen, so if nothing's happening,= ... I believe "sysctl kstat.zfs.misc.dbgmsg" would give you all currently logged ones. - Rich On Fri, Nov 5, 2021 at 1:00 PM Chris Ross wrote: > > > > > On Nov 5, 2021, at 12:39, Fabian Keil wr= ote: > > You can check with "procstat -kk" where the hanging > > processes are stuck. > > Running prostate -kk on the rsync that was hung, then killed, then SIGKIL= L=E2=80=99d shows: > > procstat -kk 35220 > PID TID COMM TDNAME KSTACK > 35220 102499 rsync - mi_switch+0xc1 _slee= p+0x1cb vm_wait_doms+0xe2 vm_wait_domain+0x51 vm_domain_alloc_fail+0x86 vm_= page_alloc_domain_after+0x7e uma_small_alloc+0x58 keg_alloc_slab+0xba zone_= import+0xee zone_alloc_item+0x6f abd_alloc_chunks+0x61 abd_alloc+0x102 arc_= hdr_alloc_abd+0xb0 arc_hdr_alloc+0x11e arc_read+0x4f4 dbuf_issue_final_pref= etch+0x108 dbuf_prefetch_impl+0x3d0 dmu_zfetch+0x558 > > > > > > The output of: > > dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, stringof(arg0= ))}' > > could be useful as well. > > I=E2=80=99ve run > > % sudo dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, stringof= (arg0))}=E2=80=99 > > But, it=E2=80=99s yielding no output. Is that normal because it outputs = information about operations (which aren=E2=80=99t happening), or should it= =E2=80=99s output have been immediate? > > - Chris From nobody Fri Nov 5 17:03:50 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 278341840213 for ; Fri, 5 Nov 2021 17:03:54 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm6Kj6JmGz4S7W for ; Fri, 5 Nov 2021 17:03:53 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 5 Nov 2021 17:03:52 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1005::15]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id f9b27734 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Fri, 5 Nov 2021 13:03:51 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: ZFS operations hanging, but no visible errors? Date: Fri, 5 Nov 2021 13:03:50 -0400 References: <20211105173935.7aa53269@fabiankeil.de> <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> To: freebsd-fs In-Reply-To: <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> Message-Id: X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4Hm6Kj6JmGz4S7W X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_FROM(0.00)[freebsd]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N > On Nov 5, 2021, at 12:59, Chris Ross wrote: >> The output of: >> dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, = stringof(arg0))}' >> could be useful as well. >=20 > I=E2=80=99ve run=20 >=20 > % sudo dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, = stringof(arg0))}=E2=80=99 >=20 > But, it=E2=80=99s yielding no output. Is that normal because it = outputs information about operations (which aren=E2=80=99t happening), = or should it=E2=80=99s output have been immediate? Ahh. I think it may not have been =E2=80=9Cno output=E2=80=9D as much = as =E2=80=9Chosed the machine=E2=80=9D. I eventually looked over at the = console which is now logging the following message about once per = second: ACPI Error: AE_NO_MEMORY, while evaluating GPE method [_L24] = (20201113/evgpe-708) And, none of the processes or shells are responsive. A =E2=80=9Cssh = server date=E2=80=9D establishes a TCP connection but gets no farther. - Chris= From nobody Fri Nov 5 17:12:39 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 791701844ED7 for ; Fri, 5 Nov 2021 17:12:43 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm6Wv1GGTz4WQD for ; Fri, 5 Nov 2021 17:12:43 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 5 Nov 2021 17:12:42 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1005::15]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 5f157f92 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 5 Nov 2021 13:12:40 -0400 (EDT) Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: ZFS operations hanging, but no visible errors? From: Chris Ross In-Reply-To: Date: Fri, 5 Nov 2021 13:12:39 -0400 Cc: freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: <0AABEDF8-665F-465F-9792-0B0BE6CAE97F@distal.com> References: <20211105173935.7aa53269@fabiankeil.de> <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> To: Rich X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4Hm6Wv1GGTz4WQD X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: N > On Nov 5, 2021, at 13:03, Rich wrote: >=20 > That would, indeed, print things as they happen, so if nothing's = happening, ... >=20 > I believe "sysctl kstat.zfs.misc.dbgmsg" would give you all currently > logged ones. Okay. Despite everything I had running being stuck I was able to log = into the console, and coincidentally or not, things have now recovered. = Well, the old commands/sessions didn=E2=80=99t, but I can log in again. = I can=E2=80=99t get to the tmux session it seems, but. I=E2=80=99m able to run that sysctl, which has a lot of data. The last = records all about two hours ago are: 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 1, ms_id 854, weight 780000000000001, = selected txg 1033574 (601067 ms ago), alloc_txg 1033313, loaded 5902891 = ms ago, max_size 2147475456 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 2, ms_id 88, weight 880000000000001, selected = txg 1033574 (601067 ms ago), alloc_txg 1020497, loaded 864138 ms ago, = max_size 17179869184 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 1, ms_id 859, weight 780000000000001, = selected txg 1033574 (601067 ms ago), alloc_txg 1033029, loaded 2201252 = ms ago, max_size 2147475456 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 1, ms_id 860, weight 780000000000001, = selected txg 1033574 (601067 ms ago), alloc_txg 1033229, loaded 3395548 = ms ago, max_size 2147303424 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 1, ms_id 863, weight 7c0000000000001, = selected txg 1033574 (601067 ms ago), alloc_txg 1033448, loaded 4046753 = ms ago, max_size 4294926336 Not sure if that helps=E2=80=A6. - Chris= From nobody Fri Nov 5 19:05:25 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9880C1846D9C for ; Fri, 5 Nov 2021 19:05:31 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm9232mVPz3pvH for ; Fri, 5 Nov 2021 19:05:31 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 5 Nov 2021 19:05:29 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1005::15]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id ba53751d (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Fri, 5 Nov 2021 15:05:28 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: ZFS operations hanging, but no visible errors? Date: Fri, 5 Nov 2021 15:05:25 -0400 References: <20211105173935.7aa53269@fabiankeil.de> <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> <0AABEDF8-665F-465F-9792-0B0BE6CAE97F@distal.com> To: freebsd-fs In-Reply-To: <0AABEDF8-665F-465F-9792-0B0BE6CAE97F@distal.com> Message-Id: <33253C33-9A8D-403B-A7E9-C511EA4ED34A@distal.com> X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4Hm9232mVPz3pvH X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: N > On Nov 5, 2021, at 13:12, Chris Ross wrote: >=20 > Okay. Despite everything I had running being stuck I was able to log = into the console, and coincidentally or not, things have now recovered. = Well, the old commands/sessions didn=E2=80=99t, but I can log in again. = I can=E2=80=99t get to the tmux session it seems, but. >=20 > I=E2=80=99m able to run that sysctl, which has a lot of data. The = last records all about two hours ago are: >=20 > 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 1, ms_id 854, weight 780000000000001, = selected txg 1033574 (601067 ms ago), alloc_txg 1033313, loaded 5902891 = ms ago, max_size 2147475456 > 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 2, ms_id 88, weight 880000000000001, selected = txg 1033574 (601067 ms ago), alloc_txg 1020497, loaded 864138 ms ago, = max_size 17179869184 > 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 1, ms_id 859, weight 780000000000001, = selected txg 1033574 (601067 ms ago), alloc_txg 1033029, loaded 2201252 = ms ago, max_size 2147475456 > 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 1, ms_id 860, weight 780000000000001, = selected txg 1033574 (601067 ms ago), alloc_txg 1033229, loaded 3395548 = ms ago, max_size 2147303424 > 1636125429 metaslab.c:2538:metaslab_unload(): metaslab_unload: txg = 1033689, spa tank, vdev_id 1, ms_id 863, weight 7c0000000000001, = selected txg 1033574 (601067 ms ago), alloc_txg 1033448, loaded 4046753 = ms ago, max_size 4294926336 >=20 > Not sure if that helps=E2=80=A6. Okay. Following up just to close out the =E2=80=9Cactive=E2=80=9D state = of the issue. It became unresponsive again moments after the above. The kernel was functional, as I was able to switch to multiple virtual consoles, but logging in only yielded a =E2=80=9CLast login=E2=80=9D line, then = nothing else. C/R=E2=80=99s were echoed on consoles, but nothing else happened. I issued a Ctrl-Alt-Delete, and it began stopping things, failed the 90 second watchdog timer and noted terminating shutdown abnormally. The kernel did eventually report =E2=80=9CAll buffers synced.=E2=80=9D = then nothing else. After about 10 minutes, I tried Ctrl-Alt-Delete again, and then = power-cycled the box. I=E2=80=99d still be interested in hearing any theories about what = happened, but I no longer have the device in this state to test. - Chris= From nobody Sun Nov 7 08:35:07 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E3B951844B98 for ; Sun, 7 Nov 2021 08:35:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hn6xm67whz3CBy; Sun, 7 Nov 2021 08:35:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 590642BF93; Sun, 7 Nov 2021 08:35:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Subject: Re: ZFS operations hanging, but no visible errors? To: Chris Ross , freebsd-fs References: <20211105173935.7aa53269@fabiankeil.de> <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> From: Andriy Gapon Message-ID: Date: Sun, 7 Nov 2021 10:35:07 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.14.0 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 In-Reply-To: <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam: Yes X-ThisMailContainsUnwantedMimeParts: N On 05/11/2021 18:59, Chris Ross wrote: > Running prostate -kk on the rsync that was hung, then killed, then SIGKILL’d shows: > > procstat -kk 35220 > PID TID COMM TDNAME KSTACK > 35220 102499 rsync - mi_switch+0xc1 _sleep+0x1cb vm_wait_doms+0xe2 vm_wait_domain+0x51 vm_domain_alloc_fail+0x86 vm_page_alloc_domain_after+0x7e uma_small_alloc+0x58 keg_alloc_slab+0xba zone_import+0xee zone_alloc_item+0x6f abd_alloc_chunks+0x61 abd_alloc+0x102 arc_hdr_alloc_abd+0xb0 arc_hdr_alloc+0x11e arc_read+0x4f4 dbuf_issue_final_prefetch+0x108 dbuf_prefetch_impl+0x3d0 dmu_zfetch+0x558 Looks like the system is out of memory. It seems that you already established that. -- Andriy Gapon From nobody Sun Nov 7 21:00:45 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8861F18405DC for ; Sun, 7 Nov 2021 21:00:47 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HnRV63gCFz51Rj for ; Sun, 7 Nov 2021 21:00:46 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6DBE325209 for ; Sun, 7 Nov 2021 21:00:45 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A7L0j9d095996 for ; Sun, 7 Nov 2021 21:00:45 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A7L0jMX095995 for fs@FreeBSD.org; Sun, 7 Nov 2021 21:00:45 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202111072100.1A7L0jMX095995@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 7 Nov 2021 21:00:45 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16363188454.f586.93418" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: Y --16363188454.f586.93418 Date: Sun, 7 Nov 2021 21:00:45 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 211491 | System hangs after "Uptime" on reboot with ZFS Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 240831 | zfs: Panic during snapshot on 12.1-STABLE r352648 Open | 243973 | [zfs] rollback segmentation fault Open | 244656 | zfs: resilver doesn't provide enough information Open | 244692 | gjournal: Does not support TRIM Open | 244899 | [PATCH] zfs: xattr on a symlink target > 136 caus Open | 251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W 8 problems total for which you should take action. --16363188454.f586.93418--