From nobody Mon Jan 22 01:21:48 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TJC9m6sdWz5899M for ; Mon, 22 Jan 2024 01:21:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TJC9m4Xy1z4jQ4 for ; Mon, 22 Jan 2024 01:21:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705886508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bi0/MZ8W2aM2ivp0gLop4qodosoqsd2q+5FdQbNpiK4=; b=EWoOMldMUwxwkwNekxzSmGAJtvjc0S+eLWkGiBWR0HMU3056Ll9Ou5lbRBjr7cVVycvoOM hVGpQvKgBq/kfbiQmfRoPzsMPzUunEIdlPHzgEq322fTOAm8vf+dv8xhZRylc+SGfN/xxQ 6c9Vtp1+GQxQ+YuOvVrFanBDNzz2/yREMMBw74u7mGKKTOqlfiMxOV9JfDPpbmb92QVTXw GORn728V+kqi+P1LUSK5Z5BQ3z6mRR/vRdtH9EFs64YpOyJHXw69z5qEI03pDy95cJXRid W6PUjpvCVHS6jT0F+oXOlizB6Iqy8coESR5Izzujysz211uvGh2LB3haE6AQHw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1705886508; a=rsa-sha256; cv=none; b=mXpaJVjzigOfKO5tb4zDkOQCNJHeGfzEuQcgdUEZoy1bUKpZ+Kcrs1ZE84E1Eh+awQeHnb uSEppSDWhJdTeuibuNE7F7un0L0iwOkbMfK4wbDxxgwjK64uZHFa6h49RBVnD1nHjyTr+R aYf78UEeDxMn/2JZ3octSP5/29GfKwMjBbPSXCvZF10EGSbkB5J/MBa9uYzSBSwagiS5oc vnhvayQf6CCOqO/0Px56HTvrPVF/+5yX7WXtKnNtRf1tPQBZhldGtUz33mTDitz4YYljmj oiICoVpeHAAveALVlC/cI1IiDpbszyI5vs0d2XaZpUyjvGEETwtl9I9R4gCMIA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TJC9m3cCFzhtV for ; Mon, 22 Jan 2024 01:21:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40M1Lmp0045954 for ; Mon, 22 Jan 2024 01:21:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40M1Lm9l045953 for fs@FreeBSD.org; Mon, 22 Jan 2024 01:21:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276420] zfs panic: VERIFY(zio->io_stage == ZIO_STAGE_VDEV_IO_START) failed Date: Mon, 22 Jan 2024 01:21:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: see_also Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276420 John F. Carr changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://github.com/openzfs/ | |zfs/issues/15801 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Jan 22 01:30:34 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TJCMv0WHfz589rD for ; Mon, 22 Jan 2024 01:30:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TJCMt6YSdz4kPw for ; Mon, 22 Jan 2024 01:30:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705887034; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WRTL02a6JzmMzQRS7dsfDnej2bxrizHuJ0sd0elgbNw=; b=nE+crKGCvZE6fN5UlV/eRe2IpRS5KcLFGJUMnKz4224y7FerEy4LR3Vew0tA1XDV9TAjnR CB2YKNtI/+fs5VRYLYJRtVZiJyvs4g4n5wf3VXtlp2KEFff6UTMjMPJtetP/gdxp1T9Ucp 2AaU+zcI09LlH8DEzfbzrmCm5vlNRFGM7H3fujfpBZHmPxpQBit5T1Oy170gtLRI9b3Xx0 E2qGhgohKp7DbF+iz4Bz5Q6jEvMVsqb+Fk0yWM4uRKvhl3fB6DH9AhL58ltB5MZ2xIRNvo 0TmsGpZawFFFGLy7s+3syAv7aWQ7BAJYqIS4c9bUqqI/l0pgyYnzrr1uGVUCvw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1705887034; a=rsa-sha256; cv=none; b=BvQEFMkStX7e3Srj3YeLhDJSBpX5Bj7Zhmfre55DO3L5xdqCNCjur7EMB55jMgnhYklk4X nMQfWBdykkQelwUtwD/pG6siWYj5NAxUGNlRERKRFngsdsz1Fa4RX84yjEN0zaH+dFWhsA wZe5GcEX5JnKuVQP3YGXTTajw6HQz8TJrns31wav8X243y9Vuv1KcUkndwoA38s2I+g+pT 0J2lM5FkQxvMrG/g10AeI1qWbe7SpxPHi9rWLfsfIdBn+Em0hNkotFuBlqtkYySofNA4fZ v5Iw5WRALmdGfRqEG1d9P06KZcVc0u/CW/N5Br/u6ZCIqLZJ/vVQkOsnxal1ew== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TJCMt4zSGzhtw for ; Mon, 22 Jan 2024 01:30:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40M1UYoi082389 for ; Mon, 22 Jan 2024 01:30:34 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40M1UYcB082388 for fs@FreeBSD.org; Mon, 22 Jan 2024 01:30:34 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed Date: Mon, 22 Jan 2024 01:30:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: see_also Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341 John F. Carr changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://github.com/openzfs/ | |zfs/issues/15802 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Jan 22 15:32:15 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TJZ3343wbz57dF7 for ; Mon, 22 Jan 2024 15:32:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TJZ332qHWz4LvS for ; Mon, 22 Jan 2024 15:32:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705937535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y0Jr2e9CBq5bivETiBKEVskSc24B45u/c8EJSKUzAhs=; b=fbcTcGmHtOkQFc6cYeqP04mVZgI7/DFaa2OHa0HAd+hVCMTckUes6ThHMaRPpTt8lb3bYv wU2StOmcL/VLH+ueIVxU5EJkW7W796CDqXyxxYO3DmPTJK5+jL1+7GXpdOiD44gG3xjipV v5fjWqsBAO9KFuiNDsP4mzQ0pXu0wduLh7C8OGfE6tU1pFDhB9s72uhmPTv9JRIrQN2yTj Y7LSF3IJuBAJ8lfofiTIsh9DuVmsQqTc7pI4+TJikhfzOB5FaKcTApjapEYkI+EDegBith 7qL372eUN1MZXU2BaGmkEbcAheCyJHwPKrxzgTsfcT8ZXJgoxSRP8lfaIAhUxg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1705937535; a=rsa-sha256; cv=none; b=J8xZknv4OIPsnBh2WAwy2uMufFGL2bCj7vg5gG9FsokldAc7G8CbPBqAy0Ynk5SYKqQ8lh gC5nq9cxxLYVFJTRj5dcPuHqG1Jy0k3K37O8411lYw+v11R4oAzMgvRqohlw1TwXFOLZgN 9u9f5r+FY2bnJRQMLVGYoLL7of7aU9pAl4un/BOhc/zn3aMyCLzAteBfQe+4a4i7wyEOzs IOi3taT56TVu2peahw8gaw92weakdpwj0MLcp7gbHKYYCDgEjQCZoTMdD7zZ0lsM0TYuhL 6R+/AgzUu51lpFYCmGhUc+ojR7i5ss04e20cUIM8Esp2nmbuZgtOrDFT4atRnQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TJZ331vSHz16pc for ; Mon, 22 Jan 2024 15:32:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40MFWFf6020837 for ; Mon, 22 Jan 2024 15:32:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40MFWFjb020831 for fs@FreeBSD.org; Mon, 22 Jan 2024 15:32:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed Date: Mon, 22 Jan 2024 15:32:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_severity Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341 John F. Carr changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Affects Only Me |Affects Some People --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Jan 23 17:43:39 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TKDwC2tbtz57mSZ for ; Tue, 23 Jan 2024 17:43:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TKDwB703yz46VS for ; Tue, 23 Jan 2024 17:43:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706031819; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fdUpMHVBzqUzTQSQGUiOf39uRG9drs061QWh87mBeqk=; b=jQ0vMYRngVSp3QHZ8yaUnudWkFCpJmBgAi6ZRpprkAhmCtcdWGl0E1nsdweBehbFYDnkLV Ht+lvglIyoNsvUNLY/OVSBiCQmqPgxrOnpcF6QBYNyvKsnYDX3T4eYX+x6gXSc74tNed9+ 12cVckIibroYK5xgEj95TaUHzH0d50QGMcEynVac+/St2C5mzmT4CeLeGDMZNShdJJisgI J9eqc5LD2Q/eE2MhFeYGJG0766ZYZfXfVYL6oqbAqHtfSZsmWFFytkng4GU+X8shG8Oe0l K/JYhu4BFEaZZKFqiD1FTkS3UF5Obew20RZLfxGWp97zre6/3RxvzrdGbAf8Pg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706031819; a=rsa-sha256; cv=none; b=hUXDH+VdsBEhsrGLlaltwDZdaBEmoCJ57Y7n0vhU9XEGHBjgwJCn5JBYTcRWlgn9ouccoP kZiDXPxuNWZnP+QgJzfbbudwQs25LlsHRhZPph89wSsy+89z6A7hLC5FG/lUxAOK8x7XwW i0mgvhYtA8DyMljb1G2CZo4kMSbxFxR2X4gQjyG2uijIM70c7RyLSr3M4fqdUehgKKtph7 RId8hb4psf/DMp4cC+PiWrBv1vT1PkJ7dR7fK0AvYpjj/7OOcMXEYNWA8BV7TgTKKhwBeU gjqENv/BkAW1tIiXtPxhUbE4VWUrImuqIaN5kVPmdlGdkb0tjm22GeYMsLUg4g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TKDwB64TgzgYw for ; Tue, 23 Jan 2024 17:43:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40NHhc80086632 for ; Tue, 23 Jan 2024 17:43:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40NHhcvE086631 for fs@FreeBSD.org; Tue, 23 Jan 2024 17:43:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 270988] vfs: Fix 'emptydir' mount option and vfs_emptydir() Date: Tue, 23 Jan 2024 17:43:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: olce@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: olce@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270988 Olivier Certner changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|fs@FreeBSD.org |olce@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Jan 24 10:39:42 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TKgSb5brqz581C5 for ; Wed, 24 Jan 2024 10:39:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TKgSb4Wg7z4DmS for ; Wed, 24 Jan 2024 10:39:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706092783; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YjSUV/pbrgQgx5JRU2L8dxKBmvSY2Iv2GlaL2f1Qy/0=; b=aJ/7vMKt8FR0aJVmdvMfwXAdNH226k8muQXhDEd1GuOt39oF6XLeVNF1ZMKV3fk0mS/PgL 6tdYAfG7n6HgmLxRRrjmplqgl7hKQevNMn2dMxyqm4PHQ7GUcHS1kKFKPtKoKpMUO/m10X sY9kh0NxFwlkQDHPuns6zK4KUJjYOzdtFeKyVN8bAwnMigV0FAKQLVfb3jU9YGaflXEMjK tIes2r0S1mBzTMBtsruLcYvwhgZQbBLN/xWXNP6XQyY2FCwnGazXSSMb/N9TSQNzniRv5e zaB4i5JxdlnjTBgUhz0KhoCsx5+grHOTrJF+GIG/kGi1h3qKBN/RXoUnKNWATg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706092783; a=rsa-sha256; cv=none; b=IIjVxo7oIT2rEoaYT89t2oJ9r/TkHwTyxiz+Ckh29qjfk0bQE/11lsggBQZYqc9EECaLsd WvXtAxxB3o8BcgU27daTQ0QEUYtShCGQcBf0iDJh6N3iXQTVZ5oxWeT0wCDpFpsLKY34Tq H/AUJ7xfgG8UcsLnskQngeOFWijv//CslAMP3uL8753uADXdd5d7jM2tQwQzyZqj65Xs80 cV3MxeCwVOeipwFZF9naS8bkNjrtLra9NRo2gkf7PjyRn8I10L4TVkp1cGFp8uwWbLB7o+ ssuNNrkMF9+PF9GRFQIDQzWcWIci7gYqDbOBXjQh6beNCB9kP1/ORE8E/OVIAA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TKgSb3d9Lz19k7 for ; Wed, 24 Jan 2024 10:39:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40OAdhVo049778 for ; Wed, 24 Jan 2024 10:39:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40OAdhUE049777 for fs@FreeBSD.org; Wed, 24 Jan 2024 10:39:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Wed, 24 Jan 2024 10:39:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #35 from Seigo Tanimura --- Created attachment 247921 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D247921&action= =3Dedit The result charts for Comment 35. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Jan 24 10:47:34 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TKgdg3kktz581H9 for ; Wed, 24 Jan 2024 10:47:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TKgdg2ZLYz4FhX for ; Wed, 24 Jan 2024 10:47:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706093255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TYof8Rla+7ykZQHHJuD7V/MoabI3YwpsI+1wDyNf69k=; b=Lc7tWHa27oTcyywfXTd85ERu4ILW+7a4l3Tz9hY8xAM/nBQmlOcwPPM3gUXuxUWj7FNG1m /p1LNPgx5o9Y9Xf4foZ1VmwxB3XCpV2gbkpWoklHFgAQsdVTrRpQ9J/1j69jLRltsOuIOK p4Z38Mxti+wkkMES2m+3MZi6io8qnI7rAy3kglpWEpbyLGY9x+rfYtlPfDbb/ckI6kLAOi z0uu37TJt7buCcuaHPsbdg1bTZK3hu4nTP2l17UX4ZvDpK7nj1x+y1LXsqW3bNTGRf6v1c 97AhtjWmFV+g1CmsU/DFXGeT950ysNcWM+NtcAKGlRi9QydQleYWu3LY50Gcuw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706093255; a=rsa-sha256; cv=none; b=Tt1AaToGYeSh7FQxMbYEVq9DH4sOSZhYERsh41MShKpp0kMaKQhRxUeQ5GN9Pxgf5Ggvur OtBHWc67eKBTgmdlY+m/5D1Y+APDHcoiHHyMXQpz9taW/RUL3QWl+xe73eU3bwC/greyOd dihtw6g+jLQFyP/dqaSB5fa/Rn3vksliuhB7ooKVcAoZf8+OvtadZrs9Q915Pov/T8Qvij ekKjIDP8smgEnN1JPwpCzbBP0aerH44mEgI+JTaTy+wBBtqg4PJVS0isw9I81BawgN8BM6 SSYTsYtj66redreXTSe6GXjiXjCQIV7QIBB+n9iXw2iKJ5m4Uk0nxw9IbfJs3g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TKgdg1dmKz19MD for ; Wed, 24 Jan 2024 10:47:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40OAlZmu091539 for ; Wed, 24 Jan 2024 10:47:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40OAlZvh091537 for fs@FreeBSD.org; Wed, 24 Jan 2024 10:47:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Wed, 24 Jan 2024 10:47:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #36 from Seigo Tanimura --- (In reply to Thomas Mueller from comment #34) I have backported the fix to stable/13 (13.3-PRERELEASE) and tested poudriere-bulk(8). The fix has also been applied to the main and stable/14 branches without any changes. Thomas, would you mind testing the backported fix to see if poudriere's bui= ld time changes in any way? * Sources on GitHub: - Repo - https://github.com/altimeter-130ft/freebsd-freebsd-src - Branches - main (Current) - Fix only - topic-openzfs-arc_prune-regulation-fix - Fix and counters - topic-openzfs-arc_prune-regulation-counters - No changes from the fix on 14.0.0-RELEASE-p2. - stable/14 (14-STABLE) - Fix only - stable/14-topic-openzfs-arc_prune-regulation-fix - Fix and counters - release/14.0.0/release-14_0_0-p2-topic-openzfs-arc_prune-regulation-counters - No changes from the fix on 14.0.0-RELEASE-p2. - releng/14.0 (14.0-RELEASE) - Fix only - release/14.0.0/release-14_0_0-p2-topic-openzfs-arc_prune-regulation= -fix - Fix and counters - release/14.0.0/release-14_0_0-p2-topic-openzfs-arc_prune-regulation-counters - The original fix branches. - stable/13 (13-STABLE / 13.3-PRERELEASE) - Fix only - stable/13-topic-openzfs-arc_prune-regulation-fix - Fix and counters - stable/13-topic-openzfs-arc_prune-regulation-counters - Backported changes - A fix equivalent to FreeBSD-EN-23:18.openzfs. - The ARC pruning task pileup is avoided by a single flag and the atomic operations on it. - Seigo's fix. - The ZFS vnode accounting, including the counters. - The ARC pruning regulation. - The improvement on vnlru_free_impl() - Changes not backported - Seigo's fix. - The counters regarding to the autotuning of ZFS ARC meta, the balancing parameter of the ARC data and metadata. - Those counters have changed significantly between 13-STABLE and 14-STABLE. * Test results Test Summary: - Branch: stable/13-topic-openzfs-arc_prune-regulation-counters - Date: 24 Jan 2024 00:10Z - 24 Jan 2024 05:59Z - Build time: 05:48:30 (367 pkgs / hr) - Failed port(s): 2 - Skipped port(s): 2 - Setup - sysctl(3) - vfs.zfs.arc_max: 4294967296 - 4GB. - vfs.zfs.arc.dnode_limit=3D8080000000 - 2.5 * (vfs.vnode.param.limit) * sizeof(dnode_t) - 2.5: experimental average dnodes per znode (2.0) + margin (0.5) - poudriere-bulk(8) - USE_TMPFS=3D"wrkdir data localbase" Result Chart Archive: (poudriere-bulk-13_3_prerelease-2024-01-24_09h10m00s.= 7z, Attachment #247921) - zfs-znodes-and-dnodes.png - The counts of the ZFS znodes and dnodes. - zfs-arc-pruning-regulation.png - The counts of the ARC prune triggers by ZFS and the skips by the fix. - zfs-dnodes-and-freeing-activity.png - The freeing activity of the ZFS znodes and dnodes. - vnode-free-calls.png - The calls to the ZFS vnode freeing functions. * Findings and Analysis - The build time was shorter than 14.0-RELEASE because emulators/mame, star= ted in 4.5 hours, benefitted from ccache and completed in just 10 minutes. That does not work on 14.0-RELEASE and all sources have to be rebuilt. - If the emulators/mame build did not use ccache, its build would take ~2= .5 hours and the whole poudriere-bulk(8) would complete in ~7 hours. This is = the same time as 14.0-RELEASE. - No ARC pruning happened during poudriere-bulk(8). - The only one pruning happened while settling down the system before poudriere-bulk(8). - On OpenZFS 2.1, the ARC pruning is not triggered by the excess unevicta= ble size in the ARC. - Above works on OpenZFS 2.2 in 14-STABLE. - Only the overcommitted dnodes and metadata size trigger the ARC on OpenZFS 2.1. - vfs.zfs.arc.dnode_limit in my setup effectively disabled the ARC prunin= g on OpenZFS 2.1. - Maybe this should be reverted to the default and retested. - The zfskern{arc_evict} thread used the CPU up to 100% in the final ~1 hou= r of the build. - The reason is not clear. - There were no significant affects to the system. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Jan 24 11:44:14 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TKhv3207tz586W4 for ; Wed, 24 Jan 2024 11:44:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TKhv30xkPz4Lw9 for ; Wed, 24 Jan 2024 11:44:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706096655; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a6NMgUKosfapHiwQvrzBZIso8UJc5H1xtDBZpqePkmI=; b=mQQSSXh8m5FcH+Tf8scOduXvF10lg3xKCr1BSrYsmLHGC39Z2HZ4Exlvw2t6/rLNLusKHz Y/XCnUy0c0UNnrcLLopeOtszhGVhpSOQlese7yupKgR+pZ4PXp/KRWvwmfNW++Qgw11blW Lh/XbdVzkpRzDlnrhILyqmxSqeD3GuPhraOd0fjWEHODnr0wHLSCXi06joOc5Yw1zusty3 4iDocqQTo2SJLV/SBkM9q7a8ToIOosFWHeWMnp9xHFpKJFHM6oU8DLFm6DKr0cCkYG2Wp7 lJfsxgKNxqifLNEONsHrjQmYRpHkY1tdkUQoeepCZy2ow5a8fdpj0o5mXF04VA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706096655; a=rsa-sha256; cv=none; b=ocakMJJ7gxv9Tb4aS/DrbvtNI/9f2r5LjNobQ0kuKtlaZgUfeNY6RXaRIzhlw3l+nYoAUH JkZ5T2PIxMMGllgCtaNFCUMPHDsbyu2QF0OZgABQvzebv9Ulhzn4K+sJcHOOjsLtcNOBFT qHESDbG6X0pk974HL3d2QuAjXg++uITelcsWv3Ou2QaOTDCqrbQ3ZO/oOsNxeUPXGoodTw UofWP5JRSvbC5nJXfOIIQbra5R/9oBHwpmyRTUaPB1Jd4mxnEU6Jd0Me9j7i3+OfZrJIAm eidBy+fYr37nnVXEwkB2kVcpvfEHyK4yR88HaDASIOBnrR0/Jk997SRZAcKUCA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TKhv301kYz1C6m for ; Wed, 24 Jan 2024 11:44:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40OBiEsU082332 for ; Wed, 24 Jan 2024 11:44:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40OBiEUF082330 for fs@FreeBSD.org; Wed, 24 Jan 2024 11:44:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Wed, 24 Jan 2024 11:44:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #37 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #36) > Thomas, would you mind testing the backported fix to see if poudriere's b= uild time changes in any way? Please hold this test until the further notice. There is a bug in the backported fix of FreeBSD-EN-23:18.openzfs that blocks the ARC pruning when it should run. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Jan 24 12:43:14 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TKkC62pPjz58Bpt for ; Wed, 24 Jan 2024 12:43:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TKkC60588z4TBZ for ; Wed, 24 Jan 2024 12:43:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706100194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SUGvFrk4TD+ZVhJ0/aRR4LLmbbiiJqZamOpE5qNTxLM=; b=uiwhTiR8xO2FGsrEpOsUNBlOgSvOJdp1Vdeza6fEYcyNuVR3tr9iNq9r6YKz/ACsTTRCO9 N7YPRfIQzA/PEXeWaXv5q8siO+qwzpKysy+iRoYZwOSe+J7ee+EYoqc1Y+AijOLf0mYgMk pR/E7HIYNNR2286VSbVLPrKFjC/Nestb9jq5Xgs+cBRvpPFjNDlb4lL+Ni7Q2UilB5MTTL duxDYrwXeI/NsL3FvAW5DLzIPPptBIbmcjfAhjc7s+FPB8VahqP8farniYQerh43VukXR8 ohXZjPPPJSDXmYfarTK4QS8GJ/K5kjGbZ8+KDxWMtMk+Lqd2oDaj8bDEQWaqrg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706100194; a=rsa-sha256; cv=none; b=GZbpHtMURVik1dsGzjTtNGlpqQKfM5wwXqop4A4YBk8M+Lw65gVW3WzxnRXsvSKWBIr2V3 l3CL8wa9vcganu8U3fJZE/gZXulgZkpyTFgVMD8XNYs0sF2EP5yWfDGHXZetgeycyC7M9U F/BLLgV8IUEnMiF/cZuI5rBZs8pFrupeRD3NqCm+Kp2GLd3ufsO93jhicYo22OyU19rwlu DDDsFdFY6Ri4k5q7cfRHANvh056uaGUgi2oIEyxRzWV/K5kBqoslV6ciiRrbMd1VRSq3N9 WseHCJozYvFaJ1/ddpoC3+K3nhb56iPuIP4EAVDh8xnQ62NnNOyLG5mE+DOuxA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TKkC56G0NzDw0 for ; Wed, 24 Jan 2024 12:43:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40OChDAf096687 for ; Wed, 24 Jan 2024 12:43:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40OChD69096686 for fs@FreeBSD.org; Wed, 24 Jan 2024 12:43:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Wed, 24 Jan 2024 12:43:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #38 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #37) The fix has been confirmed, the branches for stable/13 are now ready for testing. My apology for the trouble. The flag for the running ARC pruning is now cleared by atomic_store_rel_int= ().=20 atomic_set_rel_int() was called by mistake, which did not clear the flag. The fixed result will be shared as soon as the build test finishes on my si= de. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Jan 25 03:47:08 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TL6G532tNz57byJ for ; Thu, 25 Jan 2024 03:47:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TL6G50llcz4Llx for ; Thu, 25 Jan 2024 03:47:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706154429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1i1LOS2jdVXeps9HKLcG5XJ05mX9lQMpK4CbsNQJroA=; b=Yhqng2m42l3joZLXRKkPz8oK9vwfhxBVxoHLvapfBGKEUKecc27iOicexUjgN32vLKN4QK alzg7nOoq1c6ntwUMHx68NB9wl/7qFcPNNZq+TZPxzbeYEmBGrY7wsJOTx9QRGpDDg3s+v FhboEcBimPHq2LCm7OJ9SvEzsKCMbOh8e5mTxJCsOfMEyIS4fkg8RLMQu0cGLTEY/sc1cm MA2U5FStQpIByZfyYBZW1r3bu5qsTT/0FRC38nBfCtD/zOBftCtPs3UzQywNQ7R2RtuPbv Bo3aWvqeljqLeogA4Km+tkhw68O1G5i5Bhmpc2/AMOxQE678pdCi8QJ4Q5thKA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706154429; a=rsa-sha256; cv=none; b=wTRxUdRByfzOeSUqpcTts3tbCjPq80/AxyDVr/Aj842uUorbUehsVOsQS6Xi1J79y0XTCV I+eOaqZruR2jGZDemLeMguywjP0m6MtvaZp7DABOAqSqsQNfh+zrMjcB6u0LcIHvcKGVsp 06fg7DPeryhQKMLeH/cfH8419TIvf2sYjJovK7/l7JL/3E7M4qPVhz8WNgUc73ztTae+xZ YWdYPh96OHkGUREc2z2NNvq6vJJWI/645obCNXmbtPnlTqkhHSgtvip/bPiqVmben4JnH8 vNvC1W9kiTIxPlltlhl7QJMpkq094thq9vWqKqmlh8lj5H0rq6l9X/KW8syoVA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TL6G46t9rzj05 for ; Thu, 25 Jan 2024 03:47:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40P3l8rS039301 for ; Thu, 25 Jan 2024 03:47:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40P3l8Lw039300 for fs@FreeBSD.org; Thu, 25 Jan 2024 03:47:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Thu, 25 Jan 2024 03:47:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #39 from Seigo Tanimura --- Created attachment 247941 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D247941&action= =3Dedit The result charts for Comment 40. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Jan 25 03:50:23 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TL6Kq2hHfz57c15 for ; Thu, 25 Jan 2024 03:50:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TL6Kq15Gtz4MVq for ; Thu, 25 Jan 2024 03:50:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706154623; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mNDpXybPvMY0AGwbxjtMtsZcRJWDgg8l4nNdCKffPIc=; b=OK30YU1b29KiLAm7Sby2ziGJJIq5kzO2/ZLQyCtc1K2IMMBtVX+ThfKqsjzsN2fA5LZ7Ee I485VnVgIOm+bqOG8BoLkpvShEZVTurIJEVaLxfX1pI/kXFPW1cNDBCZ3SobOvuvRs/w6R uP3JpfQWQ9K0hfloR5SWAtIJt6B4xzBVjoWMQELdXIc5cAuMs89Wtj0IwC5X1NXfynibHK tnrypvkCm75eJoLUbL/msWshtpfhsqVy93CjyExXyO4zw6BLHXAfi/yb4iqJosOQLjQPwC kLbucwNfWxjNXTS0ShhNOM3wuQDmRCtNQ3gA1aiW5+GZGmWTm88gkiLbz1iPVg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706154623; a=rsa-sha256; cv=none; b=EmiNv1lwPilLvLIVYU3bhQ3Yw4u8Lklw83huEGNfysvA2bqAAr+exzdYYY/jN2gfNI6x8v KBfL4cYcPRRsYkg4ffF99nFxtJsxzILh9qCxzSdiRDtzccjpoxOPn9Y/OiiwNM7WpNGxOW GypZkK8tuunown9QPMIqP5fK5StW+VVEPj7VLJEKoy6xkVCvy7aav4Rg6vvZH/mpqP6FpB mK0So7XxnvYT1QVRgOj0tyFp14v4SyvvzW9A1pELVPa4mjY7rmqRDhJPFj3NGd2i3ctCU8 ye6+k/gK/c04jXzWIrZA0t4wHr1AnFPm+jKD6au3iuz5Jb0DGS2miQoAAK3F5A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TL6Kq0684zj3V for ; Thu, 25 Jan 2024 03:50:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40P3oMOW049642 for ; Thu, 25 Jan 2024 03:50:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40P3oMcJ049641 for fs@FreeBSD.org; Thu, 25 Jan 2024 03:50:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Thu, 25 Jan 2024 03:50:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #40 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #38) The results of the fixed stable/13 (13.3-PRERELEASE) branch is now ready to share. Thomas, could you please reproduce the build with this kernel and see if the build time improves? If so, I will work on merging the fix. Thanks in advance. * Sources on GitHub: The same as comment #38. * Test results Test Summary: - Branch and commit: stable/13-topic-openzfs-arc_prune-regulation-counters, ef898378041a1c67cd102e8e5eaca123a543029c - Date: 24 Jan 2024 12:10Z - 24 Jan 2024 18:02Z - Build time: 05:51:26 (363 pkgs / hr) - Failed port(s): 2 - Skipped port(s): 2 - Setup - sysctl(3) - vfs.zfs.arc_max: 4294967296 - 4GB. - vfs.zfs.arc.dnode_limit: 0 (default) - kstat.zfs.misc.arcstats.arc_dnode_limit: 322122547 (calculated automatically) - poudriere-bulk(8) - USE_TMPFS=3D"wrkdir data localbase" Result Chart Archive: (poudriere-bulk-13_3_prerelease-2024-01-24_21h20m00s.= 7z, Attachment #247941) - zfs-znodes-and-dnodes.png - The counts of the ZFS znodes and dnodes. - zfs-arc-pruning-regulation.png - The counts of the ARC prune triggers by ZFS and the skips by the fix. - zfs-dnodes-and-freeing-activity.png - The freeing activity of the ZFS znodes and dnodes. - vnode-free-calls.png - The calls to the ZFS vnode freeing functions. * Findings and Analysis - The ARC pruning has worked in the same way as 14.0-RELEASE. - The prunable znodes were pruned down to less than 10% of the dnodes. - The behaviour after 18:10Z was due to the nightly cron job started at 18:01Z. - The build time was virtually the same as comment #38. - Also virtually the same as 14.0-RELEASE. - The zfskern{arc_evict} thread used the CPU up to 100% in the final ~1 hou= r of the build. - The reason is not clear. - There were no significant affects to the system. - zfskern{arc_evict} stopped running upon completing the build. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Jan 25 06:31:53 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TL9wB5mcJz57sTk for ; Thu, 25 Jan 2024 06:31:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TL9wB4jnWz4gTy for ; Thu, 25 Jan 2024 06:31:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706164314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OGLcbyqxBK8+gVLZIpvse7S+9X7qClUIhe+QMJh9r70=; b=KcWKHpk6AJaaWhlpxdUDWRLXqhG0NLQ0qRWNDI0Y99jn6XnSI2SoMD7T7UxHaENJdLKRVV 85OJgc8cgL/9Yz2CehGRIVedrKDoxIQb4HaMxVn7ZxLNTyPZ7kAN9U+9uyv5YQbHOW95WA K5zmx81AYHTpu5u739j5i6ugQFOxlgnVb1PoUBZGduJRCwo6nMS+Uz2LyO6vDxLBw+bVub CBvEpYbXNWlpQr8P+LIpGxrdYtXDrQu91B2IVYrGVIahNU9c/nnhr8zL/agChWAnJljhpE wtCGJFyGsKgRPPn82W1WpQ/f5eI8NaSy9WETRrBU3+f4TqHUU/K2Yd53mZOQNg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706164314; a=rsa-sha256; cv=none; b=UavywMnxaoBykdm3mX7iv4/DAcVqW2atpOtioOZbmb2fRuU4vth0G0hEahVrtDtgXrx2PA lNDc8n1n9B6tCkcXye3FiQJXkwwfptlS1GEgmn4w3p2fSPGLqdEfFFM9r+tdywGNuTkIYD knt0ptIjee/PRqYXor27PJWvTRbHWw+tJKfMseaWm4A4NOVZSPvm5pIaotW4PnU+nujcGV Ep6hpUJ3dms8ag+vQg105dnxKYnlQfMgJjtbUh+3kzVfoh96i6/MiHFUjpJ7QB540fF9WN jOPjQpvkoeV4xRH8xGzHoYBdcQGe8sQiQPO0/WxSWY07/fbctz+SUN7Vv6WUWw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TL9wB3qYDzn5x for ; Thu, 25 Jan 2024 06:31:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40P6VssV081335 for ; Thu, 25 Jan 2024 06:31:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40P6Vsfe081334 for fs@FreeBSD.org; Thu, 25 Jan 2024 06:31:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Thu, 25 Jan 2024 06:31:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: thmu7@freenet.de X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #41 from Thomas Mueller --- I added changes from commits a57d4914c11f4bc6d5ed33e146f2664315f64701 4efe36b1428a9956a049fc5fc5f19d4a001d51bf=20 from your stable/13-topic-openzfs-arc_prune-regulation-fix branch to my 13-STABLE kernel and things look much better now. I have no exact comparable data though, but from observation - while poudriere is running, kernel{arc_prune} could not be=20 observed consuming 100% CPU or any other unusually high=20 values for longer amounts of time - while poudriere is running, parallel heavy disk I/O=20 (e.g. by periodic daily/weekly) will no longer slow the=20 system down - no spurious build failures ("bad file descriptor") observed for now - after a poudriere run, "pkg upgrade" of large packages, (for example texlive-texmf) no longer triggers high CPU usage of kernel{arc_prune} and also does no longer take unusually long time - poudriere pkg/hour went from 5-20 up to 55 (no exact comparable builds), just an observation from previous samples and one build with ~400 packages on the patched=20 system - no negative impact of poudriere (2 jails) running at idle=20 priority on productive work or video playback --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Jan 26 21:45:30 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TMB7t1qr6z58v4w for ; Fri, 26 Jan 2024 21:45:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TMB7t0hc3z4XyC for ; Fri, 26 Jan 2024 21:45:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706305530; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OZl6QLqfPWpYC0pbEVJs+biLL1A7mNh/QrZ6z3S+b1Q=; b=XsZobahciZOfU8GjvndwSom1OV8bSiwhM+ljpgGrXgcf5rp+MWAzGJ8DJRvyQd3MAQHY9R 3MQQUQ+1mAW3qU8mWAM89fQ9VLyc9ZwSKydPYxJLLLQEHuNwWY2HnrDE7yhA2p+z131JYH Eg8b0FTp3TSvP3LxXMe1Uj4B51AWn3A9jQli+1O/RN80pnvEOYuuFGT1rDRv7iGcLyWK4s fMHbfI8rXwq7/XxjPYLckkk7WHtwjqYPIay/VhCGMteiwofN7NVg8ZLOcTjYSijGRosFFt inYvkXWFEYqcpzUF/fIWdR2mO4wRdSUyYdOSCnwF4mgWnpiBTgnkMvXXStbpnQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706305530; a=rsa-sha256; cv=none; b=TkeT180U+AQESolvvfpkWYrgKJ4A8gowT6qlAA0K7MpuHqmpz0iBf0gGIGMNNeRxXuhPnA A8zzklLMdB+pEp7skqzfMQFuJOoEJzi63nRuA9AZQIO+HQsxTgfYHBHQ/7AqD2oQJalqfR qQySjVNx0fAOGIqDmUsQMVFY5Gi7hILA+DMeCv19mshGQZ7wMRfQpVAjFw17dDki4G0wDY bxMOK5opJyu7w3jN9p/7SDMN9n91WdF1tDlc7mTBjoPk89+zBoEIgu5V5mygclVpL4RzHw uhbFs7Wk6cAZjgJKcfpIQ2sBOFrrGzAGxZhVzl19qyCgtfAQLdPM7BvVsmxcag== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TMB7s6nWbzxTp for ; Fri, 26 Jan 2024 21:45:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40QLjTnr047293 for ; Fri, 26 Jan 2024 21:45:29 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40QLjT21047292 for fs@FreeBSD.org; Fri, 26 Jan 2024 21:45:29 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276646] failmode=wait bad idea for zroot Date: Fri, 26 Jan 2024 21:45:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276646 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --- Comment #1 from Mark Linimon --- ^Triage: do you happen to know where this default is set? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Jan 27 00:34:54 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TMFvL3BhXz598rb for ; Sat, 27 Jan 2024 00:34:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TMFvL27kXz4skM for ; Sat, 27 Jan 2024 00:34:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706315694; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e/12/3W7ZzWJa168ml3LVcepIosKLLBxmXuwFUXhO/E=; b=s1XsSi1dgVpNrFXzjpEPAv9ACdwtVAVKRjQdvf5tLelzCVKWKYUK2UiRyfSNp/RRVHwaKi 73zxPHeSDppGvhuyD1FUz+1i1Yv7eia9+795fV5W+f/R1R8YG7ARhH9DK+L2mu1OMnI6CM 88iF7SqnHHtAJHH5FU4LymkFomWmvqXWH7FVkS5nYbGjC4lbciTXX06ZkyRJ4mtfSH3mX4 VYfzQ3wm1GjwJxPa0Kcf+nfww/3klB35/8bCyptcT1M+F5W5hZxDVokZXfXQjKTPg9yeMg YQcPl4n6VsYuR9ufgfidLJFRvYqGPK6nafzXl3WKxBm1iJLerNj/DMU9g7yI9A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706315694; a=rsa-sha256; cv=none; b=tyyKo2crqXaUm11/uD2+61xXgySjPQgxn5h6jq06fMIC+ibkEfzoINpLzb4/0KuVUyQggj wGak4EpXPvYAp/SSR3g+g+vo7KsRbwtdtUnPcIxcgfz0g2nCuK2pI1oJb5XKCDcDLXDNsR 7alGkclBQCorgGvfpdmxKpE1Cj3ZwpgKX+74gYvxL7XaP2guOTL2sitO+UORM/j2D8/I+p nblasp/4VEyEuJiAwkMxHGY2nwq+zybHhPkyMBdCIP50u1lFLG5ZdZZrt25MzAQ78ZnQHR EkWSclOVKinPK6Qkg+mxdw62yJSsqD3/CGMeAj5+Pu3LV9ybbeCVmNMJZWBAcQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TMFvL1CRsz13Vl for ; Sat, 27 Jan 2024 00:34:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40R0YsmH088982 for ; Sat, 27 Jan 2024 00:34:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40R0Ysvq088981 for fs@FreeBSD.org; Sat, 27 Jan 2024 00:34:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276646] failmode=wait bad idea for zroot Date: Sat, 27 Jan 2024 00:34:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276646 Mark Millard changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marklmi26-fbsd@yahoo.com --- Comment #2 from Mark Millard --- Well, https://openzfs.github.io/openzfs-docs/man/v2.2/7/zpoolprops.7.html (for example) documents wait as the openzfs default for all contexts. bsdinstall and the like could possibly be explicit for creation of known-boot-pools instead of depending on the default. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Jan 28 21:00:13 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TNP2k1P7Nz59Hcp for ; Sun, 28 Jan 2024 21:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TNP2j5rnqz4Bp7 for ; Sun, 28 Jan 2024 21:00:13 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706475613; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=F6RYqcPTz6PgyK/djhw8yaD7bMTcQmNRsCbx+gaKB0s=; b=GOYEdjxQoPxm4nmEW5WiKyQRQwrQTttnoklvP8pMelQLOZav5cIqjAT48OtCjeXv2I7KEU inAeNictEBrEZ9raxJhnyM+gP7j/4hPA+jre4TgNqEY14839Orl4hY6jFJIylbz3Ee5cp0 nlPXC7GXvI30Ni43zXwwewJl6iQ5TIXwDp6+U6jOnQ+MObiqY8iTmnHCK8j0fl2BTFIO+k KEJE80jvFTNxgPrlHQZQXOdNuKfte5Bh/tqz3LlOPkCxS+gxA+Mu6D/inU/kfQmzuPPnwF 41AuSyBl7rcRmk2TDBWRPhWJ12i4335Gelu0pLw0visJE9K4MfDRiyZyENMOkA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706475613; a=rsa-sha256; cv=none; b=PJutBiXYFL6f2B/56FFkoGDXC7jUi9fynvRP/jc99+jKJp26noohE0kAxzF34dtMbOR3eB /zgWyq3cOedjiYJ1GXwkAsioA+cW76PqfJbmJWx11uKRw6WdCX/FKA87pfl2145Od+x+Si znUUc4voDOFjtAFC6oOpakc2+zxKnkD/yF6e1cKJOqEoQJd84v9siqiqhi4V71x8ce1c1a raEXfB/M1y+y7xmmkmsh/tE/N/o5VuCPupED2AcoEnfmx2sSyUvRA+O09Yvx5Q/PeWD6Mm uD7V5dxW4+93kJP8j5XQKzjSlV8fU50RPc5n5s7XTEx2hjHvRc9D90gn7oi4KQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TNP2j4wtrzNMQ for ; Sun, 28 Jan 2024 21:00:13 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40SL0DG5072865 for ; Sun, 28 Jan 2024 21:00:13 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40SL0DTd072864 for fs@FreeBSD.org; Sun, 28 Jan 2024 21:00:13 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202401282100.40SL0DTd072864@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 28 Jan 2024 21:00:13 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17064756135.dEF635.69338" Content-Transfer-Encoding: 7bit --17064756135.dEF635.69338 Date: Sun, 28 Jan 2024 21:00:13 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 231794 | zfs: Panic due to ARC related KVA memory exhausti Open | 251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W Open | 264139 | ata: NCQ_DSM_TRIM trim method for Seagate IronWol Open | 266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy Open | 268162 | zfskeys_enable: each successful load of a key is Open | 269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit Open | 271384 | zfs_load is not suitably documented Open | 226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 = 8 problems total for which you should take action. --17064756135.dEF635.69338 Date: Sun, 28 Jan 2024 21:00:13 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    231794 | zfs: Panic due to ARC related KVA memory exhausti
Open        |    251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W
Open        |    264139 | ata: NCQ_DSM_TRIM trim method for Seagate IronWol
Open        |    266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy 
Open        |    268162 | zfskeys_enable: each successful load of a key is 
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit
Open        |    271384 | zfs_load is not suitably documented
Open        |    226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =

8 problems total for which you should take action.
--17064756135.dEF635.69338--