From nobody Tue Nov 14 12:58:23 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SV5vM5lxfz50sps; Tue, 14 Nov 2023 12:58:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SV5vM5JByz3GR1; Tue, 14 Nov 2023 12:58:23 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1699966703; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0z8Fp4YItAuUG7y2aWkiau/D2pQEI+WL5ZVbPRSgM9o=; b=ul/aTIMbHhfTJf8WCCTQYKMHd26XIP0Oy+xsroCstg4BPLdnFqXv8ECJER4YUSlOQRnu5E wjJRIJgMTuKO8DAKnbTwi+FlEWvn2ujEc+yfTRgVGaqv8hkuFg5TNaH7V9IbuYKCaAF/q6 wKcFs6hYxXK9ouC+7gttnixJ6hgKKlsEGFFSTqgELJN1lDli7jqsXkTZRXB77LdbHkJwdF WaMf4zOds/fnQZjZg/cBfJ9f1AbcxxEh8T8rzCFKWIDepV0VIf/H3dPR8rzklCbiKL7Mco XgWPWxiS9m+9e7PzB9DUcXMTorZKYuoYqTENJXhxFgMyLHl68TNnK7NiKk4y2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1699966703; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0z8Fp4YItAuUG7y2aWkiau/D2pQEI+WL5ZVbPRSgM9o=; b=Zq07KTCXWi+2GLzDrgvvjeym9lENQ0jmZzBesYqTvvemZS8FTN8+riqLlQtnxNRTdrGObt bVMBpqPLOKWeSjQZ+JBHQxrmLs1MlQraAj6su0bfYvvZqFAXAn31F7Hl0i/DutvzWeQH/z YmTJBaI7w3G7/R6CD0ri6Gs09lJ4mYo7KOEY3K2A/BHqHC8OV2LGZv0OzG/MgT3j0w7Yte AHjZgTUjX90HItirl6SvCpXfUOt40a+nVgWoCsGwq06mP9UI8Ck/i60dXvKkHxDDIllbJB yAAU/JEHYck4Iq5yzbolg5Qh5TV3Jkf2xXOMHvAf6NCmNuXl6vbyzCrVJ+VFwg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1699966703; a=rsa-sha256; cv=none; b=Z17ZewUbKjU7Gy/WXS9aADdkgLda7G7OEhe9gXPya1xlB/HxHIaTOXUMOdK9l6E4lajEIV cWWYbI2oqgo9JS4JS9DItJ2y2vPIkd6vs/7T2Vy6K7tLArmYeGIDZd21AwuIEdIbnag3yX 0wvs/VcICALhPP3lRr9xKJT13XvBGfjx6uyw1K1XxamG01koVUerye3xWVxRJRY+SHO0sq IJ2Ye/hilUc9WQiChNIZu1PmdnFQDfaeB7QMWXmD9YpSRce2rsqy24pYTr2AysJViEiuqe lGbx9pd6lf7ba5XlPhJ2uCiHghTsLCx5tzfsvQ0HJgq/2ZKcvAJON+IKflH9Zw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SV5vM4MS1zq2x; Tue, 14 Nov 2023 12:58:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3AECwN17017303; Tue, 14 Nov 2023 12:58:23 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3AECwNsp017300; Tue, 14 Nov 2023 12:58:23 GMT (envelope-from git) Date: Tue, 14 Nov 2023 12:58:23 GMT Message-Id: <202311141258.3AECwNsp017300@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Alexander Motin Subject: git: 35372e305863 - stable/13 - nvme: Introduce longer timeouts for admin queue List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 35372e305863b7c372d8f019d3f422e7d2ee3be9 Auto-Submitted: auto-generated The branch stable/13 has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=35372e305863b7c372d8f019d3f422e7d2ee3be9 commit 35372e305863b7c372d8f019d3f422e7d2ee3be9 Author: Alexander Motin AuthorDate: 2023-11-06 16:05:48 +0000 Commit: Alexander Motin CommitDate: 2023-11-14 12:57:28 +0000 nvme: Introduce longer timeouts for admin queue KIOXIA CD8 SSDs routinely take ~25 seconds to delete non-empty namespace. In some cases like hot-plug it takes longer, triggering timeout and controller resets after just 30 seconds. Linux for many years has separate 60 seconds timeout for admin queue. This patch does the same. And it is good to be consistent. Sponsored by: iXsystems, Inc. Reviewed by: imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D42454 (cherry picked from commit 8d6c0743e36e3cff9279c40468711a82db98df23) --- sys/dev/nvme/nvme_ctrlr.c | 6 ++++++ sys/dev/nvme/nvme_private.h | 2 ++ sys/dev/nvme/nvme_qpair.c | 2 ++ sys/dev/nvme/nvme_sysctl.c | 15 ++++++++++----- 4 files changed, 20 insertions(+), 5 deletions(-) diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c index 84aa402af4f8..bcbc8dc26ee1 100644 --- a/sys/dev/nvme/nvme_ctrlr.c +++ b/sys/dev/nvme/nvme_ctrlr.c @@ -1437,6 +1437,12 @@ nvme_ctrlr_construct(struct nvme_controller *ctrlr, device_t dev) to = NVME_CAP_LO_TO(cap_lo) + 1; ctrlr->ready_timeout_in_ms = to * 500; + timeout_period = NVME_ADMIN_TIMEOUT_PERIOD; + TUNABLE_INT_FETCH("hw.nvme.admin_timeout_period", &timeout_period); + timeout_period = min(timeout_period, NVME_MAX_TIMEOUT_PERIOD); + timeout_period = max(timeout_period, NVME_MIN_TIMEOUT_PERIOD); + ctrlr->admin_timeout_period = timeout_period; + timeout_period = NVME_DEFAULT_TIMEOUT_PERIOD; TUNABLE_INT_FETCH("hw.nvme.timeout_period", &timeout_period); timeout_period = min(timeout_period, NVME_MAX_TIMEOUT_PERIOD); diff --git a/sys/dev/nvme/nvme_private.h b/sys/dev/nvme/nvme_private.h index e34dc20c2819..e2e2f5efeacc 100644 --- a/sys/dev/nvme/nvme_private.h +++ b/sys/dev/nvme/nvme_private.h @@ -85,6 +85,7 @@ MALLOC_DECLARE(M_NVME); #define NVME_MAX_CONSUMERS (2) #define NVME_MAX_ASYNC_EVENTS (8) +#define NVME_ADMIN_TIMEOUT_PERIOD (60) /* in seconds */ #define NVME_DEFAULT_TIMEOUT_PERIOD (30) /* in seconds */ #define NVME_MIN_TIMEOUT_PERIOD (5) #define NVME_MAX_TIMEOUT_PERIOD (120) @@ -287,6 +288,7 @@ struct nvme_controller { uint32_t int_coal_threshold; /** timeout period in seconds */ + uint32_t admin_timeout_period; uint32_t timeout_period; /** doorbell stride */ diff --git a/sys/dev/nvme/nvme_qpair.c b/sys/dev/nvme/nvme_qpair.c index e31e4249b20c..d810d51fadce 100644 --- a/sys/dev/nvme/nvme_qpair.c +++ b/sys/dev/nvme/nvme_qpair.c @@ -1062,6 +1062,8 @@ nvme_qpair_submit_tracker(struct nvme_qpair *qpair, struct nvme_tracker *tr) if (req->timeout) { if (req->cb_fn == nvme_completion_poll_cb) timeout = 1; + else if (qpair->id == 0) + timeout = ctrlr->admin_timeout_period; else timeout = ctrlr->timeout_period; tr->deadline = getsbinuptime() + timeout * SBT_1S; diff --git a/sys/dev/nvme/nvme_sysctl.c b/sys/dev/nvme/nvme_sysctl.c index 091c2f572eaf..c9c57ac91af5 100644 --- a/sys/dev/nvme/nvme_sysctl.c +++ b/sys/dev/nvme/nvme_sysctl.c @@ -132,8 +132,8 @@ nvme_sysctl_int_coal_threshold(SYSCTL_HANDLER_ARGS) static int nvme_sysctl_timeout_period(SYSCTL_HANDLER_ARGS) { - struct nvme_controller *ctrlr = arg1; - uint32_t newval = ctrlr->timeout_period; + uint32_t *ptr = arg1; + uint32_t newval = *ptr; int error = sysctl_handle_int(oidp, &newval, 0, req); if (error || (req->newptr == NULL)) @@ -143,7 +143,7 @@ nvme_sysctl_timeout_period(SYSCTL_HANDLER_ARGS) newval < NVME_MIN_TIMEOUT_PERIOD) { return (EINVAL); } else { - ctrlr->timeout_period = newval; + *ptr = newval; } return (0); @@ -333,10 +333,15 @@ nvme_sysctl_initialize_ctrlr(struct nvme_controller *ctrlr) nvme_sysctl_int_coal_threshold, "IU", "Interrupt coalescing threshold"); + SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, + "admin_timeout_period", CTLTYPE_UINT | CTLFLAG_RW | CTLFLAG_MPSAFE, + &ctrlr->admin_timeout_period, 0, nvme_sysctl_timeout_period, "IU", + "Timeout period for Admin queue (in seconds)"); + SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, "timeout_period", CTLTYPE_UINT | CTLFLAG_RW | CTLFLAG_MPSAFE, - ctrlr, 0, nvme_sysctl_timeout_period, "IU", - "Timeout period (in seconds)"); + &ctrlr->timeout_period, 0, nvme_sysctl_timeout_period, "IU", + "Timeout period for I/O queues (in seconds)"); SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, "num_cmds", CTLTYPE_S64 | CTLFLAG_RD | CTLFLAG_MPSAFE,