From nobody Wed Nov 29 01:55:30 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sg2TZ5HWLz52tLh; Wed, 29 Nov 2023 01:55:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sg2TZ4grsz4CTt; Wed, 29 Nov 2023 01:55:30 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701222930; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=9ZFORbckjLumWMO1PO6C/29/XvzY8rKQQ5zIv17yE9U=; b=dzbDhzdQgdd2bVZSgdnmgf66/RdUf9UfKK+X2GrVct5XdU/d3J6l51v73qfNm4xTgCdXlu PAIxgb4klCtB8zOclXCiLkY67n++7YQts+7F5D16x77604IHJHKr4fX94eowO8RjucLZNj S2h18OI/Mk1dCZM+CGgSc4LIkTB4j/ydYBXgDchNCXJTqpEnkslp+/9FLemcoBDDiiLn1Y U6abV50HYitC0X6AD9qs4IG7GWtNIW6vuES2K7ivl/BerXDMnSEWrBQVzbqvRascNCjs0f DcdEgW7E4UeuMy4psxtyNQi4kc8ggbPh0mBIBkAwUvtk6acXJkBFUuAfK6UC2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701222930; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=9ZFORbckjLumWMO1PO6C/29/XvzY8rKQQ5zIv17yE9U=; b=vBQCulZGm4CBKKajYmfXqMugEUrDzG0mnnIGnbfTSIvxUGoGw5ds/K87BRjqkU2DPZDT/I xh7+KzoI3dtwy2CmqRi7f8El3Ut3EB/ZViZDuSTQSP/D+1rq47XuZDBgfFuTgZ53UKjf+j QjeVKYB6i83DXYOwFihyU/Sv4XDht1IJY8Kg3t0U/OIKTLNtfG97gCuf303JQoto5Bt6ig 46hJrQyIIup0+JedLFcL6+vodyPE1ilbV8bvZpoEDlPm7XCtZ/kOPQaAF3dOAbh/yzE8v2 rX+DbT/auS7hnYnCo9t3anaYWxAvs4e2AHIlG6iTtm41wxIydNenddFK4J0lnQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1701222930; a=rsa-sha256; cv=none; b=usL+31X2r5g+XW3HcL9em9uz/lFUyxQHCw/CzUQPg5wTGAnUg0kvx5v5zqszlvvaO0IEV4 fwW7PsyDml83APkGqN3qGKLOfMZsS6p2uMe6eskk55x6h8qaFElLrQEluRpXEvJhbltnR4 TxbXEoxyDDjcjvHsFAOZVMLeHvBWeXx1KjuxHzBGGw2sDgVI6wlXmoBZ2fvgDF7vFWrTBC AJ6IlzeQJOWHSPMF3t+ON4z5eu7zfll7NZCwvJWS6ouJgcv0x5QbigTFYZ/GGotJg+hZ4Z guGu2vVLo9JPRH+eF1urEFffElzSDcUpfzs3Ky7NkaDRNxyloUnWoOAp9MzNVA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Sg2TZ3Jfkzmhd; Wed, 29 Nov 2023 01:55:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3AT1tUtf064157; Wed, 29 Nov 2023 01:55:30 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3AT1tUIM064154; Wed, 29 Nov 2023 01:55:30 GMT (envelope-from git) Date: Wed, 29 Nov 2023 01:55:30 GMT Message-Id: <202311290155.3AT1tUIM064154@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Warner Losh Subject: git: b411372b7d17 - main - mpi3mr: Don't hold fwevt_lock over call to taskqueue_drain List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: imp X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b411372b7d17ae7e5d6c944732d41b979bde2ac4 Auto-Submitted: auto-generated The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=b411372b7d17ae7e5d6c944732d41b979bde2ac4 commit b411372b7d17ae7e5d6c944732d41b979bde2ac4 Author: Warner Losh AuthorDate: 2023-11-29 01:48:48 +0000 Commit: Warner Losh CommitDate: 2023-11-29 01:48:48 +0000 mpi3mr: Don't hold fwevt_lock over call to taskqueue_drain Holding fwevt_lock when we call taskqueue_drain can lead to deadlock because it's draining a queue needs fwevt_lock to do work, so that other thread will try to take out the lock and block, making the thread never finish and taskqueue_drain never complete. There's a witness warning/error for this which was exposed when the lock was converted to a MTX_DEF lock from a MTX_SPIN prior to committing to the FreeBSD tree. The lock appears to be to protect against additional items being added to the event list while we're doing a reset. Since the taskqueue is blocked, items can get added to the list, but won't be processed during the reset, but there is still a (likely small) race between the taskqueue_drain and the taskqueue_block calls where an interrupt could fire on another CPU, resulting in a task being enqueued and started before the block can take effect. The only way to fix that race is to turn off interrupt processing during a reset. So we replace a deadlock with a smaller race. Sponsored by: Netflix Reviewed by: sumit.saxena_broadcom.com, mav, jhb Differential Revision: https://reviews.freebsd.org/D42537 --- sys/dev/mpi3mr/mpi3mr.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/sys/dev/mpi3mr/mpi3mr.c b/sys/dev/mpi3mr/mpi3mr.c index 776bfbdaee23..478b0944defa 100644 --- a/sys/dev/mpi3mr/mpi3mr.c +++ b/sys/dev/mpi3mr/mpi3mr.c @@ -2784,7 +2784,6 @@ int mpi3mr_initialize_ioc(struct mpi3mr_softc *sc, U8 init_type) mtx_init(&sc->sense_buf_q_lock, "Sense buffer Queue lock", NULL, MTX_SPIN); mtx_init(&sc->chain_buf_lock, "Chain buffer lock", NULL, MTX_SPIN); mtx_init(&sc->cmd_pool_lock, "Command pool lock", NULL, MTX_DEF); -// mtx_init(&sc->fwevt_lock, "Firmware Event lock", NULL, MTX_SPIN); mtx_init(&sc->fwevt_lock, "Firmware Event lock", NULL, MTX_DEF); mtx_init(&sc->target_lock, "Target lock", NULL, MTX_SPIN); mtx_init(&sc->reset_mutex, "Reset lock", NULL, MTX_DEF); @@ -5825,11 +5824,14 @@ static int mpi3mr_issue_reset(struct mpi3mr_softc *sc, U16 reset_type, inline void mpi3mr_cleanup_event_taskq(struct mpi3mr_softc *sc) { - mtx_lock(&sc->fwevt_lock); - taskqueue_drain(sc->cam_sc->ev_tq, &sc->cam_sc->ev_task); + /* + * Block the taskqueue before draining. This means any new tasks won't + * be queued to a worker thread. But it doesn't stop the current workers + * that are running. taskqueue_drain waits for those correctly in the + * case of thread backed taskqueues. + */ taskqueue_block(sc->cam_sc->ev_tq); - mtx_unlock(&sc->fwevt_lock); - return; + taskqueue_drain(sc->cam_sc->ev_tq, &sc->cam_sc->ev_task); } /**