From nobody Fri Aug 7 14:43:07 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hGn236MxJz6nCNt for ; Fri, 07 Aug 2026 14:43:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hGn23590Lz3Y6W for ; Fri, 07 Aug 2026 14:43:07 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1786113787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0SfPc6e6CU7f8XwdPC2+vNlV8rGjm7tDeMSDZWELeRo=; b=jk6Ct0vtq0h+jWKRD1+Ix7kIjFd/gHvqj8jw+Uw4viSHUba7kfDaIpxRXzfyYn2Gps744F /PEVYzUgnPFV4ULbPXQsPi/fbEUJsTQ4LEpclDqZFTdBjcTLWg892C0BBtumHVU9HtnDoU SnlywOqbZcpzVHGmZrnIZCHgGpxjR4Nh3KpC25vDASj67v95Cxld3U+1ASUjj0LqcyPoit VR8+4OM79zcteTYG+kGDYoov+Okad9ukefIdo2dZ0Im3AND9wl5XQM+vbZ/cGdVye63lJt 0rPbWQdeF/hPWnJUD0EospgaPJdTSVONSCcSx8Zn1h79ZC04xwn7lHdXKlT69g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1786113787; a=rsa-sha256; cv=none; b=oMnW/36wBuWwC2HsYIqoEhIJXw9/lsp292zbBcF5w/2f/yHIGlGlutyO8IE9VSIIDb6nYl w8gWsX73STNilsw3BmjCc7N2V4WeEe7nEi1BA1aTexxVycv1iBoVnZIj2daiqP2ruJxqav RA9E110JDMki98D9/1jOVnZWStdzwjws8uOP44TgN6BPMwhCX+QwbGQNWPt8cSB4x+7XCX LmuBcvDrnp2cqnO05lEO17EhtLieT8HW0O4I6rntNK3YdD3y2C+EyjzA3pKeLabn0TRgYj seI8MYvxLI2zTjm4iVOsqEKbsjqhT+NBSEGhe/anyz/3qRs3gJEz6rZcqPeZKQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1786113787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0SfPc6e6CU7f8XwdPC2+vNlV8rGjm7tDeMSDZWELeRo=; b=E1zWnhkPKNWrvKetvLBpzyQo+eX5nYz0nxzHyuNq9KD9UHGQ1HMG5HUYxjm6uul1lp4WgE ZuhBaQGROzzsy74VctuwBzUjKP5xA/JZ2vhBe4hWub3a5YTWXdWGNNCvZ1EeDJ3p0jYEQv e7X0faR//HpgaB4YlRmmrgPaVQN/tNMftre2DKj7WfnyUbPCgAOva5ULqkvtFe7bsO/k2p O/SlA5x17GV9zFLz4bXwh6gkNFrhaHi7asnOf7ixPCMqwbjKPRhaOIFfILD7lLsJ+OCjxN 07PYn4l39xQOJX3WuTMBVoRUT89G3DC4nTWGVoOap/Vep937deUiyLr5zezERg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hGn2341Syz1B2c for ; Fri, 07 Aug 2026 14:43:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 47079 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 07 Aug 2026 14:43:07 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Kevin Bowling Subject: git: aea4240ef583 - main - ixgbe: Quiesce VFs across PF reset List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: aea4240ef5834fb4a47f80c659c80f902cb4bb06 Auto-Submitted: auto-generated Date: Fri, 07 Aug 2026 14:43:07 +0000 Message-Id: <6a75eefb.47079.78db7538@gitrepo.freebsd.org> The branch main has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=aea4240ef5834fb4a47f80c659c80f902cb4bb06 commit aea4240ef5834fb4a47f80c659c80f902cb4bb06 Author: Kevin Bowling AuthorDate: 2026-08-06 08:21:45 +0000 Commit: Kevin Bowling CommitDate: 2026-08-07 14:38:12 +0000 ixgbe: Quiesce VFs across PF reset Stop VF transmit and receive in hardware, clear PF-side mailbox CTS, and notify active VFs before resetting a PF. A PF reset invalidates VF queue state, so the no-CTS control message makes cooperative VFs discard stale state and renegotiate after the PF returns. The hardware queue gates synchronously prevent further VF DMA. Do not hold the exclusive iflib context lock for a fixed VF-watchdog interval after the reset. Report the PF link transition directly instead of dispatching mailbox work from the stop path, which could otherwise re-enable VF I/O mid-reset. The CTS, PF-control, and VF queue controls follow the reset mechanisms used by DPDK. MFC after: 2 weeks --- sys/dev/ixgbe/if_ix.c | 11 +++++++++-- sys/dev/ixgbe/if_sriov.c | 34 ++++++++++++++++++++++++++++++++++ sys/dev/ixgbe/ixgbe_sriov.h | 2 ++ 3 files changed, 45 insertions(+), 2 deletions(-) diff --git a/sys/dev/ixgbe/if_ix.c b/sys/dev/ixgbe/if_ix.c index 5ab30a8054a9..e0e627b915d9 100644 --- a/sys/dev/ixgbe/if_ix.c +++ b/sys/dev/ixgbe/if_ix.c @@ -4757,8 +4757,10 @@ ixgbe_if_stop(if_ctx_t ctx) INIT_DEBUGOUT("ixgbe_if_stop: begin\n"); - if (sc->feat_en & IXGBE_FEATURE_SRIOV) + if (sc->feat_en & IXGBE_FEATURE_SRIOV) { ixgbe_disable_mdd(hw); + ixgbe_quiesce_vfs(sc); + } ixgbe_reset_hw(hw); hw->adapter_stopped = false; ixgbe_stop_adapter(hw); @@ -4767,7 +4769,12 @@ ixgbe_if_stop(if_ctx_t ctx) /* Update the stack */ sc->link_up = false; - ixgbe_if_update_admin_status(ctx); + if (sc->link_active) { + if (bootverbose) + device_printf(sc->dev, "Link is Down\n"); + iflib_link_state_change(ctx, LINK_STATE_DOWN, 0); + sc->link_active = false; + } /* reprogram the RAR[0] in case user changed it. */ ixgbe_set_rar(&sc->hw, 0, sc->hw.mac.addr, 0, IXGBE_RAH_AV); diff --git a/sys/dev/ixgbe/if_sriov.c b/sys/dev/ixgbe/if_sriov.c index af0027a073c8..8611f2685611 100644 --- a/sys/dev/ixgbe/if_sriov.c +++ b/sys/dev/ixgbe/if_sriov.c @@ -283,6 +283,40 @@ ixgbe_ping_all_vfs(struct ixgbe_softc *sc) } } /* ixgbe_ping_all_vfs */ +/* + * Stop VF DMA before resetting the PF. A PF reset invalidates the VF queue + * state, so allowing an active VF to resume with its old rings can strand + * descriptors in both the VF and PF. Clearing CTS makes a cooperative VF + * renegotiate its state after the PF comes back; it is deliberately separate + * from IXGBE_VF_IO_DISABLED, which records a persistent administrative or + * recovery decision. + */ +void +ixgbe_quiesce_vfs(struct ixgbe_softc *sc) +{ + struct ixgbe_hw *hw; + struct ixgbe_vf *vf; + uint32_t index, mask, vfre, vfte; + int i; + + hw = &sc->hw; + for (i = 0; i < sc->num_vfs; i++) { + vf = &sc->vfs[i]; + if (!(vf->flags & IXGBE_VF_ACTIVE)) + continue; + + vf->flags &= ~IXGBE_VF_CTS; + index = IXGBE_VF_INDEX(vf->pool); + mask = IXGBE_VF_BIT(vf->pool); + vfte = IXGBE_READ_REG(hw, IXGBE_VFTE(index)); + vfre = IXGBE_READ_REG(hw, IXGBE_VFRE(index)); + IXGBE_WRITE_REG(hw, IXGBE_VFTE(index), vfte & ~mask); + IXGBE_WRITE_REG(hw, IXGBE_VFRE(index), vfre & ~mask); + ixgbe_send_vf_msg(hw, vf, IXGBE_PF_CONTROL_MSG); + } + IXGBE_WRITE_FLUSH(hw); +} /* ixgbe_quiesce_vfs */ + static bool ixgbe_pf_owns_vlan(struct ixgbe_softc *sc, uint16_t tag) diff --git a/sys/dev/ixgbe/ixgbe_sriov.h b/sys/dev/ixgbe/ixgbe_sriov.h index dbe56ec1171c..de0485f6e9a2 100644 --- a/sys/dev/ixgbe/ixgbe_sriov.h +++ b/sys/dev/ixgbe/ixgbe_sriov.h @@ -89,6 +89,7 @@ int ixgbe_iov_validate(struct ixgbe_softc *, u16); void ixgbe_if_iov_uninit(if_ctx_t); void ixgbe_initialize_iov(struct ixgbe_softc *); void ixgbe_activate_vfs(struct ixgbe_softc *); +void ixgbe_quiesce_vfs(struct ixgbe_softc *); void ixgbe_recalculate_max_frame(struct ixgbe_softc *); void ixgbe_ping_all_vfs(struct ixgbe_softc *); void ixgbe_init_iov_recovery(struct ixgbe_softc *); @@ -109,6 +110,7 @@ u32 ixgbe_get_mrqc(int); #define ixgbe_uninit_iov(_a) #define ixgbe_initialize_iov(_a) #define ixgbe_activate_vfs(_a) +#define ixgbe_quiesce_vfs(_a) #define ixgbe_recalculate_max_frame(_a) #define ixgbe_ping_all_vfs(_a) #define ixgbe_init_iov_recovery(_a)