From nobody Thu Jul 30 14:06:07 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4h9rb358xcz6mLQL for ; Thu, 30 Jul 2026 14:06:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4h9rb32xXHz3cjS for ; Thu, 30 Jul 2026 14:06:07 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1785420367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zX9NGhwtAqbl7PKrM67CnNTQAnYyY0+NMtv/TWAkeW0=; b=QcwCu3e8GA0Q/eE+GuDDZ0hZWzjiM+eLibQhivP4kTn4TkQgwRaPvKlgCIZ1rAqVdCGkL0 LwFUyzUt00puS26U+K96lT1eopS/JcpzCo+LJPDYNSqgE5Ozom/rTRdygcHvTTL7VrWmC6 +ivagfA0w91SJJF5CmDDJMekGbpMeo+buCYVjCUeuzil48sH7VgN2DsiqLS4plvf+C9fYM 8mkvJ+F0pBLgjitGuBSmncW0W5mov3d9JvJVozGOBtVm4+RRs06C4Mhlb7NE9F9wztXAds pCvZ3weavc93f6cmNmWnCAsRqHU/wmZ/4j75EwGiqL5DN//KSCCvX0tJTfvVTA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1785420367; a=rsa-sha256; cv=none; b=QjMP/3axMHFpD9AM5TPshhm+N/7SuHCURNeCgO8w+HolKm4v9t+MctPSQUmBbLVFKwkPTn JQPJ+PXNJkbYWj5IWCkL+g7mKpdzs4b01+26mhF/p9QyGY74foKb32XKWXP7WDCI9t7SiK uTxuLUWb+HoNddhr8XkX4qLfdMY1ibiqPZvv4vNSIGZUUDIAcbXNb3cAUh6g8gz2+irIZ7 1UUAqUf6NysTpQ3p1uIMBui6oqfvtIH8ycpf26bVAK6oYjyuyPYQdodBn6ekp4Ze1vmRjW BabUSUSsY8T/4i3F71xSw1f/fy0AWAt32LNRy4RCMD3wDu0tVGAp7FRXYkrSQA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1785420367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zX9NGhwtAqbl7PKrM67CnNTQAnYyY0+NMtv/TWAkeW0=; b=XOJteXQti0xLttKKjaF/ZSXEzjAMQiI1YE/m4ytU+jRbK57j/FMijnLZKayzND0lPXFGbR S9iOuWwjxmDZdsaDprBUNbSOFqNMtP5d0mPcl5IRS//gqSq0nu/s8y/0zo/2oE4KzF3rkF IU7ZCM1XeUMR4ALTB0QNT/1NeYi6FINKBeEUyUpYeS7eNopNs0u+JSAtgD3XlFcZxHBZ+h 67tTtj95S5oZMSX/CFaZ3YKaC9aNAIzLE5iLQRvgrjNxVqkZ2pq0zJ2lU/Rt2Maij1ADt+ IfiNyI0I4vuQMO4MjA0ZKODq8fjXfylcShAIC8VIWClwSheAOlYgW5zUMg7/6A== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4h9rb31s1cz1rP for ; Thu, 30 Jul 2026 14:06:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 1dd35 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Thu, 30 Jul 2026 14:06:07 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Kevin Bowling Subject: git: 8c8724705613 - main - igb: Sanitize retained VF queue state List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 8c8724705613c856836c3928e7240baae03e690c Auto-Submitted: auto-generated Date: Thu, 30 Jul 2026 14:06:07 +0000 Message-Id: <6a6b5a4f.1dd35.63cb957f@gitrepo.freebsd.org> The branch main has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=8c8724705613c856836c3928e7240baae03e690c commit 8c8724705613c856836c3928e7240baae03e690c Author: Kevin Bowling AuthorDate: 2026-07-30 07:40:55 +0000 Commit: Kevin Bowling CommitDate: 2026-07-30 14:06:00 +0000 igb: Sanitize retained VF queue state 82576 and I350 VFLR leave the VF queue configuration unchanged. A VF can program transmit head write-back and leave its DMA destination for a later VF owner; mainstream VF drivers do not overwrite TDWBAL/H. Disable every receive and transmit queue assigned to the VF, wait for the enable bits to clear, then clear SRRCTL, PSRTYPE, RXCTRL, TXCTRL, and TDWBAL/H. Spin briefly for the normal transition, then sleep at 100 microsecond intervals with an approximately 1 ms bound. This prevents a VF that keeps asserting QUEUE_ENABLE from busy-waiting the PF context lock for 10 ms. If a queue does not quiesce, leave the VF disabled and NACK its reset rather than programming an active queue. Rate-limit this diagnostic independently from mailbox and malicious-driver notifications. I350 maps pool n to queue n. 82576 assigns physical queues n and n+8 to VF n, so sanitize both queues while clearing per-pool PSRTYPE once. An incoming VF initializes its active ring base, head, and tail while enabling each queue. This is also required by malicious-driver recovery, which deliberately does not assert VTCTRL.RST because doing so would discard the VF's admin-vector routing before the PF can notify it. This implements Software Clarification 3 from the 82576 and I350 specification updates. Sponsored by: BBOX.io --- sys/dev/e1000/if_igb_iov.c | 132 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 124 insertions(+), 8 deletions(-) diff --git a/sys/dev/e1000/if_igb_iov.c b/sys/dev/e1000/if_igb_iov.c index f43e994c0440..3e03d056865e 100644 --- a/sys/dev/e1000/if_igb_iov.c +++ b/sys/dev/e1000/if_igb_iov.c @@ -25,14 +25,24 @@ /* 82576 Datasheet rev. 2.0, Section 8.14.16: VMOLR[31] must be one. */ #define IGB_82576_VMOLR_RSV (1U << 31) #define IGB_82576_LVMMC_BLOCK_MASK 0x1c +#define IGB_82576_NUM_QUEUES 16 #define IGB_82576_QUEUE_MASK 0xffff #define IGB_82576_STAGGERED_QUEUE_SHIFT 8 +#define IGB_82576_VF_QUEUE_STRIDE 8 +#define IGB_82576_VF_QUEUES 2 #define IGB_I350_DTXCTL_ENABLE_SPOOF_QUEUE (1U << 2) #define IGB_I350_LVMMC_MAC_VLAN_SPOOF (1U << 25) #define IGB_I350_LVMMC_LAST_Q_SHIFT 29 #define IGB_I350_LVMMC_LAST_Q_MASK 0x7 +#define IGB_I350_NUM_QUEUES 8 #define IGB_I350_QUEUE_MASK 0xff #define IGB_I350_RESET_ACK_TIMEOUT (100 * SBT_1MS) +#define IGB_I350_VF_QUEUES 1 +#define IGB_IOV_QUEUE_DISABLE_BUSY_RETRIES 10 +#define IGB_IOV_QUEUE_DISABLE_DELAY_US 10 +#define IGB_IOV_QUEUE_DISABLE_PAUSE (100 * SBT_1US) +#define IGB_IOV_QUEUE_DISABLE_RETRIES 20 +#define IGB_IOV_VF_QUEUES_MAX 2 #define IGB_VF_CTS (1U << 0) #define IGB_VF_CAP_MAC (1U << 1) @@ -59,6 +69,7 @@ struct igb_vf { struct timeval last_mbx_log; struct timeval last_spoof_log; struct timeval last_mdd_log; + struct timeval last_queue_log; sbintime_t mbx_retry_at; sbintime_t mdd_notify_at; sbintime_t vlan_token_time; @@ -1022,12 +1033,103 @@ igb_iov_vf_vlan_is_default(const struct igb_vf *vf) return (true); } -static void +static bool +igb_iov_sanitize_vf_queues(struct e1000_softc *sc, + struct igb_vf *vf) +{ + struct e1000_hw *hw; + u16 qid[IGB_IOV_VF_QUEUES_MAX]; + u32 rxdctl, txdctl; + int i, nqueues, retry; + + hw = &sc->hw; + switch (hw->mac.type) { + case e1000_82576: + nqueues = IGB_82576_VF_QUEUES; + qid[0] = vf->pool; + qid[1] = vf->pool + IGB_82576_VF_QUEUE_STRIDE; + break; + case e1000_i350: + nqueues = IGB_I350_VF_QUEUES; + qid[0] = vf->pool; + break; + default: + return (true); + } + + /* + * I350 maps pool n to queue n. 82576 gives VF n physical queues n + * and n + 8, so both retained queue configurations must be cleared. + */ + for (i = 0; i < nqueues; i++) + KASSERT(qid[i] < (hw->mac.type == e1000_82576 ? + IGB_82576_NUM_QUEUES : IGB_I350_NUM_QUEUES), + ("%s: invalid VF queue %u", __func__, qid[i])); + + /* + * The 82576 and I350 specification updates, Software Clarification 3, + * note that VFLR does not reset the VF queue configuration. Clear the + * PF-programmable state before acknowledging the reset so a new VF + * owner cannot inherit it, particularly a descriptor-head write-back + * DMA address. The new VF driver initializes its active ring pointers + * during queue setup. + * + * Disable every queue first, then wait for outstanding DMA activity to + * stop before clearing TDWBAL/H and the remaining retained state. + * Spin only for the normal fast transition, then sleep so a VF that + * keeps asserting QUEUE_ENABLE cannot busy-wait the PF for 10 ms. + */ + for (i = 0; i < nqueues; i++) { + E1000_WRITE_REG(hw, E1000_RXDCTL(qid[i]), 0); + E1000_WRITE_REG(hw, E1000_TXDCTL(qid[i]), 0); + } + E1000_WRITE_FLUSH(hw); + for (retry = 0; retry < IGB_IOV_QUEUE_DISABLE_RETRIES; retry++) { + for (i = 0; i < nqueues; i++) { + rxdctl = E1000_READ_REG(hw, E1000_RXDCTL(qid[i])); + txdctl = E1000_READ_REG(hw, E1000_TXDCTL(qid[i])); + if ((rxdctl & E1000_RXDCTL_QUEUE_ENABLE) != 0 || + (txdctl & E1000_TXDCTL_QUEUE_ENABLE) != 0) + break; + } + if (i == nqueues) + break; + if (retry + 1 < IGB_IOV_QUEUE_DISABLE_RETRIES) { + if (retry < IGB_IOV_QUEUE_DISABLE_BUSY_RETRIES) + DELAY(IGB_IOV_QUEUE_DISABLE_DELAY_US); + else + pause_sbt("igbqds", + IGB_IOV_QUEUE_DISABLE_PAUSE, 0, + C_PREL(1)); + } + } + if (retry == IGB_IOV_QUEUE_DISABLE_RETRIES) { + if (ratecheck(&vf->last_queue_log, + &igb_iov_mbx_log_interval)) + device_printf(sc->dev, + "could not disable queues for VF %u; " + "reset deferred\n", vf->pool); + return (false); + } + + for (i = 0; i < nqueues; i++) { + E1000_WRITE_REG(hw, E1000_SRRCTL(qid[i]), 0); + E1000_WRITE_REG(hw, E1000_DCA_RXCTRL(qid[i]), 0); + E1000_WRITE_REG(hw, E1000_TDWBAL(qid[i]), 0); + E1000_WRITE_REG(hw, E1000_TDWBAH(qid[i]), 0); + E1000_WRITE_REG(hw, E1000_DCA_TXCTRL(qid[i]), 0); + } + E1000_WRITE_REG(hw, E1000_PSRTYPE(vf->pool), 0); + E1000_WRITE_FLUSH(hw); + return (true); +} + +static bool igb_iov_reset_event_common(struct e1000_softc *sc, struct igb_vf *vf, bool reset_intrs) { struct e1000_hw *hw; - bool rebuild_mta, rebuild_vlan; + bool rebuild_mta, rebuild_vlan, sanitized; u32 reg; hw = &sc->hw; @@ -1040,6 +1142,7 @@ igb_iov_reset_event_common(struct e1000_softc *sc, struct igb_vf *vf, if (reset_intrs) E1000_WRITE_REG(hw, E1000_VTCTRL(vf->pool), E1000_VTCTRL_RST); + sanitized = igb_iov_sanitize_vf_queues(sc, vf); E1000_WRITE_REG(hw, E1000_VMVIR(vf->pool), 0); igb_iov_clear_mac_filters(sc, vf); igb_iov_clear_rar(sc, vf->rar_index); @@ -1048,12 +1151,13 @@ igb_iov_reset_event_common(struct e1000_softc *sc, struct igb_vf *vf, igb_iov_rebuild_mta(sc); if (rebuild_vlan) igb_iov_rebuild_vlan(sc); + return (sanitized); } -static void +static bool igb_iov_reset_event(struct e1000_softc *sc, struct igb_vf *vf) { - igb_iov_reset_event_common(sc, vf, true); + return (igb_iov_reset_event_common(sc, vf, true)); } static void @@ -1070,8 +1174,11 @@ igb_iov_mdd_reset_event(struct e1000_softc *sc, struct igb_vf *vf) * message to make the guest reinitialize. FreeBSD and DPDK consume * that message directly; Linux ACKs it and the PF's non-CTS ACK path * replies with the NACK that schedules igbvf's reset task. + * + * Sanitization failure leaves the pool disabled. The VF reset + * handshake retries it and is NACKed while a queue remains active. */ - igb_iov_reset_event_common(sc, vf, false); + (void)igb_iov_reset_event_common(sc, vf, false); } static void @@ -1081,7 +1188,11 @@ igb_iov_reset_msg(struct e1000_softc *sc, struct igb_vf *vf) u32 msg[3], reg; hw = &sc->hw; - igb_iov_reset_event(sc, vf); + if (!igb_iov_reset_event(sc, vf)) { + msg[0] = E1000_VF_RESET | E1000_VT_MSGTYPE_NACK; + e1000_write_mbx(hw, msg, 1, vf->pool); + return; + } igb_iov_map_rar(sc, vf->rar_index, vf->mac, vf->pool); igb_iov_set_anti_spoof(sc, vf); @@ -1369,8 +1480,13 @@ igb_iov_handle_mbx(struct e1000_softc *sc) if (!(vf->flags & IGB_VF_ACTIVE)) continue; now = getsbinuptime(); - if (e1000_check_for_rst(hw, vf->pool) == 0) - igb_iov_reset_event(sc, vf); + if (e1000_check_for_rst(hw, vf->pool) == 0) { + /* + * The old VF is gone. A new owner's reset handshake + * reruns sanitization before enabling its pool. + */ + (void)igb_iov_reset_event(sc, vf); + } if ((vf->flags & (IGB_VF_MBX_PENDING | IGB_VF_MBX_GAVE_UP)) == 0 && e1000_check_for_msg(hw, vf->pool) == 0) {