From nobody Thu Aug 3 22:36:05 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RH3bT4b0Jz4V7TS; Thu, 3 Aug 2023 22:36:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RH3bT46RWz3dPT; Thu, 3 Aug 2023 22:36:05 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1691102165; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=boV0jpb89AK+3nzernSYTHJO+aEfcDeQKM8b068znOo=; b=fgFUV6XZX9HCh6jyB/ExKsZi90eWunDu9R8qM6CRVZ2/f7mFVa9bYv3uj+KOvG3tr6LQq0 /H7OSfXLE9xyaVbGppKOb+Wu6lg9BLIyg8Jb6543/ZbcKfx7ocENC5BR9caL2fW+qIo6GO ETgdm7ZNrFpbThamQesJiqAc2nltyKO2OBxwqh5UAn3OF2zKikgjYY57i+cKt6sexdd+kZ WvUGFjxCKvYBvlQU9M0V+dHM2XrOfNswsSY44nlry42Kisq682SI/O4Nfy1lGIR8iOkLZO W/hcZU0LKPJhLteC+9DsAXzJbeQq0yRqoEjH+waaFjzxknD5+Hj4sjecc4Tb2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1691102165; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=boV0jpb89AK+3nzernSYTHJO+aEfcDeQKM8b068znOo=; b=iYspPJsxAAJdGHfoRlDjYeDVAkpur1H7p1kgDK2S5apmATqcWSaJUIHwq8RsdFD6on2UjP EKXyLKo2LPegBEUOrEem2MouTYoZu+WmSizie3HIAwVruGhcZlN6mSZtScFDCDLo+IqLRh hRCFY0fbC5ca4pBMEpAZIEQHNk/Kp6hIeiPO0K0tL4NdaudUUZKaWlsBy5/aWDNHGUwooW I9eqfcOXvIMT0WNfNnTFP3zgkSv8K7yRLJwzTXQ+V8Y1RBFhhKhLK1sFt7v5Xrcc2nkLT1 F0w7tCex2rv2H7YhTT3/TyRqtNVgZQ1KtLOCql/8DVvEojDSllwY4FBfa63/Zg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1691102165; a=rsa-sha256; cv=none; b=njXFFn9SHKfkdIVKWwcjJtb5we5Z/iarA0T3m5fjHM6mJDZTJ34A0M0+aV/IHgjJ7Nxu4/ UhLFiyFAcWCLvFMgAUYfBX4NE0mA/I83uAnkWQuY4DhGdpar31ZZKV0emhxCDZFdewwQxb hXAVMKR9Puk0tNIDw1HLuMY0x7fwZdJlBJbcOGO3gtJ3OyuAJfV0hp1ttadjpN47TJaE0A QsZDi0uDfF8q43w/hpVXvibC5jLxHXh7gU3k84YwkR8iv0r1UJzkAG9Wllo/RdGn1K6ErW f6AmbpaTiUuH3C2nQERRHX1xvvaDHkLX+YRsW060FEanQNbbokTJ6m+jKgNhiA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RH3bT2nX1zf1g; Thu, 3 Aug 2023 22:36:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 373Ma5RS038273; Thu, 3 Aug 2023 22:36:05 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 373Ma5EM038272; Thu, 3 Aug 2023 22:36:05 GMT (envelope-from git) Date: Thu, 3 Aug 2023 22:36:05 GMT Message-Id: <202308032236.373Ma5EM038272@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Kevin Bowling Subject: git: f1b5488f7bba - main - e1000: Enable TSO for lem(4) and em(4) List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: f1b5488f7bba7f25a57750f87cbcbccbd5b9d16b Auto-Submitted: auto-generated The branch main has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=f1b5488f7bba7f25a57750f87cbcbccbd5b9d16b commit f1b5488f7bba7f25a57750f87cbcbccbd5b9d16b Author: Kevin Bowling AuthorDate: 2023-08-03 20:49:15 +0000 Commit: Kevin Bowling CommitDate: 2023-08-03 22:35:28 +0000 e1000: Enable TSO for lem(4) and em(4) Most em(4) devices now enjoy TSO and TSO6, matching NetBSD and Linux defaults. A prior commit automasks TSO on 10/100 Ethernet due to errata and other bugs for IPv6 were fixed recently allowing this. Mike Karels identified a performance anomaly on Intel 82574L devices. These are multiqueue enabled on FreeBSD since the conversion to iflib. I am investigating whether this can be fixed, in the mean time MSI-X with checksum offloads remain default. i219 SPT devices have an errata that downclocks the DMA engine, which results in TSO not being able to acheive line rate. Therefore, it is disabled on: * Intel(R) I219-LM and I219-V SPT * Intel(R) I219-LM and I219-V SPT-H (2) * Intel(R) I219-LM and I219-V LBG (3) * Intel(R) I219-LM and I219-V SPT (4) * Intel(R) I219-LM and I219-V SPT (5) Many lem(4) devices enjoy TSO, exceptions being 82542, 82543, 82547. TSO6 may be possible for some chipsets but I am still working through my testing matrix and that is hidden behind hw.em.unsupported_tso. If you encounter issues, you may disable TSO with for example: ifconfig em0 -tso -tso6. I ask to be informed of any deviations from normal operation requiring this. Thanks to cc@ for access to emulab.net. On a sample I219 system it saves about 16% CPU on IPv4 and 19% on IPv6. iperf3 -Vc reported numbers: total% user% system% IPv4 TSO 21.3 7 14.4 21.4 6 15.4 21.5 6 15.5 IPv4 no TSO 36.8 5.4 31.4 38.5 5.1 33.5 38.2 5.7 32.6 IPv4 no TSO no TXCSUM 45.1 5.8 39.3 46 6.3 39.7 46.2 5.9 40.4 IPv6 TSO6 21.7 5.4 16.3 21.6 5.1 16.5 21.9 5.6 16.3 IPv6 no TSO6 41.2 5.2 36 41 5.1 36 40.8 5.2 35.7 IPv6 no TSO6 no TXCSUM6 49 5.9 43.1 48.8 4.9 43.9 49 5.6 43.4 Tested by: cc (lem(4)), karels (82574L) MFC after: 3 months Relnotes: yes Sponsored by: BBOX.io Differential Revision: https://reviews.freebsd.org/D41170 --- sys/dev/e1000/if_em.c | 66 +++++++++++++++++++++++++++++---------------------- 1 file changed, 37 insertions(+), 29 deletions(-) diff --git a/sys/dev/e1000/if_em.c b/sys/dev/e1000/if_em.c index 1da0579cdce9..fefd1e3f411c 100644 --- a/sys/dev/e1000/if_em.c +++ b/sys/dev/e1000/if_em.c @@ -903,25 +903,19 @@ em_if_attach_pre(if_ctx_t ctx) scctx->isc_tx_tso_size_max = EM_TSO_SIZE; scctx->isc_tx_tso_segsize_max = EM_TSO_SEG_SIZE; scctx->isc_capabilities = scctx->isc_capenable = EM_CAPS; - /* - * For EM-class devices, don't enable IFCAP_{TSO4,VLAN_HWTSO,TSO6} - * by default as we don't have workarounds for all associated - * silicon errata. E. g., with several MACs such as 82573E, - * TSO only works at Gigabit speed and otherwise can cause the - * hardware to hang (which also would be next to impossible to - * work around given that already queued TSO-using descriptors - * would need to be flushed and vlan(4) reconfigured at runtime - * in case of a link speed change). Moreover, MACs like 82579 - * still can hang at Gigabit even with all publicly documented - * TSO workarounds implemented. Generally, the penality of - * these workarounds is rather high and may involve copying - * mbuf data around so advantages of TSO lapse. Still, TSO may - * work for a few MACs of this class - at least when sticking - * with Gigabit - in which case users may enable TSO manually. - */ - scctx->isc_capenable &= ~(IFCAP_TSO4 | IFCAP_VLAN_HWTSO | IFCAP_TSO6); scctx->isc_tx_csum_flags = CSUM_TCP | CSUM_UDP | CSUM_IP_TSO | CSUM_IP6_TCP | CSUM_IP6_UDP; + + /* Disable TSO on 82574L due to performance loss being investigated */ + if (hw->mac.type == e1000_82574) + scctx->isc_capenable &= ~IFCAP_TSO; + /* + * Disable TSO on SPT due to errata that downclocks DMA performance + * i218-i219 Specification Update 1.5.4.5 + */ + if (hw->mac.type == e1000_pch_spt) + scctx->isc_capenable &= ~IFCAP_TSO; + /* * We support MSI-X with 82574 only, but indicate to iflib(4) * that it shall give MSI at least a try with other devices. @@ -944,24 +938,38 @@ em_if_attach_pre(if_ctx_t ctx) scctx->isc_capabilities = scctx->isc_capenable = LEM_CAPS; if (em_unsupported_tso) scctx->isc_capabilities |= IFCAP_TSO6; - /* - * For LEM-class devices, don't enable IFCAP_{TSO4,VLAN_HWTSO} - * by default as we don't have workarounds for all associated - * silicon errata. TSO4 may work on > 82544 but its status - * is unknown by the authors. Please report any success or failures. - */ - scctx->isc_capenable &= ~(IFCAP_TSO4 | IFCAP_VLAN_HWTSO); scctx->isc_tx_csum_flags = CSUM_TCP | CSUM_UDP | CSUM_IP_TSO | CSUM_IP6_TCP | CSUM_IP6_UDP; + /* 82541ER doesn't do HW tagging */ + if (hw->device_id == E1000_DEV_ID_82541ER || + hw->device_id == E1000_DEV_ID_82541ER_LOM) { + scctx->isc_capabilities &= ~IFCAP_VLAN_HWTAGGING; + scctx->isc_capenable = scctx->isc_capabilities; + } + /* This is the first e1000 chip and it does not do offloads */ + if (hw->mac.type == e1000_82542) { + scctx->isc_capabilities &= ~(IFCAP_HWCSUM | IFCAP_VLAN_HWCSUM | + IFCAP_HWCSUM_IPV6 | IFCAP_VLAN_HWTAGGING | + IFCAP_VLAN_HWFILTER | IFCAP_TSO | IFCAP_VLAN_HWTSO); + scctx->isc_capenable = scctx->isc_capabilities; + } + /* These can't do TSO for various reasons */ + if (hw->mac.type < e1000_82544 || hw->mac.type == e1000_82547 || + hw->mac.type == e1000_82547_rev_2) { + scctx->isc_capabilities &= ~(IFCAP_TSO | IFCAP_VLAN_HWTSO); + scctx->isc_capenable = scctx->isc_capabilities; + } + /* XXXKB: No IPv6 before this? */ + if (hw->mac.type < e1000_82545){ + scctx->isc_capabilities &= ~IFCAP_HWCSUM_IPV6; + scctx->isc_capenable = scctx->isc_capabilities; + } /* "PCI/PCI-X SDM 4.0" page 33 (b) - FDX requirement on these chips */ - if (hw->mac.type == e1000_82542 || hw->mac.type == e1000_82547 || - hw->mac.type == e1000_82547_rev_2) + if (hw->mac.type == e1000_82547 || hw->mac.type == e1000_82547_rev_2) scctx->isc_capenable &= ~(IFCAP_HWCSUM | IFCAP_VLAN_HWCSUM | IFCAP_HWCSUM_IPV6); - /* 82541ER doesn't do HW tagging */ - if (hw->device_id == E1000_DEV_ID_82541ER || hw->device_id == E1000_DEV_ID_82541ER_LOM) - scctx->isc_capenable &= ~IFCAP_VLAN_HWTAGGING; + /* INTx only */ scctx->isc_msix_bar = 0; }