Date: Thu, 3 Aug 2023 22:36:05 GMT From: Kevin Bowling <kbowling@FreeBSD.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: git: f1b5488f7bba - main - e1000: Enable TSO for lem(4) and em(4) Message-ID: <202308032236.373Ma5EM038272@gitrepo.freebsd.org>
next in thread | raw e-mail | index | archive | help
The branch main has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=f1b5488f7bba7f25a57750f87cbcbccbd5b9d16b commit f1b5488f7bba7f25a57750f87cbcbccbd5b9d16b Author: Kevin Bowling <kbowling@FreeBSD.org> AuthorDate: 2023-08-03 20:49:15 +0000 Commit: Kevin Bowling <kbowling@FreeBSD.org> CommitDate: 2023-08-03 22:35:28 +0000 e1000: Enable TSO for lem(4) and em(4) Most em(4) devices now enjoy TSO and TSO6, matching NetBSD and Linux defaults. A prior commit automasks TSO on 10/100 Ethernet due to errata and other bugs for IPv6 were fixed recently allowing this. Mike Karels identified a performance anomaly on Intel 82574L devices. These are multiqueue enabled on FreeBSD since the conversion to iflib. I am investigating whether this can be fixed, in the mean time MSI-X with checksum offloads remain default. i219 SPT devices have an errata that downclocks the DMA engine, which results in TSO not being able to acheive line rate. Therefore, it is disabled on: * Intel(R) I219-LM and I219-V SPT * Intel(R) I219-LM and I219-V SPT-H (2) * Intel(R) I219-LM and I219-V LBG (3) * Intel(R) I219-LM and I219-V SPT (4) * Intel(R) I219-LM and I219-V SPT (5) Many lem(4) devices enjoy TSO, exceptions being 82542, 82543, 82547. TSO6 may be possible for some chipsets but I am still working through my testing matrix and that is hidden behind hw.em.unsupported_tso. If you encounter issues, you may disable TSO with for example: ifconfig em0 -tso -tso6. I ask to be informed of any deviations from normal operation requiring this. Thanks to cc@ for access to emulab.net. On a sample I219 system it saves about 16% CPU on IPv4 and 19% on IPv6. iperf3 -Vc reported numbers: total% user% system% IPv4 TSO 21.3 7 14.4 21.4 6 15.4 21.5 6 15.5 IPv4 no TSO 36.8 5.4 31.4 38.5 5.1 33.5 38.2 5.7 32.6 IPv4 no TSO no TXCSUM 45.1 5.8 39.3 46 6.3 39.7 46.2 5.9 40.4 IPv6 TSO6 21.7 5.4 16.3 21.6 5.1 16.5 21.9 5.6 16.3 IPv6 no TSO6 41.2 5.2 36 41 5.1 36 40.8 5.2 35.7 IPv6 no TSO6 no TXCSUM6 49 5.9 43.1 48.8 4.9 43.9 49 5.6 43.4 Tested by: cc (lem(4)), karels (82574L) MFC after: 3 months Relnotes: yes Sponsored by: BBOX.io Differential Revision: https://reviews.freebsd.org/D41170 --- sys/dev/e1000/if_em.c | 66 +++++++++++++++++++++++++++++---------------------- 1 file changed, 37 insertions(+), 29 deletions(-) diff --git a/sys/dev/e1000/if_em.c b/sys/dev/e1000/if_em.c index 1da0579cdce9..fefd1e3f411c 100644 --- a/sys/dev/e1000/if_em.c +++ b/sys/dev/e1000/if_em.c @@ -903,25 +903,19 @@ em_if_attach_pre(if_ctx_t ctx) scctx->isc_tx_tso_size_max = EM_TSO_SIZE; scctx->isc_tx_tso_segsize_max = EM_TSO_SEG_SIZE; scctx->isc_capabilities = scctx->isc_capenable = EM_CAPS; - /* - * For EM-class devices, don't enable IFCAP_{TSO4,VLAN_HWTSO,TSO6} - * by default as we don't have workarounds for all associated - * silicon errata. E. g., with several MACs such as 82573E, - * TSO only works at Gigabit speed and otherwise can cause the - * hardware to hang (which also would be next to impossible to - * work around given that already queued TSO-using descriptors - * would need to be flushed and vlan(4) reconfigured at runtime - * in case of a link speed change). Moreover, MACs like 82579 - * still can hang at Gigabit even with all publicly documented - * TSO workarounds implemented. Generally, the penality of - * these workarounds is rather high and may involve copying - * mbuf data around so advantages of TSO lapse. Still, TSO may - * work for a few MACs of this class - at least when sticking - * with Gigabit - in which case users may enable TSO manually. - */ - scctx->isc_capenable &= ~(IFCAP_TSO4 | IFCAP_VLAN_HWTSO | IFCAP_TSO6); scctx->isc_tx_csum_flags = CSUM_TCP | CSUM_UDP | CSUM_IP_TSO | CSUM_IP6_TCP | CSUM_IP6_UDP; + + /* Disable TSO on 82574L due to performance loss being investigated */ + if (hw->mac.type == e1000_82574) + scctx->isc_capenable &= ~IFCAP_TSO; + /* + * Disable TSO on SPT due to errata that downclocks DMA performance + * i218-i219 Specification Update 1.5.4.5 + */ + if (hw->mac.type == e1000_pch_spt) + scctx->isc_capenable &= ~IFCAP_TSO; + /* * We support MSI-X with 82574 only, but indicate to iflib(4) * that it shall give MSI at least a try with other devices. @@ -944,24 +938,38 @@ em_if_attach_pre(if_ctx_t ctx) scctx->isc_capabilities = scctx->isc_capenable = LEM_CAPS; if (em_unsupported_tso) scctx->isc_capabilities |= IFCAP_TSO6; - /* - * For LEM-class devices, don't enable IFCAP_{TSO4,VLAN_HWTSO} - * by default as we don't have workarounds for all associated - * silicon errata. TSO4 may work on > 82544 but its status - * is unknown by the authors. Please report any success or failures. - */ - scctx->isc_capenable &= ~(IFCAP_TSO4 | IFCAP_VLAN_HWTSO); scctx->isc_tx_csum_flags = CSUM_TCP | CSUM_UDP | CSUM_IP_TSO | CSUM_IP6_TCP | CSUM_IP6_UDP; + /* 82541ER doesn't do HW tagging */ + if (hw->device_id == E1000_DEV_ID_82541ER || + hw->device_id == E1000_DEV_ID_82541ER_LOM) { + scctx->isc_capabilities &= ~IFCAP_VLAN_HWTAGGING; + scctx->isc_capenable = scctx->isc_capabilities; + } + /* This is the first e1000 chip and it does not do offloads */ + if (hw->mac.type == e1000_82542) { + scctx->isc_capabilities &= ~(IFCAP_HWCSUM | IFCAP_VLAN_HWCSUM | + IFCAP_HWCSUM_IPV6 | IFCAP_VLAN_HWTAGGING | + IFCAP_VLAN_HWFILTER | IFCAP_TSO | IFCAP_VLAN_HWTSO); + scctx->isc_capenable = scctx->isc_capabilities; + } + /* These can't do TSO for various reasons */ + if (hw->mac.type < e1000_82544 || hw->mac.type == e1000_82547 || + hw->mac.type == e1000_82547_rev_2) { + scctx->isc_capabilities &= ~(IFCAP_TSO | IFCAP_VLAN_HWTSO); + scctx->isc_capenable = scctx->isc_capabilities; + } + /* XXXKB: No IPv6 before this? */ + if (hw->mac.type < e1000_82545){ + scctx->isc_capabilities &= ~IFCAP_HWCSUM_IPV6; + scctx->isc_capenable = scctx->isc_capabilities; + } /* "PCI/PCI-X SDM 4.0" page 33 (b) - FDX requirement on these chips */ - if (hw->mac.type == e1000_82542 || hw->mac.type == e1000_82547 || - hw->mac.type == e1000_82547_rev_2) + if (hw->mac.type == e1000_82547 || hw->mac.type == e1000_82547_rev_2) scctx->isc_capenable &= ~(IFCAP_HWCSUM | IFCAP_VLAN_HWCSUM | IFCAP_HWCSUM_IPV6); - /* 82541ER doesn't do HW tagging */ - if (hw->device_id == E1000_DEV_ID_82541ER || hw->device_id == E1000_DEV_ID_82541ER_LOM) - scctx->isc_capenable &= ~IFCAP_VLAN_HWTAGGING; + /* INTx only */ scctx->isc_msix_bar = 0; }
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202308032236.373Ma5EM038272>