Date: Wed, 27 Nov 2019 09:50:15 -0800 From: BulkMailForRudy <crapsh@monkeybrains.net> To: freebsd-net@freebsd.org Subject: Re: ix0 and ix1 ifconfig options different on Supermicro board Message-ID: <9f75e32b-be73-cfb8-b80d-1b2eff1caf17@monkeybrains.net> In-Reply-To: <CA%2B_eA9iV=MZSrC%2BQHWnM6kQcTj35wpkeaUfC5ON39eUj4ZevSQ@mail.gmail.com> References: <e4f76015-a661-30aa-a9ef-166f683a6082@monkeybrains.net> <c0cd7a45-d986-2f40-db65-610b543ffebe@monkeybrains.net> <CA%2B_eA9iV=MZSrC%2BQHWnM6kQcTj35wpkeaUfC5ON39eUj4ZevSQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Thank you, Vincenzo. To anyone else looking this up, 'iperf' is not dependent on the TSO/LRO. iperf -c 10.1.1.1 -P 4 ---> 9.4Gbps iperf3 -c 10.1.1.1 -P 4 ---> 5.1Gbps I still find it odd that the default FreeBSD install has different options for ix0 and ix1. Rudy On 11/23/19 12:51 AM, Vincenzo Maffione wrote: > Hi, > TSO/LRO (for IPv4 and/or IPv6) will increase TCP bulk throughput on > machine X for those TCP connection where X is one of the two endpoints, > that is TCP connections that are local to X. That's why you are seing iperf > achieving higher throughput with TSO/LRO enabled. > TSO means that your local TCP stack will pass down large (e.g. 32K) packets > to the NIC driver, and the NIC will take care of segmentation. This is > beneficial for two reasons: (1) the segmentation work is done in hardware > rather than in the CPU, and this is typically faster (and also, you save > the CPU time for other stuff); (2) the per-packet cost of protocol > processing (TCP, IP, Ethernet) is amortized over a large amount of bytes, > which means that your total per-byte CPU time will be way lower. Most of > the gain actually comes from (2). > LRO is similar, but in the receive direction. > > However, if your device is a router it means that it forwards packets. > Therefore the local TCP stack is not involved, so TSO simply does not apply > (at least in FreeBSD). > I think LRO applies, but there is a latency hit, as suggested by the wiki > page you pointed. > > So no, enabling TSO/LRO will not increase the forwarding rate, but possibly > increase latency. You should keep it disabled. > > Cheers, > Vincenzo > > Il giorno ven 22 nov 2019 alle ore 22:47 BulkMailForRudy < > crapsh@monkeybrains.net> ha scritto: > >> I just did another test to a machine with a Chelsio card. >> >> Server D (cxl3) -> Server A = 3.5Gbps >> >> Turning on flags lro tso4 tso6 vlanhwtso , yields >> >> Server D (cxl3) -> Server A = 9.1 Gbps >> >> Oddly, this was an ipv4 iperf, but tso6 seems to help. >> >> I had settings turned off per >> https://wiki.freebsd.org/10gFreeBSD/Router#Disabling_LRO_and_TSO >> >> Servers A,B, and C are all running services. Server D is acting as a >> router. Are the LRO and TSO only for TCP to the box, or will it >> increase speeds for forwarding if I enable it? >> >> >> Thanks, >> >> Rudy >> >> >> On 11/22/19 1:30 PM, BulkMailForRudy wrote: >>> I have nearly identical setups, but ix0 and ix1 are getting different >>> options at boot. This seems to be the only difference I see between >>> machines and I am trying to answer the question, Why can Server A >>> iperf close to line rate while the other servers can not? >>> >>> The Test: iperf -P 3 -c REMOTE_ADDR >>> >>> Server A (ix1) -> Server C (ix0) = 9.4Gbps >>> Server B (ix0)-> Server C (ix0) = 5.6Gbps >>> Server C (ix0)-> A (ix1) or B (ix0) = 5.0Gbps >>> >>> >>> The motherboards are identical between A,B and C and the configs very >>> similar. The only difference is that Server A is plugged into ix1 >>> while Server B and C are using ix0. >>> >>> >>> I am not modifying the flags at boot (eg ifconfig -tso), yet ix0 lacks >>> TXCSUM,TSO4,TSO6,LRO,WOL. >>> >>> ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric >>> 0 mtu 1500 >>> >> options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6> >> >>> ether *ac:1f:6b:6a:14:6*4 >>> media: Ethernet autoselect (10Gbase-T <full-duplex>) >>> ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 >>> >> options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> >> >>> ether *ac:1f:6b:6a:14:6*5 >>> media: Ethernet autoselect (10Gbase-T <full-duplex>) >>> >>> I did try adding some flags to ix0 and -- not sure if this was the >>> reason -- the box started acting oddly and I ended up rebooting it. >>> >>> >>> My hunch has is that there is somethign with the TSO4. >>> >>> >>> Rudy >>> >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9f75e32b-be73-cfb8-b80d-1b2eff1caf17>