From owner-freebsd-net@freebsd.org Thu Dec 10 18:29:21 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 043B29D7934 for ; Thu, 10 Dec 2015 18:29:21 +0000 (UTC) (envelope-from dennix.pearson@gmail.com) Received: from mail-io0-x242.google.com (mail-io0-x242.google.com [IPv6:2607:f8b0:4001:c06::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C306C17F0 for ; Thu, 10 Dec 2015 18:29:20 +0000 (UTC) (envelope-from dennix.pearson@gmail.com) Received: by iofh3 with SMTP id h3so8917498iof.3 for ; Thu, 10 Dec 2015 10:29:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Oai9mh2biU3svXg7mRALAx5gxxMN35jv9KEZ+tA7rJs=; b=NkHrtFXJ/yggHrRiUYupI04hvHaXI+tDdG3IxHgiBZ9Opxb1+5ZSBTPbffEL3+Oh7O VNW5a+hYPZ/u2VT2tIGNRrVgANRRtKuuHesB5IP+z/Pyjmaks10qasmM83dGsXecO+os +7pFSZWIRVJSbx7xpYn6NnhVEu5ReVHjjsb9LZh+wvg+tEowZ37b8bWomyI5FDf57pDb lwXqkWYznwrrO8L8xiYwO33ajAwzupzlsQVKnjbuBrwbV7O86ijW6ilQ5eNR7dLVNAQV Swi7T1b7hstIML+S96NhLFjsPwhqxsLKkWF4YtYy42hOzZ/QA6+lqR9E/hjhNnT5tfiT HHeg== MIME-Version: 1.0 X-Received: by 10.107.135.94 with SMTP id j91mr12924026iod.54.1449772160256; Thu, 10 Dec 2015 10:29:20 -0800 (PST) Received: by 10.79.27.149 with HTTP; Thu, 10 Dec 2015 10:29:20 -0800 (PST) In-Reply-To: References: <5aae0ee63c44627223d5d179f1901d00@pyret.net> <0E4C2D93-FBAF-48CB-A704-499ABFC892B9@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2A8A@ORSMSX111.amr.corp.intel.com> <99E53825-99F8-4E82-A710-6BC07B123F77@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2D52@ORSMSX111.amr.corp.intel.com> Date: Thu, 10 Dec 2015 16:29:20 -0200 Message-ID: Subject: Re: ixl 40G bad performance? From: Denis Pearson To: "Eggert, Lars" Cc: "Pieper, Jeffrey E" , Kevin Oberman , Daniel Engberg , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2015 18:29:21 -0000 On Thu, Dec 10, 2015 at 2:18 PM, Eggert, Lars wrote: > On 2015-10-26, at 18:40, Eggert, Lars wrote: > > On 2015-10-26, at 17:08, Pieper, Jeffrey E > wrote: > >> As a caveat, this was using default netperf message sizes. > > > > I get the same ~3 Gb/s with the default netperf sizes and driver 1.4.5. > > Now there is version 1.4.8 on the Intel website, but it doesn't change > things for me. > I had the opportunity to see similar numbers and behavior while using XL710 1.4.3 as of FreeBSD r291085 while in DPDK poll mode, but driver 1.2.8 as of r292035 was providing expected numbers. While removing rxcsum/txcsum did not provide differences, fully removing RSS + disabling rx/cxsum support provided better numbers. However now with driver 1.4.8 and the same set of hardware setup, except for a different transceiver, I can get 36Gbps/24Mpps with no further tweaks, so if you can replace your transceiver, shall be a different test as a starting point. > > > When you tcpdump during the run, do you see TSO/LRO in effect, i.e., do > you see "segments" > 32K in the trace? > > I still see no TSO/LRO in effect when tcpdump'ing on the receiver; note > how all the packets are 1448 bytes: > > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on ixl0, link-type EN10MB (Ethernet), capture size 262144 bytes > 17:02:42.328782 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [S], seq > 15244366, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478099 > ecr 0], length 0 > 17:02:42.328808 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [S.], seq > 1819579546, ack 15244367, win 65535, options [mss 1460,nop,wscale > 6,sackOK,TS val 3553932482 ecr 478099], length 0 > 17:02:42.328842 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [.], ack 1, win > 1040, options [nop,nop,TS val 478099 ecr 3553932482], length 0 > 17:02:42.329804 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [P.], seq 1:657, > ack 1, win 1040, options [nop,nop,TS val 478100 ecr 3553932482], length 656 > 17:02:42.331671 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [P.], seq 1:657, > ack 657, win 1040, options [nop,nop,TS val 3553932485 ecr 478100], length > 656 > 17:02:42.331717 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [S], seq > 1387798477, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478102 > ecr 0], length 0 > 17:02:42.331729 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [S.], seq > 4085135109, ack 1387798478, win 65535, options [mss 1460,nop,wscale > 6,sackOK,TS val 2829000022 ecr 478102], length 0 > 17:02:42.331781 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], ack 1, win > 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 0 > 17:02:42.331796 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 1:1449, > ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 > 17:02:42.331800 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 1449:2897, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331807 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 2897, > win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 > 17:02:42.331809 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 2897:4345, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331813 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 4345:5793, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331817 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 5793, > win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 > 17:02:42.331818 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 5793:7241, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331821 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 7241:8689, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331825 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 8689, > win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 > 17:02:42.331826 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 8689:10137, ack 1, win 1040, options [nop,nop,TS val 478102 ecr > 2829000022], length 1448 > 17:02:42.331829 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 10137:11585, ack 1, win 1040, options [nop,nop,TS val 478102 ecr > 2829000022], length 1448 > ... > > Doing the same trace over 10G ix interfaces shows most segments in the > 8-32K range, indicating that TSO/LRO are in use (and results in 9.9G > throughput). > > Lars >