From owner-freebsd-current@freebsd.org Mon Jul 13 07:44:09 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 808EB35BDBD for ; Mon, 13 Jul 2020 07:44:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-19.consmr.mail.gq1.yahoo.com (sonic306-19.consmr.mail.gq1.yahoo.com [98.137.68.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B4wdN3Ynsz4XtY for ; Mon, 13 Jul 2020 07:44:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: Ei.7EnEVM1mjWONkhy7iS22p5bVwhzS_I.FcWWYgWaHktgCTN9hn9wOIqgv8ApO A3Hz8K7yN9pL7kIak9GHPCq4wCFNRNFvzes5ITbMPkSH9uD3YupVLvrJW_k7y6POlPCv2VWjDkHm ouEPNZK4xz2JpdtskBAbZ_iysEAd8Ke2zM3CX03zFS28Dq955pE6Z4MsmRdsqKORtbJyBWw2A3Vr L_Pi1Zt4zOrfsXOo.ykxLYIys6n4EsDT996upBz7oGmH9sgv7O3zBx5FfBwXF9Xt3rKo2xgxvwuJ uw6NrkPe6p.5m5iLELD0bZ_V6Zod.zbBFybFN8PeuCoDUCcWWaxAZ7hJizn6XtPSHx9t.dJl0wUi hizRsou3SWs1Ixgh.ubSnvvQmaSLrbD.OeRbxWVr3FQoDyb4dd8fphJhAKbSu9qLy_XuwjQK0Tgt GKTH8Q2_oPb2lUhKJr26DaOCzmV1nlvnPtz5UH.Povu_.h4pDLBiD0M7WjRdVz6mtnjsk2rCTvdh Lz.fnI9Q6oSGemlrK5E.YMmZw.wToUHfyz3MHXuCl5MukpPBEN15mg9wOV.hho1qcwGF7C6_BkNi lgQLBRreHv0YFRxcpIkevtT1thGwGg79W2BJ7bF2MZo.CJ1m87GrvuaV7XMA3LK94um3AqVlIhIl sEC1_a.XW4squgiosGu_vEyXb5fFsD25CSAV9jskWt5SwvLiIoJJeT2WOcYogoPkNZSQWOZJLkfX 7n1RT8zjGq83cW2Pc_KXN5xUM92UGujCa2quVma6stBX2H7N2Qo3DO1u1dPSrmd7.BBukskQjz9B 5a5gVMeHZTnJ9ynrFRBcq7ks557ZrC.MEjqTqe9UBPtcKIWhF.3Gh_qagauB6.DjXUq2pTqbSBKJ FnOEQWFUlFpQ0SIJ39_OWCGpZ2VpdyF4IPEbBa3d980D8sWicVk5Iv.N_lArbSMXjAjsrcYxi62X uQQpaj27ikQ5hLQec3f3EGGnBt1gNaR11pFWo19BqR9oZ7n6fh6a4OvdMAp6nmBy5hKV98H7fsbv RQ714vO3SKvUdLuwYKwfRgwitGfjyQqLjCCEnCm_QWqJ_EmiRE3wke05Eak4x4sWm5LDFZAB9Q4Q ERbTGXA4hhnxd6tFf9_pw6vt.G2FzvCND029QTG855KlwttjIaLHeQF5t1dvvBBmx_Vy2tNP14r1 vDPgFg_GImx.MdZSgj58LBIutptM.Vo.mBBC31D620X6XWhiOWZTnBtGJRBpAIlhSBZeolitTnVV 9Lw6wXFWe_HUqxVHQXhYnmJvWQ5KCBRKhxCSotfvJLEqG5gsnjwtzCgEF2__qZAinkkInhywfTOF WdsxLFrDQ73hhB52pjUuoX2CvxJwohNSRP3GlHSd9vgla0bWAvLVS85USxqLBgHniTtyqrq6SzvR BYKWX78Pfo7In98a2GTsomkR42b3Gkt5pmu3dFZWY8yPkzDcSISLIwMfnWkM- Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Mon, 13 Jul 2020 07:44:06 +0000 Received: by smtp423.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 802c6e00640156d2aaa020c5511e02ea; Mon, 13 Jul 2020 07:44:04 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: slow USB 3.0 on -current From: Mark Millard In-Reply-To: <20200713045149.GL4213@funkthat.com> Date: Mon, 13 Jul 2020 00:44:02 -0700 Cc: FreeBSD Current Content-Transfer-Encoding: quoted-printable Message-Id: <1F4704D0-DD42-4DD2-A15C-D89FEF2FA382@yahoo.com> References: <9D8F806C-2F11-4338-9905-E91BBCDEFC01.ref@yahoo.com> <9D8F806C-2F11-4338-9905-E91BBCDEFC01@yahoo.com> <20200713045149.GL4213@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 4B4wdN3Ynsz4XtY X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.11 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.82:from]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-1.00)[-0.996]; NEURAL_HAM_MEDIUM(-1.02)[-1.017]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.82:from]; NEURAL_HAM_SHORT(-0.60)[-0.600]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2020 07:44:09 -0000 On 2020-Jul-12, at 21:51, John-Mark Gurney wrote: > Mark Millard wrote this message on Sun, Jul 12, 2020 at 18:26 -0700: >> John-Mark Gurney jmg at funkthat.com wrote on >> Sat Jul 11 22:44:36 UTC 2020 : >>=20 >>> I'm having issues getting good ethernet performance from a USB = ethernet >>> adapter (ure) under FreeBSD on an HP EliteDesk 705 G2 Mini[1]. It's = an >>> AMD PRO A10-8700B based system using the AMD A78 FCH chipset. >>>=20 >>> Under FreeBSD -current (r362596), 12.1-R and 11.4-R, the RealTek USB >>> adapter only gets around 10MB/sec performance. During the transfer, >>> the CPU usage is only around 3-5%, so it's definitely not CPU bound. >>>=20 >>> I have tested Windows 10 and NetBSD 9.0 performance, and both = provide >>> 100MB/sec+ w/o troubles. >>>=20 >>> I have attached dmesg from both FreeBSD -current and NetBSD 9.0. >>>=20 >>> Any hints on how to fix this? >>>=20 >>> This may be related, but I'm also having issues w/ booting when I = have >>> both a SD USB 2.0 card reader AND the ure plugged into USB 3.0 = ports. >>>=20 >>> If I move the SD card reader to USB 2.0, the umass device will = attach >>> and work. I have also attached a clip of the dmesg from that >>> happening. >>>=20 >>> Has anyone else seen this issue? Ideas or thoughts on how to = resolve >>> the performance issues? >>=20 >> It might prove useful to use iperf3 with >>=20 >> # iperf3 -s >>=20 >> on one machine and doing >>=20 >> # iperf3 -c ADDR >> . . . >> # iperf3 -R -c ADDR >> . . . >>=20 >> on the other. (That last swaps the >> sender/receiver status.) >>=20 >> All 3 commands will have output. The >> -s one will produce output for each of >> the -c ones. >>=20 >> The outputs for the sender(s) will include Cwnd >> (congestion window size) information that may >> be relevant. It will report bit rate and >> retry count sampling (and overall figures). >=20 > Here is the results for FreeBSD w/ USB3 ure. .80 is the USB3 > adapter side: > gold,pts,/home/jmg,502$iperf3 -c 192.168.0.80 > Connecting to host 192.168.0.80, port 5201 > [ 5] local 192.168.0.2 port 50042 connected to 192.168.0.80 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 8.94 MBytes 75.0 Mbits/sec 931 15.5 = KBytes > [ 5] 1.00-2.00 sec 9.98 MBytes 83.7 Mbits/sec 919 27.3 = KBytes > [ 5] 2.00-3.00 sec 9.95 MBytes 83.5 Mbits/sec 954 5.71 = KBytes > [ 5] 3.00-4.00 sec 9.97 MBytes 83.7 Mbits/sec 939 28.7 = KBytes > [ 5] 4.00-5.00 sec 9.97 MBytes 83.6 Mbits/sec 951 17.3 = KBytes > [ 5] 5.00-6.00 sec 9.99 MBytes 83.8 Mbits/sec 913 31.5 = KBytes > [ 5] 6.00-7.00 sec 9.96 MBytes 83.5 Mbits/sec 956 20.1 = KBytes > [ 5] 7.00-8.00 sec 10.0 MBytes 83.9 Mbits/sec 913 33.0 = KBytes > [ 5] 8.00-9.00 sec 9.97 MBytes 83.6 Mbits/sec 945 24.4 = KBytes > [ 5] 9.00-10.00 sec 9.99 MBytes 83.8 Mbits/sec 916 34.4 = KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 98.7 MBytes 82.8 Mbits/sec 9337 = sender > [ 5] 0.00-10.25 sec 98.7 MBytes 80.8 Mbits/sec = receiver >=20 > iperf Done. > gold,pts,/home/jmg,503$iperf3 -R -c 192.168.0.80 > Connecting to host 192.168.0.80, port 5201 > Reverse mode, remote host 192.168.0.80 is sending > [ 5] local 192.168.0.2 port 51024 connected to 192.168.0.80 port 5201 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 9.69 MBytes 81.3 Mbits/sec > [ 5] 1.00-2.00 sec 10.7 MBytes 89.8 Mbits/sec > [ 5] 2.00-3.00 sec 10.7 MBytes 89.8 Mbits/sec > [ 5] 3.00-4.00 sec 10.7 MBytes 89.8 Mbits/sec > [ 5] 4.00-5.00 sec 10.7 MBytes 89.8 Mbits/sec > [ 5] 5.00-6.00 sec 10.7 MBytes 89.8 Mbits/sec > [ 5] 6.00-7.00 sec 10.7 MBytes 89.8 Mbits/sec > [ 5] 7.00-8.00 sec 10.4 MBytes 87.6 Mbits/sec > [ 5] 8.00-9.00 sec 10.7 MBytes 89.9 Mbits/sec > [ 5] 9.00-10.00 sec 10.7 MBytes 89.8 Mbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 106 MBytes 88.9 Mbits/sec 1381 = sender > [ 5] 0.00-10.00 sec 106 MBytes 88.7 Mbits/sec = receiver >=20 > iperf Done. The "iperf3 -s" should have had output with the Cwnd figures for the "Reverse mode" case above (and the distribution for the 1381 Retr total). They might not match when the earlier figures that you did report for the non-Reverse mode. > As you can see, it matches what I measured earlier. >=20 > And just to prove that the machine CAN move 100MB/sec, I've run iperf3 > using the onboard wired ethernet... I need multiple interfaces, which = is > why I'm bothering trying to get USB3 ethernet working. >=20 > This is using the onboard bge interface. It's IP is .79: > gold,pts,/home/jmg,507$iperf3 -c 192.168.0.79 > Connecting to host 192.168.0.79, port 5201 > [ 5] local 192.168.0.2 port 61500 connected to 192.168.0.79 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 101 MBytes 850 Mbits/sec 0 488 = KBytes > [ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec 0 488 = KBytes > [ 5] 2.00-3.00 sec 112 MBytes 943 Mbits/sec 0 731 = KBytes > [ 5] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 0 731 = KBytes > [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 731 = KBytes > [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 731 = KBytes > [ 5] 6.00-7.00 sec 112 MBytes 940 Mbits/sec 0 731 = KBytes > [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 731 = KBytes > [ 5] 8.00-9.00 sec 112 MBytes 940 Mbits/sec 0 731 = KBytes > [ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 0 731 = KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 1.08 GBytes 932 Mbits/sec 0 = sender > [ 5] 0.00-10.01 sec 1.08 GBytes 931 Mbits/sec = receiver >=20 > iperf Done. > gold,pts,/home/jmg,508$iperf3 -R -c 192.168.0.79 > Connecting to host 192.168.0.79, port 5201 > Reverse mode, remote host 192.168.0.79 is sending > [ 5] local 192.168.0.2 port 61726 connected to 192.168.0.79 port 5201 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 97.0 MBytes 814 Mbits/sec > [ 5] 1.00-2.00 sec 109 MBytes 917 Mbits/sec > [ 5] 2.00-3.00 sec 110 MBytes 920 Mbits/sec > [ 5] 3.00-4.00 sec 109 MBytes 917 Mbits/sec > [ 5] 4.00-5.00 sec 110 MBytes 920 Mbits/sec > [ 5] 5.00-6.00 sec 109 MBytes 916 Mbits/sec > [ 5] 6.00-7.00 sec 110 MBytes 919 Mbits/sec > [ 5] 7.00-8.00 sec 109 MBytes 917 Mbits/sec > [ 5] 8.00-9.00 sec 110 MBytes 919 Mbits/sec > [ 5] 9.00-10.00 sec 109 MBytes 916 Mbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.03 sec 1.06 GBytes 905 Mbits/sec 2866 = sender > [ 5] 0.00-10.00 sec 1.06 GBytes 907 Mbits/sec = receiver >=20 > iperf Done. Similar "iperf3 -s" output point here. >> Comparing the output of using iperf3 under >> NetBSD 9.0 or Windows 10 could be instructive. >=20 > And NetBSD 9.0. Here, NetBSD got assigned .50, but it's also using > the USB3 adapter. >=20 > gold,pts,/home/jmg,505$iperf3 -c 192.168.0.50 > Connecting to host 192.168.0.50, port 5201 > [ 5] local 192.168.0.2 port 55535 connected to 192.168.0.50 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 96.7 MBytes 811 Mbits/sec 0 50.9 = KBytes > [ 5] 1.00-2.00 sec 112 MBytes 938 Mbits/sec 0 82.1 = KBytes > [ 5] 2.00-3.00 sec 112 MBytes 938 Mbits/sec 14 114 = KBytes > [ 5] 3.00-4.00 sec 110 MBytes 924 Mbits/sec 60 146 = KBytes > [ 5] 4.00-5.00 sec 112 MBytes 938 Mbits/sec 107 178 = KBytes > [ 5] 5.00-6.00 sec 112 MBytes 939 Mbits/sec 122 193 = KBytes > [ 5] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 144 193 = KBytes > [ 5] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 165 193 = KBytes > [ 5] 8.00-9.00 sec 112 MBytes 939 Mbits/sec 144 193 = KBytes > [ 5] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 165 193 = KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 1.08 GBytes 924 Mbits/sec 921 = sender > [ 5] 0.00-10.00 sec 1.08 GBytes 924 Mbits/sec = receiver >=20 > iperf Done. > gold,pts,/home/jmg,506$iperf3 -R -c 192.168.0.50 > Connecting to host 192.168.0.50, port 5201 > Reverse mode, remote host 192.168.0.50 is sending > [ 5] local 192.168.0.2 port 55691 connected to 192.168.0.50 port 5201 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 78.3 MBytes 657 Mbits/sec > [ 5] 1.00-2.00 sec 88.4 MBytes 742 Mbits/sec > [ 5] 2.00-3.00 sec 88.5 MBytes 742 Mbits/sec > [ 5] 3.00-4.00 sec 88.4 MBytes 741 Mbits/sec > [ 5] 4.00-5.00 sec 88.7 MBytes 744 Mbits/sec > [ 5] 5.00-6.00 sec 88.2 MBytes 740 Mbits/sec > [ 5] 6.00-7.00 sec 87.8 MBytes 737 Mbits/sec > [ 5] 7.00-8.00 sec 88.5 MBytes 742 Mbits/sec > [ 5] 8.00-9.00 sec 89.0 MBytes 746 Mbits/sec > [ 5] 9.00-10.00 sec 88.8 MBytes 745 Mbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 875 MBytes 734 Mbits/sec 0 = sender > [ 5] 0.00-10.00 sec 874 MBytes 734 Mbits/sec = receiver >=20 > iperf Done. Similar "iperf3 -s" output point here. > As you can see, both match approximately what I measured other = methods, > so, it's definitely not the way I measured performance. >=20 >> My observation would be that neither type >> of USB3 Ethernet adapter that I've tried >=20 > What is the chipset that you tried? One of the earlier ones that I > tried was an axe iirc, and was limited to around 500Mbps or so... Hmm. I only seem to be able to find one type. Its been a while since I've used the other and I do not know where it is at. For what I found: ugen0.2: at usbus0 axge0 on uhub0 axge0: on usbus0 miibus1: on axge0 rgephy0: PHY 3 on = miibus1 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, = 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, = 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow (I have access to more than one instance of the above.) The iperf3 output that I reported was for using of of the above. Note that when the USB3 EtherNet was reciveing Cwnd was reported as 29.8 KBytes or smaller for the example run, much like your output reporting 34.4 KBytes or less for the example run. This may suggest some common constraint across various USB3 EtherNet devices. The Cwnd figures are probably too small to get near 900 Mbit/s+. But, even with a (smaller but) similar Cwnd figure my example was getting faster transfers than your example. I got smaller Retr figures as well. It leaves me wondering if there are packets being rejected in your context that are not in my context. If what I reported would still be too slow, there may be two (or more) points to be addressed to get things going fast enough for you. You might be able to avoid one of the points by using a type of device that already does somewhat better. May be ask for the fastest examples folks have observed? >> (different chipsets) get anywhere near >> 100 MByte/s when ifconfig reports >> 1000baseT . The Cwnd figures >> are smaller than for the built-in Ethernets >> that manage much faster overall transfer >> rates. >=20 > [...] >=20 >> I'll note that between machines with built-in EtherNet >> that can sustain fast transfers overall, the Cwnd figures >> tend to vary but can reach 1 MBytes+. The Retr counts >> tend to still exist. >>=20 >> By contrast, when the USB3 EtherNet is receiving above, >> the maximum Cwnd reported above for the sender at the >> time was: 29.8 KBytes. >>=20 >> I have not tried NetBSD, Windows 10, or Linux comparisons. >=20 > As you can see above, NetBSD easily achieves around 8-10x the > speed using the exact same USB3 device as FreeBSD does, so the > hardware CAN push the speeds, just FreeBSD cannot. The small Cwnd figures like 34.4 KBytes suggest that a small receiver window (Rwnd) might be specified in the TCP header that the destination provided for that transfer direction. Cwnd can increase up to the Rwnd unless duplicate ACKs or timeouts occur, as I understand. (I'm no expert.) This is part of the reason I thought that posting the output (with Cwnd) for both transfer directions could be of use: it gives a hint about what is controlling the Cwnd if the two directions have widely different figures vs. if both directions get similarly small figures. Comparisons with the other OS's figures in both directions could possibly also be suggestive, or so I hoped. May be the comparison with the figures that I reported gives someone some hints about what might be going on in the two contexts. > Hence, my original post, what can I do to possibly get FreeBSD's > performance up to what the hardware can achieve? >=20 Hopefully my notes prove of some use --but I'm not likely to directly solve the problem. It would be handy for me if USB3 EtherNet performed significantly better than I have observed. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)