Date: Mon, 13 Jul 2020 12:03:49 -0700 From: John-Mark Gurney <jmg@funkthat.com> To: Mark Millard <marklmi@yahoo.com> Cc: FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: slow USB 3.0 on -current Message-ID: <20200713190349.GM4213@funkthat.com> In-Reply-To: <1F4704D0-DD42-4DD2-A15C-D89FEF2FA382@yahoo.com> References: <9D8F806C-2F11-4338-9905-E91BBCDEFC01.ref@yahoo.com> <9D8F806C-2F11-4338-9905-E91BBCDEFC01@yahoo.com> <20200713045149.GL4213@funkthat.com> <1F4704D0-DD42-4DD2-A15C-D89FEF2FA382@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Mark Millard wrote this message on Mon, Jul 13, 2020 at 00:44 -0700:
> On 2020-Jul-12, at 21:51, John-Mark Gurney <jmg at funkthat.com> wrote:
>
> > Mark Millard wrote this message on Sun, Jul 12, 2020 at 18:26 -0700:
> >> John-Mark Gurney jmg at funkthat.com wrote on
> >> Sat Jul 11 22:44:36 UTC 2020 :
> >>
> >>> I'm having issues getting good ethernet performance from a USB ethernet
> >>> adapter (ure) under FreeBSD on an HP EliteDesk 705 G2 Mini[1]. It's an
> >>> AMD PRO A10-8700B based system using the AMD A78 FCH chipset.
> >>>
> >>> Under FreeBSD -current (r362596), 12.1-R and 11.4-R, the RealTek USB
> >>> adapter only gets around 10MB/sec performance. During the transfer,
> >>> the CPU usage is only around 3-5%, so it's definitely not CPU bound.
> >>>
> >>> I have tested Windows 10 and NetBSD 9.0 performance, and both provide
> >>> 100MB/sec+ w/o troubles.
> >>>
> >>> I have attached dmesg from both FreeBSD -current and NetBSD 9.0.
> >>>
> >>> Any hints on how to fix this?
> >>>
> >>> This may be related, but I'm also having issues w/ booting when I have
> >>> both a SD USB 2.0 card reader AND the ure plugged into USB 3.0 ports.
> >>>
> >>> If I move the SD card reader to USB 2.0, the umass device will attach
> >>> and work. I have also attached a clip of the dmesg from that
> >>> happening.
> >>>
> >>> Has anyone else seen this issue? Ideas or thoughts on how to resolve
> >>> the performance issues?
> >>
> >> It might prove useful to use iperf3 with
> >>
> >> # iperf3 -s
> >>
> >> on one machine and doing
> >>
> >> # iperf3 -c ADDR
> >> . . .
> >> # iperf3 -R -c ADDR
> >> . . .
> >>
> >> on the other. (That last swaps the
> >> sender/receiver status.)
> >>
> >> All 3 commands will have output. The
> >> -s one will produce output for each of
> >> the -c ones.
> >>
> >> The outputs for the sender(s) will include Cwnd
> >> (congestion window size) information that may
> >> be relevant. It will report bit rate and
> >> retry count sampling (and overall figures).
[...]
> The "iperf3 -s" should have had output with the Cwnd
> figures for the "Reverse mode" case above (and the
> distribution for the 1381 Retr total). They might
> not match when the earlier figures that you did report
> for the non-Reverse mode.
If you can tell how the Cwnd figures would help you figure out how to
make the USB3 bus run faster, I'll spend the time to retest and give
you the numbers, but I don't see how they can...
[...]
> > As you can see, both match approximately what I measured other methods,
> > so, it's definitely not the way I measured performance.
> >
> >> My observation would be that neither type
> >> of USB3 Ethernet adapter that I've tried
> >
> > What is the chipset that you tried? One of the earlier ones that I
> > tried was an axe iirc, and was limited to around 500Mbps or so...
>
> Hmm. I only seem to be able to find one type. Its been a
> while since I've used the other and I do not know where
> it is at. For what I found:
>
> ugen0.2: <ASIX Elec. AX88179> at usbus0
> axge0 on uhub0
> axge0: <NetworkInterface> on usbus0
> miibus1: <MII bus> on axge0
> rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 3 on miibus1
> rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
>
> (I have access to more than one instance of the above.)
Yeah, these are the ones that are known to not be able to get close
to gige speeds, unlike the RealTek one that I am using now...
I forgot that axge is the gigabit version of axe...
> The iperf3 output that I reported was for using
> of of the above. Note that when the USB3 EtherNet
> was reciveing Cwnd was reported as 29.8 KBytes
> or smaller for the example run, much like your
> output reporting 34.4 KBytes or less for the
> example run.
They grew to match the speeds that the link could do..
> This may suggest some common constraint across various
> USB3 EtherNet devices. The Cwnd figures are probably
> too small to get near 900 Mbit/s+.
As you can see, the NetBSD results was able to grow the
Cwnd large enough to obtain performance...
The stats that I provided were from the non-USB3 machine, and for
tx'ing to matches the issue I raised in the original post... I could
provide the Cwnd, but I don't see how that will debug a USB3 speed
issue...
The stats show that the Cwnd can grow on other OS's (NetBSD), and
on wired (bge) fine.
> But, even with a (smaller but) similar Cwnd figure
> my example was getting faster transfers than your
> example. I got smaller Retr figures as well. It
> leaves me wondering if there are packets being
> rejected in your context that are not in my
> context.
ping times to the machine via USB3 is higher than native gige, but
that isn't too surprising due to the extra latency introduced by them..
it's:
round-trip min/avg/max/stddev = 0.743/0.826/0.963/0.074 ms
Where as to a slower machine (PINE A64-LTS, arm64) with a
couple extra switches in between:
round-trip min/avg/max/stddev = 0.199/0.230/0.254/0.019 ms
> If what I reported would still be too slow, there
> may be two (or more) points to be addressed to
> get things going fast enough for you. You
> might be able to avoid one of the points by
> using a type of device that already does somewhat
> better. May be ask for the fastest examples
> folks have observed?
>
> >> (different chipsets) get anywhere near
> >> 100 MByte/s when ifconfig reports
> >> 1000baseT <full-duplex>. The Cwnd figures
> >> are smaller than for the built-in Ethernets
> >> that manage much faster overall transfer
> >> rates.
> >
> > [...]
> >
> >> I'll note that between machines with built-in EtherNet
> >> that can sustain fast transfers overall, the Cwnd figures
> >> tend to vary but can reach 1 MBytes+. The Retr counts
> >> tend to still exist.
> >>
> >> By contrast, when the USB3 EtherNet is receiving above,
> >> the maximum Cwnd reported above for the sender at the
> >> time was: 29.8 KBytes.
> >>
> >> I have not tried NetBSD, Windows 10, or Linux comparisons.
> >
> > As you can see above, NetBSD easily achieves around 8-10x the
> > speed using the exact same USB3 device as FreeBSD does, so the
> > hardware CAN push the speeds, just FreeBSD cannot.
>
> The small Cwnd figures like 34.4 KBytes suggest that a
> small receiver window (Rwnd) might be specified in the
> TCP header that the destination provided for that
> transfer direction. Cwnd can increase up to the
> Rwnd unless duplicate ACKs or timeouts occur, as I
> understand. (I'm no expert.)
>
> This is part of the reason I thought that posting the
> output (with Cwnd) for both transfer directions could
> be of use: it gives a hint about what is controlling
> the Cwnd if the two directions have widely different
> figures vs. if both directions get similarly small
> figures. Comparisons with the other OS's figures in
> both directions could possibly also be suggestive,
> or so I hoped.
>
> May be the comparison with the figures that I reported
> gives someone some hints about what might be going on
> in the two contexts.
>
> > Hence, my original post, what can I do to possibly get FreeBSD's
> > performance up to what the hardware can achieve?
> >
>
> Hopefully my notes prove of some use --but I'm not likely
> to directly solve the problem. It would be handy for
> me if USB3 EtherNet performed significantly better than
> I have observed.
Just for giggles, I used iperf3 to test UDP performance to eliminate
TCP, and yep, the physical interface is limited to about 91.1Mbps...
When usb3 interface transmits, I get:
gold,pts,/home/jmg,506$iperf3 -R -u -b 1000M -c 192.168.0.80
Connecting to host 192.168.0.80, port 5201
Reverse mode, remote host 192.168.0.80 is sending
[ 5] local 192.168.0.2 port 42932 connected to 192.168.0.80 port 5201
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 11.0 MBytes 92.1 Mbits/sec 0.271 ms 773186/781071 (99%)
[ 5] 1.00-2.00 sec 10.9 MBytes 91.1 Mbits/sec 0.303 ms 235306/243103 (97%)
[ 5] 2.00-3.00 sec 10.9 MBytes 91.1 Mbits/sec 0.345 ms 236328/244125 (97%)
[ 5] 3.00-4.00 sec 10.9 MBytes 91.1 Mbits/sec 0.354 ms 235799/243598 (97%)
[ 5] 4.00-5.00 sec 10.9 MBytes 91.1 Mbits/sec 0.404 ms 233614/241411 (97%)
[ 5] 5.00-6.00 sec 10.9 MBytes 91.1 Mbits/sec 0.464 ms 235559/243356 (97%)
[ 5] 6.00-7.00 sec 10.9 MBytes 91.1 Mbits/sec 0.515 ms 236272/244070 (97%)
[ 5] 7.00-8.00 sec 10.9 MBytes 91.1 Mbits/sec 0.139 ms 234354/242152 (97%)
[ 5] 8.00-9.00 sec 10.9 MBytes 91.1 Mbits/sec 0.150 ms 236887/244684 (97%)
[ 5] 9.00-10.00 sec 10.9 MBytes 91.1 Mbits/sec 0.149 ms 236459/244257 (97%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-12.22 sec 133 MBytes 91.1 Mbits/sec 0.000 ms 0/2974757 (0%) sender
[ 5] 0.00-10.00 sec 109 MBytes 91.2 Mbits/sec 0.149 ms 2893764/2971827 (97%) receiver
iperf Done.
and when the USB3 ethernet receives:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.2, port 20603
[ 5] local 192.168.0.80 port 5201 connected to 192.168.0.2 port 25350
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 9.42 MBytes 79.0 Mbits/sec 0.186 ms 111653/118418 (94%)
[ 5] 1.00-2.00 sec 10.9 MBytes 91.1 Mbits/sec 0.165 ms 229690/237489 (97%)
[ 5] 2.00-3.00 sec 10.9 MBytes 91.1 Mbits/sec 0.459 ms 236066/243864 (97%)
[ 5] 3.00-4.00 sec 10.9 MBytes 91.1 Mbits/sec 0.219 ms 228274/236072 (97%)
[ 5] 4.00-5.00 sec 10.9 MBytes 91.1 Mbits/sec 0.168 ms 224079/231877 (97%)
[ 5] 5.00-6.00 sec 10.9 MBytes 91.1 Mbits/sec 0.157 ms 233127/240924 (97%)
[ 5] 6.00-7.00 sec 10.9 MBytes 91.1 Mbits/sec 0.157 ms 232302/240100 (97%)
[ 5] 7.00-8.00 sec 10.9 MBytes 91.1 Mbits/sec 0.376 ms 232491/240289 (97%)
[ 5] 8.00-9.00 sec 10.9 MBytes 91.1 Mbits/sec 0.197 ms 230407/238205 (97%)
[ 5] 9.00-10.00 sec 10.9 MBytes 91.1 Mbits/sec 0.158 ms 243574/251372 (97%)
[ 5] 10.00-10.38 sec 1.42 MBytes 31.6 Mbits/sec 0.136 ms 30894/31914 (97%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.38 sec 109 MBytes 87.8 Mbits/sec 0.136 ms 2232557/2310524 (97%) receiver
The transmitter reports:
[ 5] 0.00-10.00 sec 1.12 GBytes 959 Mbits/sec 0.000 ms 0/2311155 (0%) sender
so, it is able to transmit at close to gigabit speed.
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200713190349.GM4213>