Date: Tue, 29 Sep 2015 23:10:19 -0700 From: Kevin Oberman <rkoberman@gmail.com> To: javocado <javocado@gmail.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Network tuning help needed - asymmetric speed Message-ID: <CAN6yY1tVQU2Om8kvyz_ueCX-mpbo88PnaE2qSs3CY2W1vS=LMg@mail.gmail.com> In-Reply-To: <CAP1HOmSMne6YyNGAXS-jDMGoYEsPUYnpRO3dHvrp2qP%2BZ3btmw@mail.gmail.com> References: <CAP1HOmSMne6YyNGAXS-jDMGoYEsPUYnpRO3dHvrp2qP%2BZ3btmw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Sep 29, 2015 at 4:22 PM, javocado <javocado@gmail.com> wrote: > I am trying to figure out what set of tunables need to be tweaked in order > to get an Internet connection operating at decent speed in _both_ > directions. Here are my particulars: > > Source: FreeBSD 8.4 AMD > Target: Ubuntu 14.04 x64 > > > Iperf tests: > > Source -> Target: > > # iperf -t10 -P1 -i1 -c xxxxxxxx > ------------------------------------------------------------ > Client connecting to xxxxxxxxx, TCP port 5001 > TCP window size: 2.16 MByte (default) > ------------------------------------------------------------ > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 1.0 sec 26.4 MBytes 221 Mbits/sec > [ 3] 1.0- 2.0 sec 9.12 MBytes 76.5 Mbits/sec > [ 3] 2.0- 3.0 sec 3.38 MBytes 28.3 Mbits/sec > [ 3] 3.0- 4.0 sec 3.88 MBytes 32.5 Mbits/sec > [ 3] 4.0- 5.0 sec 1.62 MBytes 13.6 Mbits/sec > [ 3] 5.0- 6.0 sec 2.38 MBytes 19.9 Mbits/sec > [ 3] 6.0- 7.0 sec 2.88 MBytes 24.1 Mbits/sec > [ 3] 7.0- 8.0 sec 1.00 MBytes 8.39 Mbits/sec > [ 3] 8.0- 9.0 sec 1.12 MBytes 9.44 Mbits/sec > [ 3] 9.0-10.0 sec 1.88 MBytes 15.7 Mbits/sec > [ 3] 0.0-10.1 sec 53.9 MBytes 44.9 Mbits/sec > > subsequent identical iperf tests: > > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 1.0 sec 3.50 MBytes 29.4 Mbits/sec > [ 3] 1.0- 2.0 sec 3.75 MBytes 31.5 Mbits/sec > [ 3] 2.0- 3.0 sec 6.38 MBytes 53.5 Mbits/sec > [ 3] 3.0- 4.0 sec 2.12 MBytes 17.8 Mbits/sec > [ 3] 4.0- 5.0 sec 3.25 MBytes 27.3 Mbits/sec > [ 3] 5.0- 6.0 sec 4.25 MBytes 35.7 Mbits/sec > [ 3] 6.0- 7.0 sec 1.88 MBytes 15.7 Mbits/sec > [ 3] 7.0- 8.0 sec 4.12 MBytes 34.6 Mbits/sec > [ 3] 8.0- 9.0 sec 1.25 MBytes 10.5 Mbits/sec > [ 3] 9.0-10.0 sec 1.00 MBytes 8.39 Mbits/sec > [ 3] 0.0-10.1 sec 31.8 MBytes 26.4 Mbits/sec > > > Target -> Source: > > # iperf -t10 -P1 -i1 -c xxxxxx > ------------------------------------------------------------ > Client connecting to xxxxxxxx, TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 1.0 sec 90.8 MBytes 761 Mbits/sec > [ 3] 1.0- 2.0 sec 104 MBytes 871 Mbits/sec > [ 3] 2.0- 3.0 sec 107 MBytes 900 Mbits/sec > [ 3] 3.0- 4.0 sec 96.0 MBytes 805 Mbits/sec > [ 3] 4.0- 5.0 sec 97.8 MBytes 820 Mbits/sec > [ 3] 5.0- 6.0 sec 102 MBytes 857 Mbits/sec > [ 3] 6.0- 7.0 sec 104 MBytes 873 Mbits/sec > [ 3] 7.0- 8.0 sec 104 MBytes 868 Mbits/sec > [ 3] 8.0- 9.0 sec 104 MBytes 873 Mbits/sec > [ 3] 9.0-10.0 sec 104 MBytes 871 Mbits/sec > [ 3] 0.0-10.0 sec 1014 MBytes 850 Mbits/sec > > > Traceroutes: > > Source -> Target: > > traceroute to xxxxxxx 64 hops max, 52 byte packets > 1 6.978 ms 1.989 ms 2.002 ms > 2 10.954 ms 0.983 ms > 1.957 ms > 3 52.189 ms 5.998 ms 22.044 ms > 4 26.091 ms 22.056 ms 24.017 ms > 5 21.492 ms 21.029 ms > 6 21.047 ms 21.093 ms 21.998 ms > 7 20.897 ms 20.744 ms 23.042 ms > 8 20.699 ms 20.655 ms 20.526 ms > > > Target -> Source: > traceroute to xxxxxx, 30 hops max, 60 byte packets > 1 0.782 ms 0.761 ms 0.784 ms > 2 1.072 ms 1.028 ms 1.002 ms > 3 0.689 ms 0.665 ms 0.796 ms > 4 42.513 ms 42.596 ms 42.568 ms > 5 0.917 ms 0.895 ms 0.866 ms > 6 20.209 ms 28.508 ms 28.507 ms > 7 20.346 ms 20.352 ms 20.392 ms > 8 30.392 ms 30.404 ms 30.387 ms > 9 20.542 ms 20.720 ms 20.899 ms > > > We thought there could just be a bad hop or perhaps it was hardware at > fault, but when we run our iperf from a different source (a stock Ubuntu > 12.04 box) connected to the same switch as the FreeBSD source, we get > great/better/acceptable results: > > > (Ubuntu 12.04) Source -> Target: > > $ iperf -t10 -P1 -i1 -w1M -c xxxxxxx > ------------------------------------------------------------ > Client connecting to xxxxxxxx, TCP port 5001 > TCP window size: 256 KByte (WARNING: requested 1.00 MByte) > ------------------------------------------------------------ > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 1.0 sec 9.25 MBytes 77.6 Mbits/sec > [ 3] 1.0- 2.0 sec 10.2 MBytes 86.0 Mbits/sec > [ 3] 2.0- 3.0 sec 10.6 MBytes 89.1 Mbits/sec > [ 3] 3.0- 4.0 sec 11.1 MBytes 93.3 Mbits/sec > [ 3] 4.0- 5.0 sec 11.2 MBytes 94.4 Mbits/sec > [ 3] 5.0- 6.0 sec 10.8 MBytes 90.2 Mbits/sec > [ 3] 6.0- 7.0 sec 11.5 MBytes 96.5 Mbits/sec > [ 3] 7.0- 8.0 sec 11.2 MBytes 94.4 Mbits/sec > [ 3] 8.0- 9.0 sec 11.0 MBytes 92.3 Mbits/sec > [ 3] 9.0-10.0 sec 11.1 MBytes 93.3 Mbits/sec > [ 3] 0.0-10.0 sec 108 MBytes 90.7 Mbits/sec > > > So, it would seem that FreeBSD is very well tuned for Target -> Source > traffic (nearly 1Gbps), but the Source -> Target direction is terrible > (20Mbps). Since the RTT is the same and the # of hops is virtually > unchanged, we don't see any reason the speed should be different outbound, > and further we see that when we use Ubuntu -> Target, we get decent speed > (100Mbps). Thus we conclude the hardware and hops must not the issue. So, > what we're missing in our FreeBSD tuning that is making the outbound > transfers so slow (while the inbound transfers are great)? > > Thanks > > [See original message for the details which I elided] First, I know that I was able to most data both from and to a FreeBSD system under v8 over a transcontinental path (San Jose to New York City) at multi-gigabit rates. It took a bit of tuning, but was not difficult. I see several possible issues. What interface are you using There have been performance issues with drivers on some interfaces. Many are fixed in more recent versions of FreeBSD. 8.4 is past it's EOL and fixes are not being made any longer. (Note, I think that this is unlikely, but it is possible.) It could also be bad hardware. Was the Ubuntu system connected to the same port using the same cable? Can you boot Ubuntu on the same hardware and confirm whether it works as well on the same hardware? Unless you can do this or test an on another system running FreeBSD with the same results, you really can't exclude hardware as the source of the problem. Tuning could be an issue. It is hard to tell without a LOT more information. Collecting data with siftr(4) is quite simple and can provide very useful data as to exactly what is impacting performance. The man page describes it and how to run it quite clearly. For lots of tuning recommendations for truly high performance, see FasterData <http://fasterdata.es.net/>. Unfortunately, since my retirement five years ago, I believe that all work has moved to Linux, so there may not be much specific to FreeBSD. I can say that, with properly functioning hardware, it was not at all hard to exceed 900 Mbps on a GigE on FreeBSD 8. I retired before 9 was released, but I am pretty sure that this would not be a problem on newer versions, either. I should also admit that, after four years away from dealing with these issues, I am far less competent than I was, so I may not be as much help as I would like. -- Kevin Oberman, Part time goatherd and retired Network Engineer
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1tVQU2Om8kvyz_ueCX-mpbo88PnaE2qSs3CY2W1vS=LMg>