Date: Wed, 30 Sep 2015 10:32:17 -0700 From: Adrian Chadd <adrian.chadd@gmail.com> To: javocado <javocado@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Network tuning help needed - asymmetric speed Message-ID: <CAJ-Vmom77oj=oZNukiXzjVQ1tHcfLcFFCo9hjkz73iaAu3BEoQ@mail.gmail.com> In-Reply-To: <CAP1HOmSMne6YyNGAXS-jDMGoYEsPUYnpRO3dHvrp2qP%2BZ3btmw@mail.gmail.com> References: <CAP1HOmSMne6YyNGAXS-jDMGoYEsPUYnpRO3dHvrp2qP%2BZ3btmw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
hi,, Please retry with freebsd-head or freebsd stable/10. Bugs keep being fixed every day in those drivers. :) -a On 29 September 2015 at 16:22, javocado <javocado@gmail.com> wrote: > I am trying to figure out what set of tunables need to be tweaked in order > to get an Internet connection operating at decent speed in _both_ > directions. Here are my particulars: > > Source: FreeBSD 8.4 AMD > Target: Ubuntu 14.04 x64 > > > Iperf tests: > > Source -> Target: > > # iperf -t10 -P1 -i1 -c xxxxxxxx > ------------------------------------------------------------ > Client connecting to xxxxxxxxx, TCP port 5001 > TCP window size: 2.16 MByte (default) > ------------------------------------------------------------ > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 1.0 sec 26.4 MBytes 221 Mbits/sec > [ 3] 1.0- 2.0 sec 9.12 MBytes 76.5 Mbits/sec > [ 3] 2.0- 3.0 sec 3.38 MBytes 28.3 Mbits/sec > [ 3] 3.0- 4.0 sec 3.88 MBytes 32.5 Mbits/sec > [ 3] 4.0- 5.0 sec 1.62 MBytes 13.6 Mbits/sec > [ 3] 5.0- 6.0 sec 2.38 MBytes 19.9 Mbits/sec > [ 3] 6.0- 7.0 sec 2.88 MBytes 24.1 Mbits/sec > [ 3] 7.0- 8.0 sec 1.00 MBytes 8.39 Mbits/sec > [ 3] 8.0- 9.0 sec 1.12 MBytes 9.44 Mbits/sec > [ 3] 9.0-10.0 sec 1.88 MBytes 15.7 Mbits/sec > [ 3] 0.0-10.1 sec 53.9 MBytes 44.9 Mbits/sec > > subsequent identical iperf tests: > > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 1.0 sec 3.50 MBytes 29.4 Mbits/sec > [ 3] 1.0- 2.0 sec 3.75 MBytes 31.5 Mbits/sec > [ 3] 2.0- 3.0 sec 6.38 MBytes 53.5 Mbits/sec > [ 3] 3.0- 4.0 sec 2.12 MBytes 17.8 Mbits/sec > [ 3] 4.0- 5.0 sec 3.25 MBytes 27.3 Mbits/sec > [ 3] 5.0- 6.0 sec 4.25 MBytes 35.7 Mbits/sec > [ 3] 6.0- 7.0 sec 1.88 MBytes 15.7 Mbits/sec > [ 3] 7.0- 8.0 sec 4.12 MBytes 34.6 Mbits/sec > [ 3] 8.0- 9.0 sec 1.25 MBytes 10.5 Mbits/sec > [ 3] 9.0-10.0 sec 1.00 MBytes 8.39 Mbits/sec > [ 3] 0.0-10.1 sec 31.8 MBytes 26.4 Mbits/sec > > > Target -> Source: > > # iperf -t10 -P1 -i1 -c xxxxxx > ------------------------------------------------------------ > Client connecting to xxxxxxxx, TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 1.0 sec 90.8 MBytes 761 Mbits/sec > [ 3] 1.0- 2.0 sec 104 MBytes 871 Mbits/sec > [ 3] 2.0- 3.0 sec 107 MBytes 900 Mbits/sec > [ 3] 3.0- 4.0 sec 96.0 MBytes 805 Mbits/sec > [ 3] 4.0- 5.0 sec 97.8 MBytes 820 Mbits/sec > [ 3] 5.0- 6.0 sec 102 MBytes 857 Mbits/sec > [ 3] 6.0- 7.0 sec 104 MBytes 873 Mbits/sec > [ 3] 7.0- 8.0 sec 104 MBytes 868 Mbits/sec > [ 3] 8.0- 9.0 sec 104 MBytes 873 Mbits/sec > [ 3] 9.0-10.0 sec 104 MBytes 871 Mbits/sec > [ 3] 0.0-10.0 sec 1014 MBytes 850 Mbits/sec > > > Traceroutes: > > Source -> Target: > > traceroute to xxxxxxx 64 hops max, 52 byte packets > 1 6.978 ms 1.989 ms 2.002 ms > 2 10.954 ms 0.983 ms > 1.957 ms > 3 52.189 ms 5.998 ms 22.044 ms > 4 26.091 ms 22.056 ms 24.017 ms > 5 21.492 ms 21.029 ms > 6 21.047 ms 21.093 ms 21.998 ms > 7 20.897 ms 20.744 ms 23.042 ms > 8 20.699 ms 20.655 ms 20.526 ms > > > Target -> Source: > traceroute to xxxxxx, 30 hops max, 60 byte packets > 1 0.782 ms 0.761 ms 0.784 ms > 2 1.072 ms 1.028 ms 1.002 ms > 3 0.689 ms 0.665 ms 0.796 ms > 4 42.513 ms 42.596 ms 42.568 ms > 5 0.917 ms 0.895 ms 0.866 ms > 6 20.209 ms 28.508 ms 28.507 ms > 7 20.346 ms 20.352 ms 20.392 ms > 8 30.392 ms 30.404 ms 30.387 ms > 9 20.542 ms 20.720 ms 20.899 ms > > > We thought there could just be a bad hop or perhaps it was hardware at > fault, but when we run our iperf from a different source (a stock Ubuntu > 12.04 box) connected to the same switch as the FreeBSD source, we get > great/better/acceptable results: > > > (Ubuntu 12.04) Source -> Target: > > $ iperf -t10 -P1 -i1 -w1M -c xxxxxxx > ------------------------------------------------------------ > Client connecting to xxxxxxxx, TCP port 5001 > TCP window size: 256 KByte (WARNING: requested 1.00 MByte) > ------------------------------------------------------------ > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 1.0 sec 9.25 MBytes 77.6 Mbits/sec > [ 3] 1.0- 2.0 sec 10.2 MBytes 86.0 Mbits/sec > [ 3] 2.0- 3.0 sec 10.6 MBytes 89.1 Mbits/sec > [ 3] 3.0- 4.0 sec 11.1 MBytes 93.3 Mbits/sec > [ 3] 4.0- 5.0 sec 11.2 MBytes 94.4 Mbits/sec > [ 3] 5.0- 6.0 sec 10.8 MBytes 90.2 Mbits/sec > [ 3] 6.0- 7.0 sec 11.5 MBytes 96.5 Mbits/sec > [ 3] 7.0- 8.0 sec 11.2 MBytes 94.4 Mbits/sec > [ 3] 8.0- 9.0 sec 11.0 MBytes 92.3 Mbits/sec > [ 3] 9.0-10.0 sec 11.1 MBytes 93.3 Mbits/sec > [ 3] 0.0-10.0 sec 108 MBytes 90.7 Mbits/sec > > > So, it would seem that FreeBSD is very well tuned for Target -> Source > traffic (nearly 1Gbps), but the Source -> Target direction is terrible > (20Mbps). Since the RTT is the same and the # of hops is virtually > unchanged, we don't see any reason the speed should be different outbound, > and further we see that when we use Ubuntu -> Target, we get decent speed > (100Mbps). Thus we conclude the hardware and hops must not the issue. So, > what we're missing in our FreeBSD tuning that is making the outbound > transfers so slow (while the inbound transfers are great)? > > Thanks > > > > Source settings (FreeBSD): > > igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO> > ether xxxxxx > inet xxxxxx netmask 0xfffffff8 broadcast xxxxx > inet6 xxxxx%igb0 prefixlen 64 scopeid 0x1 > inet6 xxxxxxx prefixlen 64 > nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > media: Ethernet autoselect (1000baseT <full-duplex>) > status: active > > kern.ipc.maxsockbuf: 16777216 > kern.ipc.sockbuf_waste_factor: 8 > kern.ipc.somaxconn: 128 > kern.ipc.max_linkhdr: 16 > kern.ipc.max_protohdr: 60 > kern.ipc.max_hdr: 76 > kern.ipc.max_datalen: 92 > kern.ipc.nmbjumbo16: 3200 > kern.ipc.nmbjumbo9: 6400 > kern.ipc.nmbjumbop: 12800 > kern.ipc.nmbclusters: 25600 > kern.ipc.piperesizeallowed: 1 > kern.ipc.piperesizefail: 0 > kern.ipc.pipeallocfail: 0 > kern.ipc.pipefragretry: 0 > kern.ipc.pipekva: 21315584 > kern.ipc.maxpipekva: 4080218112 > kern.ipc.msgseg: 2048 > kern.ipc.msgssz: 8 > kern.ipc.msgtql: 40 > kern.ipc.msgmnb: 2048 > kern.ipc.msgmni: 40 > kern.ipc.msgmax: 16384 > kern.ipc.semaem: 16384 > kern.ipc.semvmx: 32767 > kern.ipc.semusz: 152 > kern.ipc.semume: 10 > kern.ipc.semopm: 100 > kern.ipc.semmsl: 60 > kern.ipc.semmnu: 30 > kern.ipc.semmns: 60 > kern.ipc.semmni: 10 > kern.ipc.semmap: 30 > kern.ipc.shm_allow_removed: 0 > kern.ipc.shm_use_phys: 0 > kern.ipc.shmall: 8192 > kern.ipc.shmseg: 128 > kern.ipc.shmmni: 192 > kern.ipc.shmmin: 1 > kern.ipc.shmmax: 33554432 > kern.ipc.maxsockets: 25600 > kern.ipc.numopensockets: 1281 > kern.ipc.nsfbufsused: 0 > kern.ipc.nsfbufspeak: 0 > kern.ipc.nsfbufs: 0 > net.inet.tcp.rfc1323: 1 > net.inet.tcp.mssdflt: 1460 > net.inet.tcp.keepidle: 7200000 > net.inet.tcp.keepintvl: 75000 > net.inet.tcp.sendspace: 2263000 > net.inet.tcp.recvspace: 2263000 > net.inet.tcp.keepinit: 75000 > net.inet.tcp.delacktime: 100 > net.inet.tcp.v6mssdflt: 1024 > net.inet.tcp.cc.available: newreno > net.inet.tcp.cc.algorithm: newreno > net.inet.tcp.hostcache.purge: 0 > net.inet.tcp.hostcache.prune: 300 > net.inet.tcp.hostcache.expire: 3600 > net.inet.tcp.hostcache.count: 193 > net.inet.tcp.hostcache.bucketlimit: 30 > net.inet.tcp.hostcache.hashsize: 512 > net.inet.tcp.hostcache.cachelimit: 15360 > net.inet.tcp.read_locking: 1 > net.inet.tcp.recvbuf_max: 16777216 > net.inet.tcp.recvbuf_inc: 524288 > net.inet.tcp.recvbuf_auto: 1 > net.inet.tcp.insecure_rst: 0 > net.inet.tcp.ecn.maxretries: 1 > net.inet.tcp.ecn.enable: 0 > net.inet.tcp.abc_l_var: 2 > net.inet.tcp.rfc3465: 1 > net.inet.tcp.rfc3390: 1 > net.inet.tcp.rfc3042: 1 > net.inet.tcp.drop_synfin: 1 > net.inet.tcp.delayed_ack: 1 > net.inet.tcp.blackhole: 0 > net.inet.tcp.log_in_vain: 0 > net.inet.tcp.sendbuf_max: 16777216 > net.inet.tcp.sendbuf_inc: 16384 > net.inet.tcp.sendbuf_auto: 1 > net.inet.tcp.tso: 1 > net.inet.tcp.local_slowstart_flightsize: 4 > net.inet.tcp.slowstart_flightsize: 1550 > net.inet.tcp.path_mtu_discovery: 1 > net.inet.tcp.reass.overflows: 481332 > net.inet.tcp.reass.cursegments: 4 > net.inet.tcp.reass.maxsegments: 1680 > net.inet.tcp.sack.globalholes: 0 > net.inet.tcp.sack.globalmaxholes: 65536 > net.inet.tcp.sack.maxholes: 128 > net.inet.tcp.sack.enable: 1 > net.inet.tcp.inflight.stab: 20 > net.inet.tcp.inflight.max: 1073725440 > net.inet.tcp.inflight.min: 6144 > net.inet.tcp.inflight.rttthresh: 10 > net.inet.tcp.inflight.debug: 0 > net.inet.tcp.inflight.enable: 0 > net.inet.tcp.isn_reseed_interval: 0 > net.inet.tcp.icmp_may_rst: 1 > net.inet.tcp.pcbcount: 461 > net.inet.tcp.do_tcpdrain: 1 > net.inet.tcp.tcbhashsize: 512 > net.inet.tcp.log_debug: 0 > net.inet.tcp.minmss: 216 > net.inet.tcp.syncache.rst_on_sock_fail: 1 > net.inet.tcp.syncache.rexmtlimit: 3 > net.inet.tcp.syncache.hashsize: 512 > net.inet.tcp.syncache.count: 4294967277 > net.inet.tcp.syncache.cachelimit: 15360 > net.inet.tcp.syncache.bucketlimit: 30 > net.inet.tcp.syncookies_only: 0 > net.inet.tcp.syncookies: 1 > net.inet.tcp.timer_race: 1 > net.inet.tcp.rexmit_drop_options: 1 > net.inet.tcp.finwait2_timeout: 60000 > net.inet.tcp.fast_finwait2_recycle: 0 > net.inet.tcp.always_keepalive: 1 > net.inet.tcp.rexmit_slop: 200 > net.inet.tcp.rexmit_min: 30 > net.inet.tcp.msl: 30000 > net.inet.tcp.nolocaltimewait: 0 > net.inet.tcp.maxtcptw: 5120 > > > Target settings (Ubuntu): > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:840363516 errors:0 dropped:0 overruns:0 frame:0 > TX packets:152325250 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:2086148084848 (2.0 TB) TX bytes:64233189137 (64.2 GB) > > net.ipv4.tcp_abort_on_overflow = 0 > net.ipv4.tcp_adv_win_scale = 1 > net.ipv4.tcp_allowed_congestion_control = cubic reno > net.ipv4.tcp_app_win = 31 > net.ipv4.tcp_autocorking = 1 > net.ipv4.tcp_available_congestion_control = cubic reno > net.ipv4.tcp_base_mss = 512 > net.ipv4.tcp_challenge_ack_limit = 100 > net.ipv4.tcp_congestion_control = cubic > net.ipv4.tcp_dsack = 1 > net.ipv4.tcp_early_retrans = 3 > net.ipv4.tcp_ecn = 2 > net.ipv4.tcp_fack = 1 > net.ipv4.tcp_fastopen = 1 > net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000 > net.ipv4.tcp_fin_timeout = 60 > net.ipv4.tcp_frto = 2 > net.ipv4.tcp_fwmark_accept = 0 > net.ipv4.tcp_invalid_ratelimit = 500 > net.ipv4.tcp_keepalive_intvl = 75 > net.ipv4.tcp_keepalive_probes = 9 > net.ipv4.tcp_keepalive_time = 7200 > net.ipv4.tcp_limit_output_bytes = 131072 > net.ipv4.tcp_low_latency = 0 > net.ipv4.tcp_max_orphans = 32768 > net.ipv4.tcp_max_reordering = 300 > net.ipv4.tcp_max_syn_backlog = 256 > net.ipv4.tcp_max_tw_buckets = 32768 > net.ipv4.tcp_mem = 524288 524288 524288 > net.ipv4.tcp_min_tso_segs = 2 > net.ipv4.tcp_moderate_rcvbuf = 1 > net.ipv4.tcp_mtu_probing = 0 > net.ipv4.tcp_no_metrics_save = 0 > net.ipv4.tcp_notsent_lowat = -1 > net.ipv4.tcp_orphan_retries = 0 > net.ipv4.tcp_reordering = 3 > net.ipv4.tcp_retrans_collapse = 1 > net.ipv4.tcp_retries1 = 3 > net.ipv4.tcp_retries2 = 15 > net.ipv4.tcp_rfc1337 = 0 > net.ipv4.tcp_rmem = 4096 87380 16777216 > net.ipv4.tcp_sack = 1 > net.ipv4.tcp_slow_start_after_idle = 1 > net.ipv4.tcp_stdurg = 0 > net.ipv4.tcp_syn_retries = 6 > net.ipv4.tcp_synack_retries = 5 > net.ipv4.tcp_syncookies = 0 > net.ipv4.tcp_thin_dupack = 0 > net.ipv4.tcp_thin_linear_timeouts = 0 > net.ipv4.tcp_timestamps = 1 > net.ipv4.tcp_tso_win_divisor = 3 > net.ipv4.tcp_tw_recycle = 0 > net.ipv4.tcp_tw_reuse = 0 > net.ipv4.tcp_window_scaling = 1 > net.ipv4.tcp_wmem = 4096 65536 16777216 > net.ipv4.tcp_workaround_signed_windows = 0 > net.core.busy_poll = 0 > net.core.busy_read = 0 > net.core.default_qdisc = pfifo_fast > net.core.dev_weight = 64 > net.core.flow_limit_cpu_bitmap = 00 > net.core.flow_limit_table_len = 4096 > net.core.message_burst = 10 > net.core.message_cost = 5 > net.core.netdev_budget = 300 > net.core.netdev_max_backlog = 1000 > net.core.netdev_tstamp_prequeue = 1 > net.core.optmem_max = 20480 > net.core.rmem_default = 524288 > net.core.rmem_max = 33554432 > net.core.rps_sock_flow_entries = 0 > net.core.somaxconn = 128 > net.core.tstamp_allow_data = 1 > net.core.warnings = 0 > net.core.wmem_default = 524288 > net.core.wmem_max = 33554432 > net.core.xfrm_acq_expires = 30 > net.core.xfrm_aevent_etime = 10 > net.core.xfrm_aevent_rseqth = 2 > net.core.xfrm_larval_drop = 1 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmom77oj=oZNukiXzjVQ1tHcfLcFFCo9hjkz73iaAu3BEoQ>