Date: Tue, 29 Sep 2015 16:22:48 -0700 From: javocado <javocado@gmail.com> To: freebsd-net@freebsd.org Subject: Network tuning help needed - asymmetric speed Message-ID: <CAP1HOmSMne6YyNGAXS-jDMGoYEsPUYnpRO3dHvrp2qP%2BZ3btmw@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
I am trying to figure out what set of tunables need to be tweaked in order
to get an Internet connection operating at decent speed in _both_
directions. Here are my particulars:
Source: FreeBSD 8.4 AMD
Target: Ubuntu 14.04 x64
Iperf tests:
Source -> Target:
# iperf -t10 -P1 -i1 -c xxxxxxxx
------------------------------------------------------------
Client connecting to xxxxxxxxx, TCP port 5001
TCP window size: 2.16 MByte (default)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 26.4 MBytes 221 Mbits/sec
[ 3] 1.0- 2.0 sec 9.12 MBytes 76.5 Mbits/sec
[ 3] 2.0- 3.0 sec 3.38 MBytes 28.3 Mbits/sec
[ 3] 3.0- 4.0 sec 3.88 MBytes 32.5 Mbits/sec
[ 3] 4.0- 5.0 sec 1.62 MBytes 13.6 Mbits/sec
[ 3] 5.0- 6.0 sec 2.38 MBytes 19.9 Mbits/sec
[ 3] 6.0- 7.0 sec 2.88 MBytes 24.1 Mbits/sec
[ 3] 7.0- 8.0 sec 1.00 MBytes 8.39 Mbits/sec
[ 3] 8.0- 9.0 sec 1.12 MBytes 9.44 Mbits/sec
[ 3] 9.0-10.0 sec 1.88 MBytes 15.7 Mbits/sec
[ 3] 0.0-10.1 sec 53.9 MBytes 44.9 Mbits/sec
subsequent identical iperf tests:
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 3.50 MBytes 29.4 Mbits/sec
[ 3] 1.0- 2.0 sec 3.75 MBytes 31.5 Mbits/sec
[ 3] 2.0- 3.0 sec 6.38 MBytes 53.5 Mbits/sec
[ 3] 3.0- 4.0 sec 2.12 MBytes 17.8 Mbits/sec
[ 3] 4.0- 5.0 sec 3.25 MBytes 27.3 Mbits/sec
[ 3] 5.0- 6.0 sec 4.25 MBytes 35.7 Mbits/sec
[ 3] 6.0- 7.0 sec 1.88 MBytes 15.7 Mbits/sec
[ 3] 7.0- 8.0 sec 4.12 MBytes 34.6 Mbits/sec
[ 3] 8.0- 9.0 sec 1.25 MBytes 10.5 Mbits/sec
[ 3] 9.0-10.0 sec 1.00 MBytes 8.39 Mbits/sec
[ 3] 0.0-10.1 sec 31.8 MBytes 26.4 Mbits/sec
Target -> Source:
# iperf -t10 -P1 -i1 -c xxxxxx
------------------------------------------------------------
Client connecting to xxxxxxxx, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 90.8 MBytes 761 Mbits/sec
[ 3] 1.0- 2.0 sec 104 MBytes 871 Mbits/sec
[ 3] 2.0- 3.0 sec 107 MBytes 900 Mbits/sec
[ 3] 3.0- 4.0 sec 96.0 MBytes 805 Mbits/sec
[ 3] 4.0- 5.0 sec 97.8 MBytes 820 Mbits/sec
[ 3] 5.0- 6.0 sec 102 MBytes 857 Mbits/sec
[ 3] 6.0- 7.0 sec 104 MBytes 873 Mbits/sec
[ 3] 7.0- 8.0 sec 104 MBytes 868 Mbits/sec
[ 3] 8.0- 9.0 sec 104 MBytes 873 Mbits/sec
[ 3] 9.0-10.0 sec 104 MBytes 871 Mbits/sec
[ 3] 0.0-10.0 sec 1014 MBytes 850 Mbits/sec
Traceroutes:
Source -> Target:
traceroute to xxxxxxx 64 hops max, 52 byte packets
1 6.978 ms 1.989 ms 2.002 ms
2 10.954 ms 0.983 ms
1.957 ms
3 52.189 ms 5.998 ms 22.044 ms
4 26.091 ms 22.056 ms 24.017 ms
5 21.492 ms 21.029 ms
6 21.047 ms 21.093 ms 21.998 ms
7 20.897 ms 20.744 ms 23.042 ms
8 20.699 ms 20.655 ms 20.526 ms
Target -> Source:
traceroute to xxxxxx, 30 hops max, 60 byte packets
1 0.782 ms 0.761 ms 0.784 ms
2 1.072 ms 1.028 ms 1.002 ms
3 0.689 ms 0.665 ms 0.796 ms
4 42.513 ms 42.596 ms 42.568 ms
5 0.917 ms 0.895 ms 0.866 ms
6 20.209 ms 28.508 ms 28.507 ms
7 20.346 ms 20.352 ms 20.392 ms
8 30.392 ms 30.404 ms 30.387 ms
9 20.542 ms 20.720 ms 20.899 ms
We thought there could just be a bad hop or perhaps it was hardware at
fault, but when we run our iperf from a different source (a stock Ubuntu
12.04 box) connected to the same switch as the FreeBSD source, we get
great/better/acceptable results:
(Ubuntu 12.04) Source -> Target:
$ iperf -t10 -P1 -i1 -w1M -c xxxxxxx
------------------------------------------------------------
Client connecting to xxxxxxxx, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 9.25 MBytes 77.6 Mbits/sec
[ 3] 1.0- 2.0 sec 10.2 MBytes 86.0 Mbits/sec
[ 3] 2.0- 3.0 sec 10.6 MBytes 89.1 Mbits/sec
[ 3] 3.0- 4.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 3] 4.0- 5.0 sec 11.2 MBytes 94.4 Mbits/sec
[ 3] 5.0- 6.0 sec 10.8 MBytes 90.2 Mbits/sec
[ 3] 6.0- 7.0 sec 11.5 MBytes 96.5 Mbits/sec
[ 3] 7.0- 8.0 sec 11.2 MBytes 94.4 Mbits/sec
[ 3] 8.0- 9.0 sec 11.0 MBytes 92.3 Mbits/sec
[ 3] 9.0-10.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 3] 0.0-10.0 sec 108 MBytes 90.7 Mbits/sec
So, it would seem that FreeBSD is very well tuned for Target -> Source
traffic (nearly 1Gbps), but the Source -> Target direction is terrible
(20Mbps). Since the RTT is the same and the # of hops is virtually
unchanged, we don't see any reason the speed should be different outbound,
and further we see that when we use Ubuntu -> Target, we get decent speed
(100Mbps). Thus we conclude the hardware and hops must not the issue. So,
what we're missing in our FreeBSD tuning that is making the outbound
transfers so slow (while the inbound transfers are great)?
Thanks
Source settings (FreeBSD):
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether xxxxxx
inet xxxxxx netmask 0xfffffff8 broadcast xxxxx
inet6 xxxxx%igb0 prefixlen 64 scopeid 0x1
inet6 xxxxxxx prefixlen 64
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
kern.ipc.maxsockbuf: 16777216
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.somaxconn: 128
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 60
kern.ipc.max_hdr: 76
kern.ipc.max_datalen: 92
kern.ipc.nmbjumbo16: 3200
kern.ipc.nmbjumbo9: 6400
kern.ipc.nmbjumbop: 12800
kern.ipc.nmbclusters: 25600
kern.ipc.piperesizeallowed: 1
kern.ipc.piperesizefail: 0
kern.ipc.pipeallocfail: 0
kern.ipc.pipefragretry: 0
kern.ipc.pipekva: 21315584
kern.ipc.maxpipekva: 4080218112
kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.semaem: 16384
kern.ipc.semvmx: 32767
kern.ipc.semusz: 152
kern.ipc.semume: 10
kern.ipc.semopm: 100
kern.ipc.semmsl: 60
kern.ipc.semmnu: 30
kern.ipc.semmns: 60
kern.ipc.semmni: 10
kern.ipc.semmap: 30
kern.ipc.shm_allow_removed: 0
kern.ipc.shm_use_phys: 0
kern.ipc.shmall: 8192
kern.ipc.shmseg: 128
kern.ipc.shmmni: 192
kern.ipc.shmmin: 1
kern.ipc.shmmax: 33554432
kern.ipc.maxsockets: 25600
kern.ipc.numopensockets: 1281
kern.ipc.nsfbufsused: 0
kern.ipc.nsfbufspeak: 0
kern.ipc.nsfbufs: 0
net.inet.tcp.rfc1323: 1
net.inet.tcp.mssdflt: 1460
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 2263000
net.inet.tcp.recvspace: 2263000
net.inet.tcp.keepinit: 75000
net.inet.tcp.delacktime: 100
net.inet.tcp.v6mssdflt: 1024
net.inet.tcp.cc.available: newreno
net.inet.tcp.cc.algorithm: newreno
net.inet.tcp.hostcache.purge: 0
net.inet.tcp.hostcache.prune: 300
net.inet.tcp.hostcache.expire: 3600
net.inet.tcp.hostcache.count: 193
net.inet.tcp.hostcache.bucketlimit: 30
net.inet.tcp.hostcache.hashsize: 512
net.inet.tcp.hostcache.cachelimit: 15360
net.inet.tcp.read_locking: 1
net.inet.tcp.recvbuf_max: 16777216
net.inet.tcp.recvbuf_inc: 524288
net.inet.tcp.recvbuf_auto: 1
net.inet.tcp.insecure_rst: 0
net.inet.tcp.ecn.maxretries: 1
net.inet.tcp.ecn.enable: 0
net.inet.tcp.abc_l_var: 2
net.inet.tcp.rfc3465: 1
net.inet.tcp.rfc3390: 1
net.inet.tcp.rfc3042: 1
net.inet.tcp.drop_synfin: 1
net.inet.tcp.delayed_ack: 1
net.inet.tcp.blackhole: 0
net.inet.tcp.log_in_vain: 0
net.inet.tcp.sendbuf_max: 16777216
net.inet.tcp.sendbuf_inc: 16384
net.inet.tcp.sendbuf_auto: 1
net.inet.tcp.tso: 1
net.inet.tcp.local_slowstart_flightsize: 4
net.inet.tcp.slowstart_flightsize: 1550
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.reass.overflows: 481332
net.inet.tcp.reass.cursegments: 4
net.inet.tcp.reass.maxsegments: 1680
net.inet.tcp.sack.globalholes: 0
net.inet.tcp.sack.globalmaxholes: 65536
net.inet.tcp.sack.maxholes: 128
net.inet.tcp.sack.enable: 1
net.inet.tcp.inflight.stab: 20
net.inet.tcp.inflight.max: 1073725440
net.inet.tcp.inflight.min: 6144
net.inet.tcp.inflight.rttthresh: 10
net.inet.tcp.inflight.debug: 0
net.inet.tcp.inflight.enable: 0
net.inet.tcp.isn_reseed_interval: 0
net.inet.tcp.icmp_may_rst: 1
net.inet.tcp.pcbcount: 461
net.inet.tcp.do_tcpdrain: 1
net.inet.tcp.tcbhashsize: 512
net.inet.tcp.log_debug: 0
net.inet.tcp.minmss: 216
net.inet.tcp.syncache.rst_on_sock_fail: 1
net.inet.tcp.syncache.rexmtlimit: 3
net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.count: 4294967277
net.inet.tcp.syncache.cachelimit: 15360
net.inet.tcp.syncache.bucketlimit: 30
net.inet.tcp.syncookies_only: 0
net.inet.tcp.syncookies: 1
net.inet.tcp.timer_race: 1
net.inet.tcp.rexmit_drop_options: 1
net.inet.tcp.finwait2_timeout: 60000
net.inet.tcp.fast_finwait2_recycle: 0
net.inet.tcp.always_keepalive: 1
net.inet.tcp.rexmit_slop: 200
net.inet.tcp.rexmit_min: 30
net.inet.tcp.msl: 30000
net.inet.tcp.nolocaltimewait: 0
net.inet.tcp.maxtcptw: 5120
Target settings (Ubuntu):
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:840363516 errors:0 dropped:0 overruns:0 frame:0
TX packets:152325250 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2086148084848 (2.0 TB) TX bytes:64233189137 (64.2 GB)
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_challenge_ack_limit = 100
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 32768
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 256
net.ipv4.tcp_max_tw_buckets = 32768
net.ipv4.tcp_mem = 524288 524288 524288
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_notsent_lowat = -1
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_workaround_signed_windows = 0
net.core.busy_poll = 0
net.core.busy_read = 0
net.core.default_qdisc = pfifo_fast
net.core.dev_weight = 64
net.core.flow_limit_cpu_bitmap = 00
net.core.flow_limit_table_len = 4096
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_max_backlog = 1000
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 20480
net.core.rmem_default = 524288
net.core.rmem_max = 33554432
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 128
net.core.tstamp_allow_data = 1
net.core.warnings = 0
net.core.wmem_default = 524288
net.core.wmem_max = 33554432
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAP1HOmSMne6YyNGAXS-jDMGoYEsPUYnpRO3dHvrp2qP%2BZ3btmw>