From owner-freebsd-net@freebsd.org Tue Sep 29 23:22:51 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E489A0C190 for ; Tue, 29 Sep 2015 23:22:51 +0000 (UTC) (envelope-from javocado@gmail.com) Received: from mail-la0-x22b.google.com (mail-la0-x22b.google.com [IPv6:2a00:1450:4010:c03::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A35481C3B for ; Tue, 29 Sep 2015 23:22:50 +0000 (UTC) (envelope-from javocado@gmail.com) Received: by laclj5 with SMTP id lj5so26600599lac.3 for ; Tue, 29 Sep 2015 16:22:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=q/6OrebTQAv41wcJn/U3q9KQvN5miLw9j5rN9szpf6k=; b=xrgyGqRTXlzAcEHzmzAuLjj4PpmZw4bMm0xPdCq7f4Bhrjis+pwbw2jYKS+ryTG+iD F0wcJcU79rD+9HwreZ8ZEh+i0P40NFA8BUavpbvrEvvNEHtSi32jt9ATYr4/QeZmDVkD 8IyIVbmSZD05SrAgw5ly6m+gmXfE00sqsjfh2MHmvaSv0B3UOzf1Ied1YAgDAe4IZehp 0VrsxELqLUL/Ss8Tou+/GnpoyQXIq+sDk8yfNuuzi9+L9mpldFk9uERmW/BfoK37FNxI ZToW1c9JtsAwUBFvVnKWsCIw6sK7ALn3PnZya1UODZrcXL+OS6+ftqm4d/16qluHCQS5 winA== MIME-Version: 1.0 X-Received: by 10.152.6.70 with SMTP id y6mr147772lay.101.1443568968058; Tue, 29 Sep 2015 16:22:48 -0700 (PDT) Received: by 10.114.77.134 with HTTP; Tue, 29 Sep 2015 16:22:48 -0700 (PDT) Date: Tue, 29 Sep 2015 16:22:48 -0700 Message-ID: Subject: Network tuning help needed - asymmetric speed From: javocado To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 23:22:51 -0000 I am trying to figure out what set of tunables need to be tweaked in order to get an Internet connection operating at decent speed in _both_ directions. Here are my particulars: Source: FreeBSD 8.4 AMD Target: Ubuntu 14.04 x64 Iperf tests: Source -> Target: # iperf -t10 -P1 -i1 -c xxxxxxxx ------------------------------------------------------------ Client connecting to xxxxxxxxx, TCP port 5001 TCP window size: 2.16 MByte (default) ------------------------------------------------------------ [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 26.4 MBytes 221 Mbits/sec [ 3] 1.0- 2.0 sec 9.12 MBytes 76.5 Mbits/sec [ 3] 2.0- 3.0 sec 3.38 MBytes 28.3 Mbits/sec [ 3] 3.0- 4.0 sec 3.88 MBytes 32.5 Mbits/sec [ 3] 4.0- 5.0 sec 1.62 MBytes 13.6 Mbits/sec [ 3] 5.0- 6.0 sec 2.38 MBytes 19.9 Mbits/sec [ 3] 6.0- 7.0 sec 2.88 MBytes 24.1 Mbits/sec [ 3] 7.0- 8.0 sec 1.00 MBytes 8.39 Mbits/sec [ 3] 8.0- 9.0 sec 1.12 MBytes 9.44 Mbits/sec [ 3] 9.0-10.0 sec 1.88 MBytes 15.7 Mbits/sec [ 3] 0.0-10.1 sec 53.9 MBytes 44.9 Mbits/sec subsequent identical iperf tests: [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 3.50 MBytes 29.4 Mbits/sec [ 3] 1.0- 2.0 sec 3.75 MBytes 31.5 Mbits/sec [ 3] 2.0- 3.0 sec 6.38 MBytes 53.5 Mbits/sec [ 3] 3.0- 4.0 sec 2.12 MBytes 17.8 Mbits/sec [ 3] 4.0- 5.0 sec 3.25 MBytes 27.3 Mbits/sec [ 3] 5.0- 6.0 sec 4.25 MBytes 35.7 Mbits/sec [ 3] 6.0- 7.0 sec 1.88 MBytes 15.7 Mbits/sec [ 3] 7.0- 8.0 sec 4.12 MBytes 34.6 Mbits/sec [ 3] 8.0- 9.0 sec 1.25 MBytes 10.5 Mbits/sec [ 3] 9.0-10.0 sec 1.00 MBytes 8.39 Mbits/sec [ 3] 0.0-10.1 sec 31.8 MBytes 26.4 Mbits/sec Target -> Source: # iperf -t10 -P1 -i1 -c xxxxxx ------------------------------------------------------------ Client connecting to xxxxxxxx, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 90.8 MBytes 761 Mbits/sec [ 3] 1.0- 2.0 sec 104 MBytes 871 Mbits/sec [ 3] 2.0- 3.0 sec 107 MBytes 900 Mbits/sec [ 3] 3.0- 4.0 sec 96.0 MBytes 805 Mbits/sec [ 3] 4.0- 5.0 sec 97.8 MBytes 820 Mbits/sec [ 3] 5.0- 6.0 sec 102 MBytes 857 Mbits/sec [ 3] 6.0- 7.0 sec 104 MBytes 873 Mbits/sec [ 3] 7.0- 8.0 sec 104 MBytes 868 Mbits/sec [ 3] 8.0- 9.0 sec 104 MBytes 873 Mbits/sec [ 3] 9.0-10.0 sec 104 MBytes 871 Mbits/sec [ 3] 0.0-10.0 sec 1014 MBytes 850 Mbits/sec Traceroutes: Source -> Target: traceroute to xxxxxxx 64 hops max, 52 byte packets 1 6.978 ms 1.989 ms 2.002 ms 2 10.954 ms 0.983 ms 1.957 ms 3 52.189 ms 5.998 ms 22.044 ms 4 26.091 ms 22.056 ms 24.017 ms 5 21.492 ms 21.029 ms 6 21.047 ms 21.093 ms 21.998 ms 7 20.897 ms 20.744 ms 23.042 ms 8 20.699 ms 20.655 ms 20.526 ms Target -> Source: traceroute to xxxxxx, 30 hops max, 60 byte packets 1 0.782 ms 0.761 ms 0.784 ms 2 1.072 ms 1.028 ms 1.002 ms 3 0.689 ms 0.665 ms 0.796 ms 4 42.513 ms 42.596 ms 42.568 ms 5 0.917 ms 0.895 ms 0.866 ms 6 20.209 ms 28.508 ms 28.507 ms 7 20.346 ms 20.352 ms 20.392 ms 8 30.392 ms 30.404 ms 30.387 ms 9 20.542 ms 20.720 ms 20.899 ms We thought there could just be a bad hop or perhaps it was hardware at fault, but when we run our iperf from a different source (a stock Ubuntu 12.04 box) connected to the same switch as the FreeBSD source, we get great/better/acceptable results: (Ubuntu 12.04) Source -> Target: $ iperf -t10 -P1 -i1 -w1M -c xxxxxxx ------------------------------------------------------------ Client connecting to xxxxxxxx, TCP port 5001 TCP window size: 256 KByte (WARNING: requested 1.00 MByte) ------------------------------------------------------------ [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 9.25 MBytes 77.6 Mbits/sec [ 3] 1.0- 2.0 sec 10.2 MBytes 86.0 Mbits/sec [ 3] 2.0- 3.0 sec 10.6 MBytes 89.1 Mbits/sec [ 3] 3.0- 4.0 sec 11.1 MBytes 93.3 Mbits/sec [ 3] 4.0- 5.0 sec 11.2 MBytes 94.4 Mbits/sec [ 3] 5.0- 6.0 sec 10.8 MBytes 90.2 Mbits/sec [ 3] 6.0- 7.0 sec 11.5 MBytes 96.5 Mbits/sec [ 3] 7.0- 8.0 sec 11.2 MBytes 94.4 Mbits/sec [ 3] 8.0- 9.0 sec 11.0 MBytes 92.3 Mbits/sec [ 3] 9.0-10.0 sec 11.1 MBytes 93.3 Mbits/sec [ 3] 0.0-10.0 sec 108 MBytes 90.7 Mbits/sec So, it would seem that FreeBSD is very well tuned for Target -> Source traffic (nearly 1Gbps), but the Source -> Target direction is terrible (20Mbps). Since the RTT is the same and the # of hops is virtually unchanged, we don't see any reason the speed should be different outbound, and further we see that when we use Ubuntu -> Target, we get decent speed (100Mbps). Thus we conclude the hardware and hops must not the issue. So, what we're missing in our FreeBSD tuning that is making the outbound transfers so slow (while the inbound transfers are great)? Thanks Source settings (FreeBSD): igb0: flags=8843 metric 0 mtu 1500 options=401bb ether xxxxxx inet xxxxxx netmask 0xfffffff8 broadcast xxxxx inet6 xxxxx%igb0 prefixlen 64 scopeid 0x1 inet6 xxxxxxx prefixlen 64 nd6 options=3 media: Ethernet autoselect (1000baseT ) status: active kern.ipc.maxsockbuf: 16777216 kern.ipc.sockbuf_waste_factor: 8 kern.ipc.somaxconn: 128 kern.ipc.max_linkhdr: 16 kern.ipc.max_protohdr: 60 kern.ipc.max_hdr: 76 kern.ipc.max_datalen: 92 kern.ipc.nmbjumbo16: 3200 kern.ipc.nmbjumbo9: 6400 kern.ipc.nmbjumbop: 12800 kern.ipc.nmbclusters: 25600 kern.ipc.piperesizeallowed: 1 kern.ipc.piperesizefail: 0 kern.ipc.pipeallocfail: 0 kern.ipc.pipefragretry: 0 kern.ipc.pipekva: 21315584 kern.ipc.maxpipekva: 4080218112 kern.ipc.msgseg: 2048 kern.ipc.msgssz: 8 kern.ipc.msgtql: 40 kern.ipc.msgmnb: 2048 kern.ipc.msgmni: 40 kern.ipc.msgmax: 16384 kern.ipc.semaem: 16384 kern.ipc.semvmx: 32767 kern.ipc.semusz: 152 kern.ipc.semume: 10 kern.ipc.semopm: 100 kern.ipc.semmsl: 60 kern.ipc.semmnu: 30 kern.ipc.semmns: 60 kern.ipc.semmni: 10 kern.ipc.semmap: 30 kern.ipc.shm_allow_removed: 0 kern.ipc.shm_use_phys: 0 kern.ipc.shmall: 8192 kern.ipc.shmseg: 128 kern.ipc.shmmni: 192 kern.ipc.shmmin: 1 kern.ipc.shmmax: 33554432 kern.ipc.maxsockets: 25600 kern.ipc.numopensockets: 1281 kern.ipc.nsfbufsused: 0 kern.ipc.nsfbufspeak: 0 kern.ipc.nsfbufs: 0 net.inet.tcp.rfc1323: 1 net.inet.tcp.mssdflt: 1460 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 2263000 net.inet.tcp.recvspace: 2263000 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.v6mssdflt: 1024 net.inet.tcp.cc.available: newreno net.inet.tcp.cc.algorithm: newreno net.inet.tcp.hostcache.purge: 0 net.inet.tcp.hostcache.prune: 300 net.inet.tcp.hostcache.expire: 3600 net.inet.tcp.hostcache.count: 193 net.inet.tcp.hostcache.bucketlimit: 30 net.inet.tcp.hostcache.hashsize: 512 net.inet.tcp.hostcache.cachelimit: 15360 net.inet.tcp.read_locking: 1 net.inet.tcp.recvbuf_max: 16777216 net.inet.tcp.recvbuf_inc: 524288 net.inet.tcp.recvbuf_auto: 1 net.inet.tcp.insecure_rst: 0 net.inet.tcp.ecn.maxretries: 1 net.inet.tcp.ecn.enable: 0 net.inet.tcp.abc_l_var: 2 net.inet.tcp.rfc3465: 1 net.inet.tcp.rfc3390: 1 net.inet.tcp.rfc3042: 1 net.inet.tcp.drop_synfin: 1 net.inet.tcp.delayed_ack: 1 net.inet.tcp.blackhole: 0 net.inet.tcp.log_in_vain: 0 net.inet.tcp.sendbuf_max: 16777216 net.inet.tcp.sendbuf_inc: 16384 net.inet.tcp.sendbuf_auto: 1 net.inet.tcp.tso: 1 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.slowstart_flightsize: 1550 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.reass.overflows: 481332 net.inet.tcp.reass.cursegments: 4 net.inet.tcp.reass.maxsegments: 1680 net.inet.tcp.sack.globalholes: 0 net.inet.tcp.sack.globalmaxholes: 65536 net.inet.tcp.sack.maxholes: 128 net.inet.tcp.sack.enable: 1 net.inet.tcp.inflight.stab: 20 net.inet.tcp.inflight.max: 1073725440 net.inet.tcp.inflight.min: 6144 net.inet.tcp.inflight.rttthresh: 10 net.inet.tcp.inflight.debug: 0 net.inet.tcp.inflight.enable: 0 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.pcbcount: 461 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.log_debug: 0 net.inet.tcp.minmss: 216 net.inet.tcp.syncache.rst_on_sock_fail: 1 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.count: 4294967277 net.inet.tcp.syncache.cachelimit: 15360 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncookies_only: 0 net.inet.tcp.syncookies: 1 net.inet.tcp.timer_race: 1 net.inet.tcp.rexmit_drop_options: 1 net.inet.tcp.finwait2_timeout: 60000 net.inet.tcp.fast_finwait2_recycle: 0 net.inet.tcp.always_keepalive: 1 net.inet.tcp.rexmit_slop: 200 net.inet.tcp.rexmit_min: 30 net.inet.tcp.msl: 30000 net.inet.tcp.nolocaltimewait: 0 net.inet.tcp.maxtcptw: 5120 Target settings (Ubuntu): UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:840363516 errors:0 dropped:0 overruns:0 frame:0 TX packets:152325250 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2086148084848 (2.0 TB) TX bytes:64233189137 (64.2 GB) net.ipv4.tcp_abort_on_overflow = 0 net.ipv4.tcp_adv_win_scale = 1 net.ipv4.tcp_allowed_congestion_control = cubic reno net.ipv4.tcp_app_win = 31 net.ipv4.tcp_autocorking = 1 net.ipv4.tcp_available_congestion_control = cubic reno net.ipv4.tcp_base_mss = 512 net.ipv4.tcp_challenge_ack_limit = 100 net.ipv4.tcp_congestion_control = cubic net.ipv4.tcp_dsack = 1 net.ipv4.tcp_early_retrans = 3 net.ipv4.tcp_ecn = 2 net.ipv4.tcp_fack = 1 net.ipv4.tcp_fastopen = 1 net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000 net.ipv4.tcp_fin_timeout = 60 net.ipv4.tcp_frto = 2 net.ipv4.tcp_fwmark_accept = 0 net.ipv4.tcp_invalid_ratelimit = 500 net.ipv4.tcp_keepalive_intvl = 75 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_limit_output_bytes = 131072 net.ipv4.tcp_low_latency = 0 net.ipv4.tcp_max_orphans = 32768 net.ipv4.tcp_max_reordering = 300 net.ipv4.tcp_max_syn_backlog = 256 net.ipv4.tcp_max_tw_buckets = 32768 net.ipv4.tcp_mem = 524288 524288 524288 net.ipv4.tcp_min_tso_segs = 2 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_mtu_probing = 0 net.ipv4.tcp_no_metrics_save = 0 net.ipv4.tcp_notsent_lowat = -1 net.ipv4.tcp_orphan_retries = 0 net.ipv4.tcp_reordering = 3 net.ipv4.tcp_retrans_collapse = 1 net.ipv4.tcp_retries1 = 3 net.ipv4.tcp_retries2 = 15 net.ipv4.tcp_rfc1337 = 0 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_sack = 1 net.ipv4.tcp_slow_start_after_idle = 1 net.ipv4.tcp_stdurg = 0 net.ipv4.tcp_syn_retries = 6 net.ipv4.tcp_synack_retries = 5 net.ipv4.tcp_syncookies = 0 net.ipv4.tcp_thin_dupack = 0 net.ipv4.tcp_thin_linear_timeouts = 0 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_tso_win_divisor = 3 net.ipv4.tcp_tw_recycle = 0 net.ipv4.tcp_tw_reuse = 0 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_workaround_signed_windows = 0 net.core.busy_poll = 0 net.core.busy_read = 0 net.core.default_qdisc = pfifo_fast net.core.dev_weight = 64 net.core.flow_limit_cpu_bitmap = 00 net.core.flow_limit_table_len = 4096 net.core.message_burst = 10 net.core.message_cost = 5 net.core.netdev_budget = 300 net.core.netdev_max_backlog = 1000 net.core.netdev_tstamp_prequeue = 1 net.core.optmem_max = 20480 net.core.rmem_default = 524288 net.core.rmem_max = 33554432 net.core.rps_sock_flow_entries = 0 net.core.somaxconn = 128 net.core.tstamp_allow_data = 1 net.core.warnings = 0 net.core.wmem_default = 524288 net.core.wmem_max = 33554432 net.core.xfrm_acq_expires = 30 net.core.xfrm_aevent_etime = 10 net.core.xfrm_aevent_rseqth = 2 net.core.xfrm_larval_drop = 1