From owner-freebsd-net@FreeBSD.ORG Tue Apr 7 22:37:51 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E23CDCEA for ; Tue, 7 Apr 2015 22:37:51 +0000 (UTC) Received: from mx3.wp.pl (mx3.wp.pl [212.77.101.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.wp.pl", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 69E207BF for ; Tue, 7 Apr 2015 22:37:50 +0000 (UTC) Received: (wp-smtpd smtp.wp.pl 4343 invoked from network); 8 Apr 2015 00:11:07 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wp.pl; s=1024a; t=1428444667; bh=BDQbYLXIGybD2CnAmwySaiz3LDw64RjaIQLa/8mpJog=; h=From:To:Subject; b=fY97k//LW4o+RslSYf/x+liMWvAEqy+iEDgnXWJV+pXOJPWoBv5M+HWah904dRjjx sNyOjzea9l9JevFCL0Vn+QRfTGi8d8mWW1/qHK8KSNjhTyLLBnImwdCsnE/xYYF5GE /d4NFEtT5rR2n/fD8UgrMLV4PW8nZk6TM/ix5TTQ= Received: from 250-210-250-178.ftth.cust.kwaoo.net (HELO [10.0.5.10]) (marek_sal@[178.250.210.250]) (envelope-sender ) by smtp.wp.pl (WP-SMTPD) with ECDHE-RSA-AES256-SHA encrypted SMTP for ; 8 Apr 2015 00:11:07 +0200 Message-ID: <552455EB.7080609@wp.pl> Date: Wed, 08 Apr 2015 00:10:51 +0200 From: Marek Salwerowicz User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: "freebsd-net@freebsd.org" Subject: RTT and TCP Window size doubts, bandwidth issues Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-WP-AV: skaner antywirusowy poczty Wirtualnej Polski S. A. X-WP-SPAM: NO 0000000 [oVOB] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Apr 2015 22:37:52 -0000 Hi list, I am trying to find correct setup of sysctl's for following machines (VMs under Vmware Workstation 8) to test large TCP window size: There are 2 boxes, each of them has following setup: - % uname -a FreeBSD freeA 10.1-RELEASE-p6 FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 - 4GB of RAM - 1 NIC without any modifications, iperf reports bandwidth speed ~1Gbit/s between hosts: server-side: # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ client-side: # iperf -c 192.168.108.140 ------------------------------------------------------------ Client connecting to 192.168.108.140, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.108.141 port 35282 connected with 192.168.108.140 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.46 GBytes 1.25 Gbits/sec I want to simulate (using dummynet) link between hosts with bandwidth 400mbit/s and latency ~20ms. In order to do that, I create the ipfw pipe on one box: IPFW="ipfw -q" $IPFW pipe 1 config bw 400Mbit/s delay 10ms $IPFW add 1500 pipe 1 ip from any to any after running ipfw, bandwidth with default kernel sysctl becomes lower: client-side: ------------------------------------------------------------ Client connecting to 192.168.108.140, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.108.141 port 35340 connected with 192.168.108.140 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 12.5 MBytes 10.4 Mbits/sec I'd like to achieve bandwidth ~400Mbit/s. I've modified following sysctl's (both on client- and server-side): kern.ipc.maxsockbuf=33554432 # (default 2097152) net.inet.tcp.sendbuf_max=33554432 # (default 2097152) net.inet.tcp.recvbuf_max=33554432 # (default 2097152) net.inet.tcp.cc.algorithm=htcp # (default newreno) #enabled in /boot/loader.conf also net.inet.tcp.cc.htcp.adaptive_backoff=1 # (default 0 ; disabled) net.inet.tcp.cc.htcp.rtt_scaling=1 # (default 0 ; disabled) net.inet.tcp.mssdflt=1460 # (default 536) net.inet.tcp.minmss=1300 # (default 216) net.inet.tcp.rfc1323=1 # (default 1) net.inet.tcp.rfc3390=1 # (default 1) net.inet.tcp.sendspace=8388608 # (default 32768) net.inet.tcp.recvspace=8388608 # (default 65536) net.inet.tcp.sendbuf_inc=32768 # (default 8192 ) net.inet.tcp.recvbuf_inc=65536 # (default 16384) But the results are not really good as I expected: server-side: # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 8.00 MByte (default) ------------------------------------------------------------ client-side: # iperf -c 192.168.108.140 ------------------------------------------------------------ Client connecting to 192.168.108.140, TCP port 5001 TCP window size: 8.00 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.108.141 port 21894 connected with 192.168.108.140 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 24.2 MBytes 20.2 Mbits/sec I was trying to follow the articles: - http://www.psc.edu/index.php/networking/641-tcp-tune - https://fasterdata.es.net/host-tuning/freebsd/ But can't really figure out what / how should be tuned in order to achieve good results. If anyone could find some time and give me some hints, I'd be pleased! Cheers Marek