From owner-freebsd-net@FreeBSD.ORG Wed Sep 4 15:29:07 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D9BCC2B1 for ; Wed, 4 Sep 2013 15:29:07 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6E3112B9C for ; Wed, 4 Sep 2013 15:29:07 +0000 (UTC) Received: from [192.168.43.26] (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.7/8.14.5) with ESMTP id r84FQvNw049424; Wed, 4 Sep 2013 11:26:57 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <52275136.1010105@sentex.net> Date: Wed, 04 Sep 2013 11:26:46 -0400 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Rick Macklem Subject: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) References: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca> In-Reply-To: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca> X-Enigmail-Version: 1.4.2 Content-Type: multipart/mixed; boundary="------------090102030905000709060202" X-Scanned-By: MIMEDefang 2.74 on 64.7.153.18 Cc: FreeBSD Net , David Wolfskill X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Sep 2013 15:29:08 -0000 This is a multi-part message in MIME format. --------------090102030905000709060202 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On 9/4/2013 8:50 AM, Rick Macklem wrote: > David Wolfskill wrote: >> >> >> I noticed that when I tried to write files to NFS, I could write >> small >> files OK, but larger ones seemed to ... hang. >> * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on. >> * "sysctl net.inet.tcp.tso" showed "1" -- enabled. >> >> As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked >> without >> a hitch or a whine. And I was able to copy all 117709618 bytes, not >> just >> 2097152 (2^21). >> >> Is the above expected? It came rather as a surprise to me. >> > Not surprising to me, I'm afraid. When there are serious NFS problems > like this, it is often caused by a network fabric issue and broken > TSO is at the top of the list w.r.t. cause. I was just experimenting a bit with iSCSI via FreeNAS and was a little disappointed at the speeds I was getting. So, I tried disabling tso on both boxes and it did seem to speed things up a bit. Data and testing methods attached in a txt file. I did 3 cases. Just boot up FreeNAS and the initiator without tweaks. That had the worst performance. disable tso on the nic as well as via sysctl on both boxes. That had the best performance. re-enable tso on both boxes. That had better performance than the first case, but still not as good as totally disabling it. I am guessing something is not quite being re-enabled properly ? But its different than the other two cases ?!? tgt is FreeNAS-9.1.1-RELEASE-x64 (a752d35) and initiator is r254328 9.2 AMD64 The FreeNAS box has 16G of RAM, so the file is being served out of cache as gstat shows no activity when sending out the file ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ --------------090102030905000709060202 Content-Type: text/plain; charset=windows-1252; name="stat.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="stat.txt" 3 data files. notso = tso disabled on tgt and initiator tso-boot = initiator is rebooted and tests are run tso = tso disabled and then re-enabled. For whatever reason its not as bad as post boot, but still worse than no tso 0{mdttestbox}# ministat notso tso-boot x notso + tso-boot +----------------------------------------------------------------------------------------------------------------------+ | + | | +++ x | |+ ++++ xxxx x x x x| | |AM| |___M__A_____| | +----------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 9 69086843 71749085 69475320 69810666 887628.9 + 9 55846191 56473897 56357956 56302165 208843.3 Difference at 95.0% confidence -1.35085e+07 +/- 644386 -19.3502% +/- 0.923048% (Student's t, pooled s = 644787) 0{mdttestbox}# # ministat tso notso x tso + notso +----------------------------------------------------------------------------------------------------------------------+ | x + | |x xxxxx x + x + + + + + + +| | |____MA___||___________________M___________A_________________________________| | +----------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 9 68632067 69126677 68779201 68808473 142050.01 + 9 69086843 71749085 69475320 69810666 887628.9 Difference at 95.0% confidence 1.00219e+06 +/- 635239 1.4565% +/- 0.923199% (Student's t, pooled s = 635635) 0{mdttestbox}# cat tso 68779201 68734143 68915705 68752827 68782212 69126677 68828520 68724901 68632067 0{mdttestbox}# cat notso 71749085 69256608 69097532 69086843 70587459 69179672 69475320 69754511 70108963 0{mdttestbox}# cat tso-boot 55846191 56314173 56284204 56095729 56466769 56446535 56357956 56434027 56473897 0{mdttestbox}# 0{mdttestbox}# cat t.sh #!/bin/sh command="dd if=/mnt/test of=/dev/null bs=4096k" for i in `jot 9 1`;do mount /dev/da0a /mnt eval $command 2>&1 | grep bytes | awk -F"[\( ]" '{print $8}' umount /mnt done 0{mdttestbox}# initiator is r254328 9.2 AMD64 The two boxes are linked via private gigE switch em1: flags=8843 metric 0 mtu 9000 options=4209b ether 00:1e:67:49:31:b8 inet 10.254.1.2 netmask 0xffffff00 broadcast 10.254.1.255 inet6 fe80::21e:67ff:fe49:31b8%em1 prefixlen 64 scopeid 0x3 nd6 options=23 media: Ethernet autoselect (1000baseT ) status: active ifconfig igb0 igb0: flags=8843 metric 0 mtu 9000 options=400bb ether 00:15:17:fe:a5:bc inet 10.254.1.1 netmask 0xffffff00 broadcast 10.254.1.255 inet6 fe80::215:17ff:fefe:a5bc%igb0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active Initiator just does iscontrol -c /etc/iscsi.conf -n freebsd freebsd { authmethod = none initiatorname = mdttestbox.sentex.ca TargetName = iqn.2011-03.org.example.istgt:freebsd TargetAddress = 10.254.1.1:3260,2 } ifconfig igb0 -tso ifconfig em0 -tso and sysctl -w net.inet.tcp.tso=0 --------------090102030905000709060202--