From owner-freebsd-stable@FreeBSD.ORG Sun Jan 19 14:30:50 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A843861B for ; Sun, 19 Jan 2014 14:30:50 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9177E1137 for ; Sun, 19 Jan 2014 14:30:50 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-76-21-10-192.hsd1.ca.comcast.net [76.21.10.192]) by elvis.mu.org (Postfix) with ESMTPSA id 3CABC1A3C1C for ; Sun, 19 Jan 2014 06:30:50 -0800 (PST) Message-ID: <52DBE19C.4040101@freebsd.org> Date: Sun, 19 Jan 2014 06:30:52 -0800 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jan 2014 14:30:50 -0000 What does vmstat say about memory zones? I think vmstat -m/vmstat -z and also netstat -m? Are you set with the same mbufs on both 9 and 10? -Alfred On 1/18/14 1:32 PM, Eric Dombroski wrote: > Adrian: > > Yes, no change. > > -Eric > > > > > > On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd wrote: > >> Hi, >> >> Have you tried disabling tso? >> >> Adrian >> On Jan 18, 2014 1:52 PM, "Eric Dombroski" wrote: >> >>> Hello: >>> >>> I believe there is a major performance regression between FreeBSD >>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and >>> handling incoming traffic. Below are the results of some iperf tests and >>> large dd operations over NFS. Write throughput goes from ~40Gbps to >>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes >>> unstable >>> ("no buffer space available"), requiring the interface to be taken >>> down/up. >>> >>> >>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks >>> on >>> either system. >>> >>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe >>> passthrough, although I suspect the problem manifests itself over 1Gbps >>> speeds anyway. >>> >>> Tests: >>> >>> Client (host): >>> root@gogo:~# uname -a >>> Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux >>> root@gogo:~# kvm -version >>> QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright >>> (c) 2003-2008 Fabrice Bellard >>> root@gogo:~# lsmod | grep vhost >>> vhost_net 27436 3 >>> tun 18337 8 vhost_net >>> macvtap 17633 1 vhost_net >>> >>> >>> Command: iperf -c 192.168.100.x -t 60 >>> >>> >>> Server (FreeBSD 9.2 VM): >>> >>> root@umarotest:~ # uname -a >>> FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan >>> 11 03:25:02 UTC 2014 >>> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC >>> amd64 >>> root@umarotest:~ # iperf -s >>> ------------------------------------------------------------ >>> Server listening on TCP port 5001 >>> TCP window size: 64.0 KByte (default) >>> ------------------------------------------------------------ >>> [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 >>> port 58996 >>> [ ID] Interval Transfer Bandwidth >>> [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec >>> [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 >>> port 58997 >>> [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec >>> [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 >>> port 58998 >>> [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec >>> [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 >>> port 58999 >>> [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec >>> [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 >>> port 59000 >>> [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec >>> >>> While pinging out from the server to the client, I do not get any >>> errors. >>> >>> >>> root@umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 >>> r260430: Wed Jan 8 05:10:04 UTC 2014 >>> root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC >>> amd64 >>> root@umaro:~ # iperf -s >>> ------------------------------------------------------------ >>> Server listening on TCP port 5001 >>> TCP window size: 64.0 KByte (default) >>> ------------------------------------------------------------ >>> [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 >>> port >>> 50264 >>> [ ID] Interval Transfer Bandwidth >>> [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec >>> [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 >>> port >>> 50265 >>> [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec >>> [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 >>> port >>> 50266 >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec >>> [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 >>> port >>> 50267 >>> [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec >>> [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 >>> port >>> 50268 >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec >>> >>> *** While pinging out from the server to client, frequent "ping: >>> sendto: No space left on device" errors *** >>> >>> >>> After a while, I can also reliably re-produce more egregious "ping: >>> sendto: No buffer space available" errors after doing a large sequential >>> write over NFS: >>> >>> mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5: >>> /storage/shared >>> /mnt/nfs >>> dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 >>> >>> I am going to file a freebsd bug report as well. >>> >>> Thanks, >>> Eric >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >>> > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >