From owner-freebsd-stable@FreeBSD.ORG Tue Feb 4 06:23:43 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 98299CB3; Tue, 4 Feb 2014 06:23:43 +0000 (UTC) Received: from mail-ob0-x22f.google.com (mail-ob0-x22f.google.com [IPv6:2607:f8b0:4003:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 388BF1791; Tue, 4 Feb 2014 06:23:43 +0000 (UTC) Received: by mail-ob0-f175.google.com with SMTP id wn1so8953778obc.34 for ; Mon, 03 Feb 2014 22:23:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=Q8VKVXpF+0BDPraLwOmpztuzG/gAUwa39pi6qMdMfrQ=; b=n3W9tSOUk3doY1T6Ie6rzfWD+QBjpwp9sg9dn0bVUVWWVFwZDENFd+MZdb25rS8J3b yitXkCHFnP9Wc8+8ocSLO8RpNWhh4l+Q9zCRk9a/Wfq+MuX2rZrtYrPS6yA3afgOqABK yDHms2eO/YaLR9gI+qfC0opj59j9Q6fwIMIft0S5s54JTwuh5aZW94Rld+qBrEnrzD2k 8qtNu9sdd0pRwpt5Ru36q8U7dXxGO8QBKZTOiQe98ymK19oFv0AHNX4B++rp640iQLRr MaAYI1fEAX8jVzLtMSOQpppVv2xMKDjWe+4n5ZVP/Ug1SBYnNZKNABHfZQ6wuppg+YIf Nr4A== X-Received: by 10.182.22.135 with SMTP id d7mr33523044obf.1.1391495022455; Mon, 03 Feb 2014 22:23:42 -0800 (PST) MIME-Version: 1.0 Sender: mr.kodiak@gmail.com Received: by 10.60.173.206 with HTTP; Mon, 3 Feb 2014 22:23:11 -0800 (PST) In-Reply-To: <449096679.2303214.1391470484784.JavaMail.root@uoguelph.ca> References: <52DBE19C.4040101@freebsd.org> <449096679.2303214.1391470484784.JavaMail.root@uoguelph.ca> From: Bryan Venteicher Date: Tue, 4 Feb 2014 00:23:11 -0600 X-Google-Sender-Auth: LoUe5p5mdOnK3uIUZRH6gNAyEpg Message-ID: Subject: Re: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 To: Rick Macklem Content-Type: multipart/mixed; boundary=001a11c2e19085471b04f18eaebd X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: Bryan Venteicher , Alfred Perlstein , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Feb 2014 06:23:43 -0000 --001a11c2e19085471b04f18eaebd Content-Type: text/plain; charset=ISO-8859-1 On Mon, Feb 3, 2014 at 5:34 PM, Rick Macklem wrote: > Alfred Perlstein wrote: > > What does vmstat say about memory zones? I think vmstat -m/vmstat -z > > and also netstat -m? > > > > Are you set with the same mbufs on both 9 and 10? > > > > -Alfred > > > > > > On 1/18/14 1:32 PM, Eric Dombroski wrote: > > > Adrian: > > > > > > Yes, no change. > > > > > > -Eric > > > > > > > > > > > > > > > > > > On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd > > > wrote: > > > > > >> Hi, > > >> > > >> Have you tried disabling tso? > > >> > > >> Adrian > > >> On Jan 18, 2014 1:52 PM, "Eric Dombroski" > > >> wrote: > > >> > > >>> Hello: > > >>> > > >>> I believe there is a major performance regression between FreeBSD > > >>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers > > >>> (vtnet) and > > >>> handling incoming traffic. Below are the results of some iperf > > >>> tests and > > >>> large dd operations over NFS. Write throughput goes from ~40Gbps > > >>> to > > >>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection > > >>> becomes > > >>> unstable > > >>> ("no buffer space available"), requiring the interface to be > > >>> taken > > >>> down/up. > > >>> > > >>> > > >>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl > > >>> tweaks > > >>> on > > >>> either system. > > >>> > > >>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe > > >>> passthrough, although I suspect the problem manifests itself over > > >>> 1Gbps > > >>> speeds anyway. > > >>> > > >>> Tests: > > >>> > > >>> Client (host): > > >>> root@gogo:~# uname -a > > >>> Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 > > >>> GNU/Linux > > >>> root@gogo:~# kvm -version > > >>> QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), > > >>> Copyright > > >>> (c) 2003-2008 Fabrice Bellard > > >>> root@gogo:~# lsmod | grep vhost > > >>> vhost_net 27436 3 > > >>> tun 18337 8 vhost_net > > >>> macvtap 17633 1 vhost_net > > >>> > > >>> > > >>> Command: iperf -c 192.168.100.x -t 60 > > >>> > > >>> > > >>> Server (FreeBSD 9.2 VM): > > >>> > > >>> root@umarotest:~ # uname -a > > >>> FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 > > >>> #0: Sat Jan > > >>> 11 03:25:02 UTC 2014 > > >>> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > > >>> amd64 > > >>> root@umarotest:~ # iperf -s > > >>> ------------------------------------------------------------ > > >>> Server listening on TCP port 5001 > > >>> TCP window size: 64.0 KByte (default) > > >>> ------------------------------------------------------------ > > >>> [ 4] local 192.168.100.44 port 5001 connected with > > >>> 192.168.100.1 > > >>> port 58996 > > >>> [ ID] Interval Transfer Bandwidth > > >>> [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec > > >>> [ 5] local 192.168.100.44 port 5001 connected with > > >>> 192.168.100.1 > > >>> port 58997 > > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec > > >>> [ 4] local 192.168.100.44 port 5001 connected with > > >>> 192.168.100.1 > > >>> port 58998 > > >>> [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec > > >>> [ 5] local 192.168.100.44 port 5001 connected with > > >>> 192.168.100.1 > > >>> port 58999 > > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec > > >>> [ 4] local 192.168.100.44 port 5001 connected with > > >>> 192.168.100.1 > > >>> port 59000 > > >>> [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec > > >>> > > >>> While pinging out from the server to the client, I do not > > >>> get any > > >>> errors. > > >>> > > >>> > > >>> root@umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD > > >>> 10.0-RC5 #0 > > >>> r260430: Wed Jan 8 05:10:04 UTC 2014 > > >>> root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC > > >>> amd64 > > >>> root@umaro:~ # iperf -s > > >>> ------------------------------------------------------------ > > >>> Server listening on TCP port 5001 > > >>> TCP window size: 64.0 KByte (default) > > >>> ------------------------------------------------------------ > > >>> [ 4] local 192.168.100.5 port 5001 connected with > > >>> 192.168.100.1 > > >>> port > > >>> 50264 > > >>> [ ID] Interval Transfer Bandwidth > > >>> [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec > > >>> [ 5] local 192.168.100.5 port 5001 connected with > > >>> 192.168.100.1 > > >>> port > > >>> 50265 > > >>> [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec > > >>> [ 4] local 192.168.100.5 port 5001 connected with > > >>> 192.168.100.1 > > >>> port > > >>> 50266 > > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec > > >>> [ 5] local 192.168.100.5 port 5001 connected with > > >>> 192.168.100.1 > > >>> port > > >>> 50267 > > >>> [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec > > >>> [ 4] local 192.168.100.5 port 5001 connected with > > >>> 192.168.100.1 > > >>> port > > >>> 50268 > > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec > > >>> > > >>> *** While pinging out from the server to client, frequent > > >>> "ping: > > >>> sendto: No space left on device" errors *** > > >>> > > >>> > > >>> After a while, I can also reliably re-produce more > > >>> egregious "ping: > > >>> sendto: No buffer space available" errors after doing a large > > >>> sequential > > >>> write over NFS: > > >>> > > >>> mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5: > > >>> /storage/shared > > >>> /mnt/nfs > > >>> dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 > > >>> > > >>> I am going to file a freebsd bug report as well. > > >>> > Are you able to test the driver from an up to date head? > Bryan Venteicher just committed some changes to the driver > that increase the size of the segments list (# of mbufs in chain) > and also switches it from using m_collapse() to m_defrag(). > I have no idea if these might affect what you are seeing? > > Yes, please try HEAD if you can, but I think you can take the driver from HEAD on 10 without too much work. I'll MFC my recent changes to 10 in a week or so. As for the "No buffer space available" error, can you try the attached patch [1]? It is from HEAD, but should apply to 10 as well. I have not been able to recreate this but I suspect it might fix it. [1] - http://people.freebsd.org/~bryanv/patches/vtnet_watchdog.patch rick > > > >>> Thanks, > > >>> Eric > > >>> _______________________________________________ > > >>> freebsd-stable@freebsd.org mailing list > > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > >>> To unsubscribe, send any mail to > > >>> "freebsd-stable-unsubscribe@freebsd.org" > > >>> > > > _______________________________________________ > > > freebsd-stable@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > > To unsubscribe, send any mail to > > > "freebsd-stable-unsubscribe@freebsd.org" > > > > > > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to > > "freebsd-stable-unsubscribe@freebsd.org" > > > --001a11c2e19085471b04f18eaebd Content-Type: text/x-patch; charset=US-ASCII; name="vtnet_watchdog.patch" Content-Disposition: attachment; filename="vtnet_watchdog.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hr8s11pf0 Y29tbWl0IDNmMjU3NTM4Nzk4YjQ4ZGQ3MjdlNzljOGFmNmIyYTExYmQ1MDJlNWIKQXV0aG9yOiBC cnlhbiBWZW50ZWljaGVyIDxicnlhbnZAZGFlbW9uaW50aGVjbG9zZXQub3JnPgpEYXRlOiAgIFR1 ZSBGZWIgNCAwMDowODo1MiAyMDE0IC0wNjAwCgogICAgU2ltdWxhdGUgYW4gaW50ZXJydXB0IGlm IHRoZSB3YXRjaGRvZyBkcmFpbmVkIHRoZSBUeCBjb21wbGV0ZSBxdWV1ZQoKZGlmZiAtLWdpdCBh L3N5cy9kZXYvdmlydGlvL25ldHdvcmsvaWZfdnRuZXQuYyBiL3N5cy9kZXYvdmlydGlvL25ldHdv cmsvaWZfdnRuZXQuYwppbmRleCBjYzRhZGViLi45M2RlNTA0IDEwMDY0NAotLS0gYS9zeXMvZGV2 L3ZpcnRpby9uZXR3b3JrL2lmX3Z0bmV0LmMKKysrIGIvc3lzL2Rldi92aXJ0aW8vbmV0d29yay9p Zl92dG5ldC5jCkBAIC0xNDksNyArMTQ5LDcgQEAgc3RhdGljIHZvaWQJdnRuZXRfdHhxX3RxX2Rl ZmVycmVkKHZvaWQgKiwgaW50KTsKICNlbmRpZgogc3RhdGljIHZvaWQJdnRuZXRfdHhxX3N0YXJ0 KHN0cnVjdCB2dG5ldF90eHEgKik7CiBzdGF0aWMgdm9pZAl2dG5ldF90eHFfdHFfaW50cih2b2lk ICosIGludCk7Ci1zdGF0aWMgdm9pZAl2dG5ldF90eHFfZW9mKHN0cnVjdCB2dG5ldF90eHEgKik7 CitzdGF0aWMgaW50CXZ0bmV0X3R4cV9lb2Yoc3RydWN0IHZ0bmV0X3R4cSAqKTsKIHN0YXRpYyB2 b2lkCXZ0bmV0X3R4X3ZxX2ludHIodm9pZCAqKTsKIHN0YXRpYyB2b2lkCXZ0bmV0X3R4X3N0YXJ0 X2FsbChzdHJ1Y3QgdnRuZXRfc29mdGMgKik7CiAKQEAgLTIzODUsMTggKzIzODUsMjEgQEAgdnRu ZXRfdHhxX3RxX2ludHIodm9pZCAqeHR4cSwgaW50IHBlbmRpbmcpCiAJVlRORVRfVFhRX1VOTE9D Syh0eHEpOwogfQogCi1zdGF0aWMgdm9pZAorc3RhdGljIGludAogdnRuZXRfdHhxX2VvZihzdHJ1 Y3QgdnRuZXRfdHhxICp0eHEpCiB7CiAJc3RydWN0IHZpcnRxdWV1ZSAqdnE7CiAJc3RydWN0IHZ0 bmV0X3R4X2hlYWRlciAqdHhoZHI7CiAJc3RydWN0IG1idWYgKm07CisJaW50IGRlcTsKIAogCXZx ID0gdHhxLT52dG50eF92cTsKKwlkZXEgPSAwOwogCVZUTkVUX1RYUV9MT0NLX0FTU0VSVCh0eHEp OwogCiAJd2hpbGUgKCh0eGhkciA9IHZpcnRxdWV1ZV9kZXF1ZXVlKHZxLCBOVUxMKSkgIT0gTlVM TCkgewogCQltID0gdHhoZHItPnZ0aF9tYnVmOworCQlkZXErKzsKIAogCQl0eHEtPnZ0bnR4X3N0 YXRzLnZ0eHNfb3BhY2tldHMrKzsKIAkJdHhxLT52dG50eF9zdGF0cy52dHhzX29ieXRlcyArPSBt LT5tX3BrdGhkci5sZW47CkBAIC0yNDA5LDYgKzI0MTIsOCBAQCB2dG5ldF90eHFfZW9mKHN0cnVj dCB2dG5ldF90eHEgKnR4cSkKIAogCWlmICh2aXJ0cXVldWVfZW1wdHkodnEpKQogCQl0eHEtPnZ0 bnR4X3dhdGNoZG9nID0gMDsKKworCXJldHVybiAoZGVxKTsKIH0KIAogc3RhdGljIHZvaWQKQEAg LTI1MTIsOCArMjUxNywxMyBAQCB2dG5ldF93YXRjaGRvZyhzdHJ1Y3QgdnRuZXRfdHhxICp0eHEp CiAJc2MgPSB0eHEtPnZ0bnR4X3NjOwogCiAJVlRORVRfVFhRX0xPQ0sodHhxKTsKLQlpZiAoc2Mt PnZ0bmV0X2ZsYWdzICYgVlRORVRfRkxBR19FVkVOVF9JRFgpCi0JCXZ0bmV0X3R4cV9lb2YodHhx KTsKKwlpZiAoc2MtPnZ0bmV0X2ZsYWdzICYgVlRORVRfRkxBR19FVkVOVF9JRFgpIHsKKwkJaWYg KHR4cS0+dnRudHhfd2F0Y2hkb2cgIT0gMCAmJiB2dG5ldF90eHFfZW9mKHR4cSkgIT0gMCkgewor CQkJdGFza3F1ZXVlX2VucXVldWUodHhxLT52dG50eF90cSwgJnR4cS0+dnRudHhfaW50cnRhc2sp OworCQkJVlRORVRfVFhRX1VOTE9DSyh0eHEpOworCQkJcmV0dXJuICgwKTsKKwkJfQorCX0KIAlp ZiAodHhxLT52dG50eF93YXRjaGRvZyA9PSAwIHx8IC0tdHhxLT52dG50eF93YXRjaGRvZykgewog CQlWVE5FVF9UWFFfVU5MT0NLKHR4cSk7CiAJCXJldHVybiAoMCk7Cg== --001a11c2e19085471b04f18eaebd--