From owner-freebsd-amd64@FreeBSD.ORG Sat Jan 18 19:00:01 2014 Return-Path: Delivered-To: freebsd-amd64@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E2B61F65 for ; Sat, 18 Jan 2014 19:00:00 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BC5811796 for ; Sat, 18 Jan 2014 19:00:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s0IJ00Cg006413 for ; Sat, 18 Jan 2014 19:00:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id s0IJ00JN006412; Sat, 18 Jan 2014 19:00:00 GMT (envelope-from gnats) Resent-Date: Sat, 18 Jan 2014 19:00:00 GMT Resent-Message-Id: <201401181900.s0IJ00JN006412@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Eric Dombroski Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E3A27E7D for ; Sat, 18 Jan 2014 18:58:08 +0000 (UTC) Received: from oldred.freebsd.org (oldred.freebsd.org [IPv6:2001:1900:2254:206a::50:4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C304F1783 for ; Sat, 18 Jan 2014 18:58:08 +0000 (UTC) Received: from oldred.freebsd.org ([127.0.1.6]) by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id s0IIw7RT052517 for ; Sat, 18 Jan 2014 18:58:07 GMT (envelope-from nobody@oldred.freebsd.org) Received: (from nobody@localhost) by oldred.freebsd.org (8.14.5/8.14.5/Submit) id s0IIw7Ej052467; Sat, 18 Jan 2014 18:58:07 GMT (envelope-from nobody) Message-Id: <201401181858.s0IIw7Ej052467@oldred.freebsd.org> Date: Sat, 18 Jan 2014 18:58:07 GMT From: Eric Dombroski To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Subject: amd64/185864: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 X-Mailman-Approved-At: Sat, 18 Jan 2014 19:04:52 +0000 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jan 2014 19:00:01 -0000 >Number: 185864 >Category: amd64 >Synopsis: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Jan 18 19:00:00 UTC 2014 >Closed-Date: >Last-Modified: >Originator: Eric Dombroski >Release: 10.0RC5 >Organization: >Environment: FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 r260430: Wed Jan 8 05:10:04 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 >Description: believe there is a major performance regression between FreeBSD 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and handling incoming traffic. Below are the results of some iperf tests and large dd operations over NFS. Write throughput goes from ~40Gbps to ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes unstable ("no buffer space available"), requiring the interface to be taken down/up. These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks on either system. I can't reproduce this using an Intel 1Gbps ethernet through PCIe passthrough, although I suspect the problem manifests itself over 1Gbps speeds anyway. Tests: Client (host): root@gogo:~# uname -a Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux root@gogo:~# kvm -version QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright (c) 2003-2008 Fabrice Bellard root@gogo:~# lsmod | grep vhost vhost_net 27436 3 tun 18337 8 vhost_net macvtap 17633 1 vhost_net Command: iperf -c 192.168.100.x -t 60 Server (FreeBSD 9.2 VM): root@umarotest:~ # uname -a FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan 11 03:25:02 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 root@umarotest:~ # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58996 [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58997 [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58998 [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58999 [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 59000 [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec While pinging out from the server to the client, I do not get any errors. root@umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 r260430: Wed Jan 8 05:10:04 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@umaro:~ # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50264 [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50265 [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50266 [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50267 [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50268 [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec *** While pinging out from the server to client, frequent "ping: sendto: No space left on device" errors *** After a while, I can also reliably re-produce more egregious "ping: sendto: No buffer space available" errors after doing a large sequential write over NFS: mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:/storage/shared /mnt/nfs dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 >How-To-Repeat: Perform iperf test between a KVM host and the FreeBSD 9.2 VM as a server using virtio network drivers while pinging out from FreeBSD system; compare to 10.0RC5...results go from ~40Gbps to ~2.4Gbps and the 10.0RC5 system shows many "no space left on device" errors during pings. Perform large sequential write (dd, mysqldump, etc.) operation over NFS, eventually interface goes down with "no buffer space available" errors during ping. Only remediation at that point is to take down the interface and take it up again. >Fix: >Release-Note: >Audit-Trail: >Unformatted: