From owner-freebsd-stable@FreeBSD.ORG Sat Jan 18 18:52:07 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 081D6AEB for ; Sat, 18 Jan 2014 18:52:07 +0000 (UTC) Received: from mail-wg0-f52.google.com (mail-wg0-f52.google.com [74.125.82.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7FF331751 for ; Sat, 18 Jan 2014 18:52:06 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id b13so5697074wgh.7 for ; Sat, 18 Jan 2014 10:51:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=nQu0hXR4x1893Vl3FPhD0JXXmkxit3hkG/SKVtL/B6k=; b=iT7QKBknpORRPuCBKBHl9w4Se8wgU0f6BAT8ELuBFHa0Z9OciYbr/ZIqqXEgTKE0pi /rmsjhZm/uWRPzmrmEeVj2++kniO66AKVUxV1mYs1/k89SnVp5rdQj3RkuF86wxg4akq n1C6jbi/71vlLpNQmkjxSHXQQWpmf9HMeuyLCp5zcIPwiRjBGvzbZ6GsakS2KDtuZg3U lWDNnXjUdkrfPyCxyLLhSN8NyjRjj5uKzcxGEQN3ISYmv/XwiDSzo/VjSSxhJ6L+Pb92 asThnVg4KGO/yTzKZjHih7DoV2Mu5gJUpfMsqn743e2cqr76weerh6aELqXwToWPBLsZ Ejeg== X-Gm-Message-State: ALoCoQl5DQ1q8UZfPOWoUGTurkEfHs82UiJHn8XFH9hgKsICi6MhRoiJZqhBkNZuPRjZ2WmPXVys MIME-Version: 1.0 X-Received: by 10.194.171.34 with SMTP id ar2mr65447wjc.81.1390071118780; Sat, 18 Jan 2014 10:51:58 -0800 (PST) Received: by 10.216.182.71 with HTTP; Sat, 18 Jan 2014 10:51:58 -0800 (PST) Date: Sat, 18 Jan 2014 13:51:58 -0500 Message-ID: Subject: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 From: Eric Dombroski To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jan 2014 18:52:07 -0000 Hello: I believe there is a major performance regression between FreeBSD 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and handling incoming traffic. Below are the results of some iperf tests and large dd operations over NFS. Write throughput goes from ~40Gbps to ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes unstable ("no buffer space available"), requiring the interface to be taken down/up. These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks on either system. I can't reproduce this using an Intel 1Gbps ethernet through PCIe passthrough, although I suspect the problem manifests itself over 1Gbps speeds anyway. Tests: Client (host): root@gogo:~# uname -a Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux root@gogo:~# kvm -version QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright (c) 2003-2008 Fabrice Bellard root@gogo:~# lsmod | grep vhost vhost_net 27436 3 tun 18337 8 vhost_net macvtap 17633 1 vhost_net Command: iperf -c 192.168.100.x -t 60 Server (FreeBSD 9.2 VM): root@umarotest:~ # uname -a FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan 11 03:25:02 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 root@umarotest:~ # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58996 [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58997 [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58998 [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58999 [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 59000 [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec While pinging out from the server to the client, I do not get any errors. root@umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 r260430: Wed Jan 8 05:10:04 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@umaro:~ # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50264 [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50265 [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50266 [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50267 [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50268 [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec *** While pinging out from the server to client, frequent "ping: sendto: No space left on device" errors *** After a while, I can also reliably re-produce more egregious "ping: sendto: No buffer space available" errors after doing a large sequential write over NFS: mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:/storage/shared /mnt/nfs dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 I am going to file a freebsd bug report as well. Thanks, Eric