From owner-freebsd-stable@FreeBSD.ORG Sat Jan 18 21:32:18 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3EC278A9 for ; Sat, 18 Jan 2014 21:32:18 +0000 (UTC) Received: from mail-wg0-f43.google.com (mail-wg0-f43.google.com [74.125.82.43]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BD04112DD for ; Sat, 18 Jan 2014 21:32:17 +0000 (UTC) Received: by mail-wg0-f43.google.com with SMTP id y10so5760809wgg.22 for ; Sat, 18 Jan 2014 13:32:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=cAzyqUDdKSyUcwlWEot9jqIiB756XJOz8ZE1idqS+QE=; b=h4q+IEs3HWc/zTakQHvgdPhDc4h5s0PQXNkEiS4kwDbpA1Lr6/wIQvHfwOjzypeqN3 7ggX90A7ZTy81qPOlGGBEXLAvDU3GafCdqECbLA5Sszy1eFymyGgBYzzX+IX/qDa5gmo TuGyCpDV4O9FnIutgak+408kHNJquea5/2/Xr7eC055VyljFrzzBwTdndhWmrYiXSu9j EYVra2E0lLUqTpCaU3a/sDV4woybz9nj7huMBD8zMDwBFycpJn1N+7GgEMCny7Mxv49f hll8yV0RUM2K5wShJkrm2nH++RHPFdVXfr7l5mkrEwLmR7k9OTeSIc8JIv+qwZ14mRNj CuwA== X-Gm-Message-State: ALoCoQm5dP9DOmLmWfj00IxVv9jCojpNhfCY4Nh8/KHswWPwS8PWdiPVI1uYxsqMMfyhyoV9+00a MIME-Version: 1.0 X-Received: by 10.180.218.171 with SMTP id ph11mr3776695wic.7.1390080730382; Sat, 18 Jan 2014 13:32:10 -0800 (PST) Received: by 10.216.182.71 with HTTP; Sat, 18 Jan 2014 13:32:10 -0800 (PST) In-Reply-To: References: Date: Sat, 18 Jan 2014 16:32:10 -0500 Message-ID: Subject: Re: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 From: Eric Dombroski To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Stable Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jan 2014 21:32:18 -0000 Adrian: Yes, no change. -Eric On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd wrote: > Hi, > > Have you tried disabling tso? > > Adrian > On Jan 18, 2014 1:52 PM, "Eric Dombroski" wrote: > >> Hello: >> >> I believe there is a major performance regression between FreeBSD >> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and >> handling incoming traffic. Below are the results of some iperf tests and >> large dd operations over NFS. Write throughput goes from ~40Gbps to >> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes >> unstable >> ("no buffer space available"), requiring the interface to be taken >> down/up. >> >> >> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks >> on >> either system. >> >> I can't reproduce this using an Intel 1Gbps ethernet through PCIe >> passthrough, although I suspect the problem manifests itself over 1Gbps >> speeds anyway. >> >> Tests: >> >> Client (host): >> root@gogo:~# uname -a >> Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux >> root@gogo:~# kvm -version >> QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright >> (c) 2003-2008 Fabrice Bellard >> root@gogo:~# lsmod | grep vhost >> vhost_net 27436 3 >> tun 18337 8 vhost_net >> macvtap 17633 1 vhost_net >> >> >> Command: iperf -c 192.168.100.x -t 60 >> >> >> Server (FreeBSD 9.2 VM): >> >> root@umarotest:~ # uname -a >> FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan >> 11 03:25:02 UTC 2014 >> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC >> amd64 >> root@umarotest:~ # iperf -s >> ------------------------------------------------------------ >> Server listening on TCP port 5001 >> TCP window size: 64.0 KByte (default) >> ------------------------------------------------------------ >> [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 >> port 58996 >> [ ID] Interval Transfer Bandwidth >> [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec >> [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 >> port 58997 >> [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec >> [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 >> port 58998 >> [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec >> [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 >> port 58999 >> [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec >> [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 >> port 59000 >> [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec >> >> While pinging out from the server to the client, I do not get any >> errors. >> >> >> root@umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 >> r260430: Wed Jan 8 05:10:04 UTC 2014 >> root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC >> amd64 >> root@umaro:~ # iperf -s >> ------------------------------------------------------------ >> Server listening on TCP port 5001 >> TCP window size: 64.0 KByte (default) >> ------------------------------------------------------------ >> [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 >> port >> 50264 >> [ ID] Interval Transfer Bandwidth >> [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec >> [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 >> port >> 50265 >> [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec >> [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 >> port >> 50266 >> [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec >> [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 >> port >> 50267 >> [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec >> [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 >> port >> 50268 >> [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec >> >> *** While pinging out from the server to client, frequent "ping: >> sendto: No space left on device" errors *** >> >> >> After a while, I can also reliably re-produce more egregious "ping: >> sendto: No buffer space available" errors after doing a large sequential >> write over NFS: >> >> mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5: >> /storage/shared >> /mnt/nfs >> dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 >> >> I am going to file a freebsd bug report as well. >> >> Thanks, >> Eric >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> >