Date: Tue, 4 Feb 2014 00:23:11 -0600 From: Bryan Venteicher <bryanv@freebsd.org> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Bryan Venteicher <bryanv@freebsd.org>, Alfred Perlstein <alfred@freebsd.org>, freebsd-stable@freebsd.org Subject: Re: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5 Message-ID: <CAGaYwLc-111jEF2moA8da0Nzwno=HVebcbRZ%2B2bBFFSGOkWEoQ@mail.gmail.com> In-Reply-To: <449096679.2303214.1391470484784.JavaMail.root@uoguelph.ca> References: <52DBE19C.4040101@freebsd.org> <449096679.2303214.1391470484784.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
On Mon, Feb 3, 2014 at 5:34 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> Alfred Perlstein wrote:
> > What does vmstat say about memory zones? I think vmstat -m/vmstat -z
> > and also netstat -m?
> >
> > Are you set with the same mbufs on both 9 and 10?
> >
> > -Alfred
> >
> >
> > On 1/18/14 1:32 PM, Eric Dombroski wrote:
> > > Adrian:
> > >
> > > Yes, no change.
> > >
> > > -Eric
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd
> > > <adrian.chadd@gmail.com>wrote:
> > >
> > >> Hi,
> > >>
> > >> Have you tried disabling tso?
> > >>
> > >> Adrian
> > >> On Jan 18, 2014 1:52 PM, "Eric Dombroski" <eric@edombroski.com>
> > >> wrote:
> > >>
> > >>> Hello:
> > >>>
> > >>> I believe there is a major performance regression between FreeBSD
> > >>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers
> > >>> (vtnet) and
> > >>> handling incoming traffic. Below are the results of some iperf
> > >>> tests and
> > >>> large dd operations over NFS. Write throughput goes from ~40Gbps
> > >>> to
> > >>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection
> > >>> becomes
> > >>> unstable
> > >>> ("no buffer space available"), requiring the interface to be
> > >>> taken
> > >>> down/up.
> > >>>
> > >>>
> > >>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl
> > >>> tweaks
> > >>> on
> > >>> either system.
> > >>>
> > >>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe
> > >>> passthrough, although I suspect the problem manifests itself over
> > >>> 1Gbps
> > >>> speeds anyway.
> > >>>
> > >>> Tests:
> > >>>
> > >>> Client (host):
> > >>> root@gogo:~# uname -a
> > >>> Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
> > >>> GNU/Linux
> > >>> root@gogo:~# kvm -version
> > >>> QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian),
> > >>> Copyright
> > >>> (c) 2003-2008 Fabrice Bellard
> > >>> root@gogo:~# lsmod | grep vhost
> > >>> vhost_net 27436 3
> > >>> tun 18337 8 vhost_net
> > >>> macvtap 17633 1 vhost_net
> > >>>
> > >>>
> > >>> Command: iperf -c 192.168.100.x -t 60
> > >>>
> > >>>
> > >>> Server (FreeBSD 9.2 VM):
> > >>>
> > >>> root@umarotest:~ # uname -a
> > >>> FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3
> > >>> #0: Sat Jan
> > >>> 11 03:25:02 UTC 2014
> > >>> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
> > >>> amd64
> > >>> root@umarotest:~ # iperf -s
> > >>> ------------------------------------------------------------
> > >>> Server listening on TCP port 5001
> > >>> TCP window size: 64.0 KByte (default)
> > >>> ------------------------------------------------------------
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58996
> > >>> [ ID] Interval Transfer Bandwidth
> > >>> [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec
> > >>> [ 5] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58997
> > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58998
> > >>> [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec
> > >>> [ 5] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58999
> > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 59000
> > >>> [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec
> > >>>
> > >>> While pinging out from the server to the client, I do not
> > >>> get any
> > >>> errors.
> > >>>
> > >>>
> > >>> root@umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD
> > >>> 10.0-RC5 #0
> > >>> r260430: Wed Jan 8 05:10:04 UTC 2014
> > >>> root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
> > >>> amd64
> > >>> root@umaro:~ # iperf -s
> > >>> ------------------------------------------------------------
> > >>> Server listening on TCP port 5001
> > >>> TCP window size: 64.0 KByte (default)
> > >>> ------------------------------------------------------------
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50264
> > >>> [ ID] Interval Transfer Bandwidth
> > >>> [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec
> > >>> [ 5] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50265
> > >>> [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50266
> > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec
> > >>> [ 5] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50267
> > >>> [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50268
> > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec
> > >>>
> > >>> *** While pinging out from the server to client, frequent
> > >>> "ping:
> > >>> sendto: No space left on device" errors ***
> > >>>
> > >>>
> > >>> After a while, I can also reliably re-produce more
> > >>> egregious "ping:
> > >>> sendto: No buffer space available" errors after doing a large
> > >>> sequential
> > >>> write over NFS:
> > >>>
> > >>> mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:
> > >>> /storage/shared
> > >>> /mnt/nfs
> > >>> dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000
> > >>>
> > >>> I am going to file a freebsd bug report as well.
> > >>>
> Are you able to test the driver from an up to date head?
> Bryan Venteicher just committed some changes to the driver
> that increase the size of the segments list (# of mbufs in chain)
> and also switches it from using m_collapse() to m_defrag().
> I have no idea if these might affect what you are seeing?
>
>
Yes, please try HEAD if you can, but I think you can take the driver from
HEAD on 10 without too much work. I'll MFC my recent changes to 10 in a
week or so.
As for the "No buffer space available" error, can you try the attached
patch [1]? It is from HEAD, but should apply to 10 as well. I have not been
able to recreate this but I suspect it might fix it.
[1] - http://people.freebsd.org/~bryanv/patches/vtnet_watchdog.patch
rick
>
> > >>> Thanks,
> > >>> Eric
> > >>> _______________________________________________
> > >>> freebsd-stable@freebsd.org mailing list
> > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > >>> To unsubscribe, send any mail to
> > >>> "freebsd-stable-unsubscribe@freebsd.org"
> > >>>
> > > _______________________________________________
> > > freebsd-stable@freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > > To unsubscribe, send any mail to
> > > "freebsd-stable-unsubscribe@freebsd.org"
> > >
> >
> > _______________________________________________
> > freebsd-stable@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to
> > "freebsd-stable-unsubscribe@freebsd.org"
> >
>
[-- Attachment #2 --]
commit 3f257538798b48dd727e79c8af6b2a11bd502e5b
Author: Bryan Venteicher <bryanv@daemoninthecloset.org>
Date: Tue Feb 4 00:08:52 2014 -0600
Simulate an interrupt if the watchdog drained the Tx complete queue
diff --git a/sys/dev/virtio/network/if_vtnet.c b/sys/dev/virtio/network/if_vtnet.c
index cc4adeb..93de504 100644
--- a/sys/dev/virtio/network/if_vtnet.c
+++ b/sys/dev/virtio/network/if_vtnet.c
@@ -149,7 +149,7 @@ static void vtnet_txq_tq_deferred(void *, int);
#endif
static void vtnet_txq_start(struct vtnet_txq *);
static void vtnet_txq_tq_intr(void *, int);
-static void vtnet_txq_eof(struct vtnet_txq *);
+static int vtnet_txq_eof(struct vtnet_txq *);
static void vtnet_tx_vq_intr(void *);
static void vtnet_tx_start_all(struct vtnet_softc *);
@@ -2385,18 +2385,21 @@ vtnet_txq_tq_intr(void *xtxq, int pending)
VTNET_TXQ_UNLOCK(txq);
}
-static void
+static int
vtnet_txq_eof(struct vtnet_txq *txq)
{
struct virtqueue *vq;
struct vtnet_tx_header *txhdr;
struct mbuf *m;
+ int deq;
vq = txq->vtntx_vq;
+ deq = 0;
VTNET_TXQ_LOCK_ASSERT(txq);
while ((txhdr = virtqueue_dequeue(vq, NULL)) != NULL) {
m = txhdr->vth_mbuf;
+ deq++;
txq->vtntx_stats.vtxs_opackets++;
txq->vtntx_stats.vtxs_obytes += m->m_pkthdr.len;
@@ -2409,6 +2412,8 @@ vtnet_txq_eof(struct vtnet_txq *txq)
if (virtqueue_empty(vq))
txq->vtntx_watchdog = 0;
+
+ return (deq);
}
static void
@@ -2512,8 +2517,13 @@ vtnet_watchdog(struct vtnet_txq *txq)
sc = txq->vtntx_sc;
VTNET_TXQ_LOCK(txq);
- if (sc->vtnet_flags & VTNET_FLAG_EVENT_IDX)
- vtnet_txq_eof(txq);
+ if (sc->vtnet_flags & VTNET_FLAG_EVENT_IDX) {
+ if (txq->vtntx_watchdog != 0 && vtnet_txq_eof(txq) != 0) {
+ taskqueue_enqueue(txq->vtntx_tq, &txq->vtntx_intrtask);
+ VTNET_TXQ_UNLOCK(txq);
+ return (0);
+ }
+ }
if (txq->vtntx_watchdog == 0 || --txq->vtntx_watchdog) {
VTNET_TXQ_UNLOCK(txq);
return (0);
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGaYwLc-111jEF2moA8da0Nzwno=HVebcbRZ%2B2bBFFSGOkWEoQ>