Date: Tue, 28 Jan 2014 01:59:17 -0500 From: J David <j.david.lists@gmail.com> To: wollman@freebsd.org Cc: freebsd-net@freebsd.org, Rick Macklem <rmacklem@uoguelph.ca> Subject: Re: Terrible NFS performance under 9.2-RELEASE? Message-ID: <CABXB=RQjxoe_hQitA7z=LsPx1hyhu7ci9u7G9Ne_KFHTimw%2BZw@mail.gmail.com> In-Reply-To: <CABXB=RRHWd2xo0NChe2KGRbOxUeTJxBh=oH3JfJu%2BEFV-C61pQ@mail.gmail.com> References: <20140128002826.GU13704@funkthat.com> <1415339672.17282775.1390872779067.JavaMail.root@uoguelph.ca> <201401280427.s0S4RTVn077761@hergotha.csail.mit.edu> <CABXB=RRHWd2xo0NChe2KGRbOxUeTJxBh=oH3JfJu%2BEFV-C61pQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Another way to test this is to instrument the virtio driver, which
turned out to be very straightforward:
Index: if_vtnet.c
===================================================================
--- if_vtnet.c (revision 260701)
+++ if_vtnet.c (working copy)
@@ -1886,6 +1887,7 @@
return (virtqueue_enqueue(vq, txhdr, &sg, sg.sg_nseg, 0));
fail:
+ sc->vtnet_stats.tx_excess_mbuf_drop++;
m_freem(*m_head);
*m_head = NULL;
@@ -2645,6 +2647,9 @@
SYSCTL_ADD_ULONG(ctx, child, OID_AUTO, "tx_task_rescheduled",
CTLFLAG_RD, &stats->tx_task_rescheduled,
"Times the transmit interrupt task rescheduled itself");
+ SYSCTL_ADD_ULONG(ctx, child, OID_AUTO, "tx_excess_mbuf_drop",
+ CTLFLAG_RD, &stats->tx_excess_mbuf_drop,
+ "Times packets were dropped due to excess mbufs");
}
static int
Index: if_vtnetvar.h
===================================================================
--- if_vtnetvar.h (revision 260701)
+++ if_vtnetvar.h (working copy)
@@ -48,6 +48,7 @@
unsigned long tx_csum_bad_ethtype;
unsigned long tx_tso_bad_ethtype;
unsigned long tx_task_rescheduled;
+ unsigned long tx_excess_mbuf_drop;
};
struct vtnet_softc {
This patch didn't seem harmful from a performance standpoint since if
things are working, the counter increment never gets hit.
With this change, I re-ran some 64k tests. I found that the number of
drops was very small, but not zero.
On the client, doing the write-append test (which has no reads), it
seems like it slowly builds up 8 with what appears to be some sort of
back off (each one takes longer to appear than the last):
$ sysctl dev.vtnet.1.tx_excess_mbuf_drop
dev.vtnet.1.tx_excess_mbuf_drop: 8
But after 8, it appears congestion control is clamped down so hard
that no more happen.
Once read activity starts, the server builds up more:
dev.vtnet.1.tx_excess_mbuf_drop: 53
So while there aren't a lot of these, they definitely do exist and
there's just no way they're good for performance.
Thanks!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABXB=RQjxoe_hQitA7z=LsPx1hyhu7ci9u7G9Ne_KFHTimw%2BZw>