From owner-freebsd-net@FreeBSD.ORG Tue Jan 28 14:55:59 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A5DA17F9; Tue, 28 Jan 2014 14:55:59 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 42B59151A; Tue, 28 Jan 2014 14:55:58 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqQEAFjE51KDaFve/2dsb2JhbABag0RWgn25DE+BJXSCJQEBAQMBAQEBIAQnIAsFFhgCAg0ZAikBCSYGCAcEARwEh1wIDal5n3MXgSmNBQEBGwEzB4JvgUkEiUmMDIQFkG2DSx4xgQQ5 X-IronPort-AV: E=Sophos;i="4.95,736,1384318800"; d="scan'208";a="91045198" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 28 Jan 2014 09:55:57 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 70951B4089; Tue, 28 Jan 2014 09:55:57 -0500 (EST) Date: Tue, 28 Jan 2014 09:55:57 -0500 (EST) From: Rick Macklem To: J David Message-ID: <1098090585.17554698.1390920957454.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: Terrible NFS performance under 9.2-RELEASE? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-net@freebsd.org, wollman@freebsd.org, Bryan Venteicher X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jan 2014 14:55:59 -0000 J David wrote: > Another way to test this is to instrument the virtio driver, which > turned out to be very straightforward: > > Index: if_vtnet.c > > =================================================================== > > --- if_vtnet.c (revision 260701) > > +++ if_vtnet.c (working copy) > > @@ -1886,6 +1887,7 @@ > > return (virtqueue_enqueue(vq, txhdr, &sg, sg.sg_nseg, 0)); > > > > fail: > > + sc->vtnet_stats.tx_excess_mbuf_drop++; > > m_freem(*m_head); > > *m_head = NULL; > > > > @@ -2645,6 +2647,9 @@ > > SYSCTL_ADD_ULONG(ctx, child, OID_AUTO, "tx_task_rescheduled", > > CTLFLAG_RD, &stats->tx_task_rescheduled, > > "Times the transmit interrupt task rescheduled itself"); > > + SYSCTL_ADD_ULONG(ctx, child, OID_AUTO, "tx_excess_mbuf_drop", > > + CTLFLAG_RD, &stats->tx_excess_mbuf_drop, > > + "Times packets were dropped due to excess mbufs"); > > } > > > > static int > > Index: if_vtnetvar.h > > =================================================================== > > --- if_vtnetvar.h (revision 260701) > > +++ if_vtnetvar.h (working copy) > > @@ -48,6 +48,7 @@ > > unsigned long tx_csum_bad_ethtype; > > unsigned long tx_tso_bad_ethtype; > > unsigned long tx_task_rescheduled; > > + unsigned long tx_excess_mbuf_drop; > > }; > > > > struct vtnet_softc { > > > This patch didn't seem harmful from a performance standpoint since if > things are working, the counter increment never gets hit. > > With this change, I re-ran some 64k tests. I found that the number > of > drops was very small, but not zero. > > On the client, doing the write-append test (which has no reads), it > seems like it slowly builds up 8 with what appears to be some sort of > back off (each one takes longer to appear than the last): > > > $ sysctl dev.vtnet.1.tx_excess_mbuf_drop > > dev.vtnet.1.tx_excess_mbuf_drop: 8 > > > But after 8, it appears congestion control is clamped down so hard > that no more happen. > > Once read activity starts, the server builds up more: > > dev.vtnet.1.tx_excess_mbuf_drop: 53 > > > So while there aren't a lot of these, they definitely do exist and > there's just no way they're good for performance. > It would be nice to also count the number of times m_collapse() gets called, since that will generate a lot of overhead that I think will show up on your test, since you don't have any disk activity. And I'd state that having any of these is near-disastrous for performance, since it means a timeout/retransmit of a TCP segment. For a lan environment, I would consider 1 timeout/retransmit in a million packets as a lot. rick ps: I've cc'd Bryan, since he's the guy handling virtio, I think. > Thanks! > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >