From owner-freebsd-net@FreeBSD.ORG Thu Jul 10 19:37:38 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 246AEB97 for ; Thu, 10 Jul 2014 19:37:38 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F2BDD2C9D for ; Thu, 10 Jul 2014 19:37:37 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id EA743B94E; Thu, 10 Jul 2014 15:37:36 -0400 (EDT) From: John Baldwin To: freebsd-net@freebsd.org Subject: Re: NFS client READ performance on -current Date: Thu, 10 Jul 2014 13:25:45 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <870285181.7082888.1404435061501.JavaMail.root@uoguelph.ca> In-Reply-To: <870285181.7082888.1404435061501.JavaMail.root@uoguelph.ca> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201407101325.46156.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 10 Jul 2014 15:37:37 -0400 (EDT) Cc: "Russell L. Carter" , Rick Macklem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 19:37:38 -0000 On Thursday, July 03, 2014 8:51:01 pm Rick Macklem wrote: > Russell L. Carter wrote: > > > > > > On 07/02/14 19:09, Rick Macklem wrote: > > > > > Could you please post the dmesg stuff for the network interface, > > > so I can tell what driver is being used? I'll take a look at it, > > > in case it needs to be changed to use m_defrag(). > > > > em0: port 0xd020-0xd03f > > mem > > 0xfe4a0000-0xfe4bffff,0xfe480000-0xfe49ffff irq 44 at device 0.0 on > > pci2 > > em0: Using an MSI interrupt > > em0: Ethernet address: 00:15:17:bc:29:ba > > 001.000007 [2323] netmap_attach success for em0 tx 1/1024 > > rx > > 1/1024 queues/slots > > > > This is one of those dual nic cards, so there is em1 as well... > > > Well, I took a quick look at the driver and it does use m_defrag(), but > I think that the "retry:" label it does a goto after doing so might be in > the wrong place. > > The attached untested patch might fix this. > > Is it convenient to build a kernel with this patch applied and then try > it with TSO enabled? > > rick > ps: It does have the transmit segment limit set to 32. I have no idea if > this is a hardware limitation. I think the retry is not in the wrong place, but the overhead of all those pullups is apparently quite severe. It would be interesting to test the following in addition to your change to see if it improves performance further: Index: if_em.c =================================================================== --- if_em.c (revision 268495) +++ if_em.c (working copy) @@ -1959,7 +1959,9 @@ retry: if (error == EFBIG && remap) { struct mbuf *m; - m = m_defrag(*m_headp, M_NOWAIT); + m = m_collapse(*m_headp, M_NOWAIT, EM_MAX_SCATTER); + if (m == NULL) + m = m_defrag(*m_headp, M_NOWAIT); if (m == NULL) { adapter->mbuf_alloc_failed++; m_freem(*m_headp); -- John Baldwin