From owner-freebsd-net@FreeBSD.ORG Thu Mar 27 21:37:44 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BB12B133; Thu, 27 Mar 2014 21:37:44 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 3F1CF9BF; Thu, 27 Mar 2014 21:37:43 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqsEAN+YNFODaFve/2dsb2JhbABZg0FXgwq4HYYZTVGBNXSCJQEBAQMBAQEBIAQnIAsbDgoCAg0ZAiMGAQkmBggHBAEcBIdEAwkIDa5ymxoNhysXgSmLNIE7CgYCARs0B4JvgUkElXZqgyCLN4VKg0shMXtC X-IronPort-AV: E=Sophos;i="4.97,745,1389762000"; d="scan'208";a="109827248" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 27 Mar 2014 17:37:36 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 72614B404C; Thu, 27 Mar 2014 17:37:36 -0400 (EDT) Date: Thu, 27 Mar 2014 17:37:36 -0400 (EDT) From: Rick Macklem To: Christopher Forgeron Message-ID: <380071870.1832898.1395956256461.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: RFC: How to fix the NFS/iSCSI vs TSO problem MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: pyunyh@gmail.com, FreeBSD Filesystems , Alexander Motin , araujo@freebsd.org, FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 21:37:44 -0000 Christopher Forgeron wrote: > I'm quite sure the problem is on 9.2-RELEASE, not 9.1-RELEASE or > earlier, > as a 9.2-STABLE from last year I have doesn't exhibit the problem. > New > code in if.c at line 660 looks to be what is starting this, which > makes me > wonder how TSO was being handled before 9.2. > > I also like Rick's NFS patch for cluster size. I notice an > improvement, but > don't have solid numbers yet. I'm still stress testing it as we > speak. > Unfortunately, this causes problems for small i386 systems, so I am reluctant to commit it to head. Maybe a variant that is only enabled for amd64 systems with lots of memory would be ok? > > On Wed, Mar 26, 2014 at 11:44 PM, Marcelo Araujo > wrote: > > > Hello All, > > > > > > 2014-03-27 8:27 GMT+08:00 Rick Macklem : > > > > > > Well, bumping it from 32->35 is all it would take for NFS (can't > > > comment > > > w.r.t. iSCSI). ixgbe uses 100 for the 82598 chip and 32 for the > > > 82599 > > > (just so others aren't confused by the above comment). I > > > understand > > > your point was w.r.t. using 100 without blowing the kernel stack, > > > but > > > since the testers have been using "ix" with the 82599 chip which > > > is > > > limited to 32 transmit segments... > > > > > > However, please increase any you know can be safely done from > > > 32->35, > > rick > > > > > > > > I have plenty of machines using Intel X540 that is based on 82599 > > chipset. > > I have applied Rick's patch on ixgbe to check if the packet size is > > bigger > > than 65535 or cluster is bigger than 32. So far till now, on > > FreeBSD > > 9.1-RELEASE this problem does not happens. > > > > Unfortunately all my environment here is based on 9.1-RELEASE, with > > some > > merges from 10-RELEASE such like: NFS and IXGBE. > > I can't see why it couldn't happen on 9.1 or earlier, since it just uses IP_MAXPACKET in tcp_output(). However, to make it happen NFS has to do a read reply (server) or write request (client) that is a little under 64K bytes. Normally the default will be a full 64K bytes, so for the server side it would take a read of a file where the EOF is just shy of the 64K boundary. For the client write it would be a write of a partially dirtied block where most of the block has been dirtied. (Software builds generate a lot of patially dirtied blocks, but I don't know what else would. For sequential writing it would be a file that ends just shy of a 64K boundary (similar to the server side) being written. I think it is more likely your NFS file activity and not 9.1 vs 9.2 that avoids the problem. (I suspect there are quite a few folk running NFS 9.2 or later on these ix chips who don't see the problem as well.) Fortunately you (Christopher) were able to reproduce it, so the problem could be isolated. Thanks everyone for your help with this, rick > > Also I have applied the patch that Rick sent in another email with > > the > > subject 'NFS patch to use pagesize mbuf clusters'. And we can see > > some > > performance boost over 10Gbps Intel. However here at the company, > > we are > > still doing benchmarks. If someone wants to have my benchmark > > result, I can > > send it later. > > > > I'm wondering, if this update on ixgbe from 32->35 could be applied > > also > > for versions < 9.2. I'm thinking, that this problem arise only on > > 9-STABLE > > and consequently on 9.2-RELEASE. And fortunately or not 9.1-RELEASE > > doesn't > > share it. > > > > Best Regards, > > -- > > Marcelo Araujo > > araujo@FreeBSD.org > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > > "freebsd-net-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >