From owner-freebsd-net@freebsd.org Mon Jun 29 20:49:13 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D8DD9909B3 for ; Mon, 29 Jun 2015 20:49:13 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 08A8B1CC2 for ; Mon, 29 Jun 2015 20:49:12 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DUAACZrpFV/61jaINYA4M0MV8Ggxi6DglxBWYFhTNKAoF4FAEBAQEBAQGBCoQjAQEEAQEBICsgCxACAQgOCgICDRkCAicBCSYCBAgHBAEcBIgODbMYlkcBAQEBAQEBAQEBAQEBAQEBAQEBAQEXgSGKKYQ0AQEcJBAHEYIcDC8SgTEFhVqBKoUYh2iDel+EMYQFQpJig1sCJmOBJIIPIjEBAQWBBTqBAgEBAQ X-IronPort-AV: E=Sophos;i="5.15,372,1432612800"; d="scan'208";a="220906425" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 29 Jun 2015 16:49:12 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id CA31E15F533; Mon, 29 Jun 2015 16:49:11 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id sVSsMTdHTvaQ; Mon, 29 Jun 2015 16:49:11 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 32CD515F538; Mon, 29 Jun 2015 16:49:11 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id BYMPwDnAfVKJ; Mon, 29 Jun 2015 16:49:11 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 1987915F533; Mon, 29 Jun 2015 16:49:11 -0400 (EDT) Date: Mon, 29 Jun 2015 16:49:11 -0400 (EDT) From: Rick Macklem To: Scott Larson Cc: freebsd-net@freebsd.org, carsten aulbert , Gerrit =?utf-8?B?S8O8aG4=?= Message-ID: <716312459.1492862.1435610951085.JavaMail.zimbra@uoguelph.ca> In-Reply-To: References: <20150625145238.12cf9da3b368ef0b9a30f193@aei.mpg.de> <1629011632.413406.1435365728977.JavaMail.zimbra@uoguelph.ca> <20150629091958.af9720d478a8903ab28adc1d@aei.mpg.de> <1542755683.900028.1435580547450.JavaMail.zimbra@uoguelph.ca> Subject: Re: NFS on 10G interface terribly slow MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: NFS on 10G interface terribly slow Thread-Index: rxB4/LLCssngG4Vps3yd1jST4Tl4zw== X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 20:49:13 -0000 Scott Larson wrote: > 82599 in our case. One problem I do have is the stack likes to blow up > on occasion with the right combo of high load and high throughput while TSO > is enabled, possibly relating to the 10.x driver issue you've pointed out. > But when it comes to the throughput they'll blast 10G with no problem. > Thanks for the info. So long as your mbuf cluster pool is large enough, I think the m_defrag() calls will just result in increased CPU overheads and probably don't introduce much delay. I have no idea why the stack would blow up sometimes. If you can catch the backtrace for one of these and post it, it might become obvious. (Or you could just try increasing KSTACK_PAGES in sys/amd64/include/param.h and see if the stack still blows up. Alternately, I think you can set KSTACK_PAGES in your kernel config file.) rick > > *[image: userimage]Scott Larson[image: los angeles] > Lead > Systems Administrator[image: wdlogo] [image: > linkedin] [image: facebook] > [image: twitter] > [image: instagram] > T 310 823 8238 x1106 > <310%20823%208238%20x1106> | M 310 904 8818 <310%20904%208818>* > > On Mon, Jun 29, 2015 at 5:22 AM, Rick Macklem wrote: > > > Gerrit Kuhn wrote: > > > On Fri, 26 Jun 2015 20:42:08 -0400 (EDT) Rick Macklem > > > wrote about Re: NFS on 10G interface terribly > > slow: > > > > > > RM> Btw, can you tell us what Intel chip(s) you're using? > > > > > > I have > > > > > > ix0@pci0:5:0:0: class=0x020000 card=0x00028086 chip=0x15288086 rev=0x01 > > > hdr=0x00 vendor = 'Intel Corporation' > > > device = 'Ethernet Controller 10-Gigabit X540-AT2' > > > class = network > > > subclass = ethernet > > > > > Yea, I don't know how to decode this either. I was actually interested in > > what chip Scott was using and getting wire speed. > > As noted in the other reply, since disabling TSO didn't help, you probably > > aren't affected by this issue. > > > > rick > > > > > RM> For example, from the "ix" driver: > > > RM> #define IXGBE_82598_SCATTER 100 > > > RM> #define IXGBE_82599_SCATTER 32 > > > > > > Hm, I cannot find out into which chipset number this translates for my > > > device... > > > > > > RM> Btw, it appears that the driver in head/current now sets > > > RM> if_hw_tsomaxsegcount, but the driver in stable/10 does not. This > > means > > > RM> that the 82599 chip will end up doing the m_defrag() calls for 10.x. > > > > > > So the next step could even be updating to -current... > > > OTOH, I get the same (bad) resulsts, no matter if TSO is enabled or > > > disabled on the interface. > > > > > > > > > cu > > > Gerrit > > > _______________________________________________ > > > freebsd-net@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >