From owner-freebsd-net@freebsd.org Mon Jun 29 12:20:20 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECF1C98ECD3 for ; Mon, 29 Jun 2015 12:20:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 97FB62879 for ; Mon, 29 Jun 2015 12:20:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CdBACUNpFV/61jaINbhEQGxH8CgW4QAQEBAQEBAYEKhCIBAQEDASNWEAIBCBgCAg0ZAgJXAgQTiCcIsG+WCgEBAQEBAQEDAQEBAQEBARuBIYophDsCFTQHgmiBQwWMHIdopA4CJmODMyIxgQRCgQIBAQE X-IronPort-AV: E=Sophos;i="5.13,698,1427774400"; d="scan'208";a="220751015" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 29 Jun 2015 08:20:13 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id EB19D15F533; Mon, 29 Jun 2015 08:20:12 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 8WNhVAubLZOO; Mon, 29 Jun 2015 08:20:12 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 6781115F538; Mon, 29 Jun 2015 08:20:12 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id gwu2KzuRTP-G; Mon, 29 Jun 2015 08:20:12 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 486FB15F533; Mon, 29 Jun 2015 08:20:12 -0400 (EDT) Date: Mon, 29 Jun 2015 08:20:12 -0400 (EDT) From: Rick Macklem To: Gerrit =?utf-8?B?S8O8aG4=?= Cc: Scott Larson , freebsd-net@freebsd.org, carsten aulbert Message-ID: <1467277373.898391.1435580412070.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20150629091958.af9720d478a8903ab28adc1d@aei.mpg.de> References: <20150625145238.12cf9da3b368ef0b9a30f193@aei.mpg.de> <1629011632.413406.1435365728977.JavaMail.zimbra@uoguelph.ca> <20150629091958.af9720d478a8903ab28adc1d@aei.mpg.de> Subject: Re: NFS on 10G interface terribly slow MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: NFS on 10G interface terribly slow Thread-Index: n557BLWTv+gy3b98Cnbfy5ol11/2YQ== X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 12:20:20 -0000 Gerrit Kuhn wrote: > On Fri, 26 Jun 2015 20:42:08 -0400 (EDT) Rick Macklem > wrote about Re: NFS on 10G interface terribly slow: > > RM> Btw, can you tell us what Intel chip(s) you're using? > > I have > > ix0@pci0:5:0:0: class=0x020000 card=0x00028086 chip=0x15288086 rev=0x01 > hdr=0x00 vendor = 'Intel Corporation' > device = 'Ethernet Controller 10-Gigabit X540-AT2' > class = network > subclass = ethernet > > RM> For example, from the "ix" driver: > RM> #define IXGBE_82598_SCATTER 100 > RM> #define IXGBE_82599_SCATTER 32 > > Hm, I cannot find out into which chipset number this translates for my > device... > > RM> Btw, it appears that the driver in head/current now sets > RM> if_hw_tsomaxsegcount, but the driver in stable/10 does not. This means > RM> that the 82599 chip will end up doing the m_defrag() calls for 10.x. > > So the next step could even be updating to -current... > OTOH, I get the same (bad) resulsts, no matter if TSO is enabled or > disabled on the interface. > Since disabling TSO had no effect, I don't think updating would matter. If you can test against a different NFS server, that might indicate whether or not the Solaris server is the bottleneck. If the Solaris server is using ZFS, setting sync=disabled might help w.r.t. write performance. It is, however, somewhat dangerous w.r.t. loss of recently written data when the server crashes. (Server has told client data is safely on stable storage so client will not re-write the block(s) although data wasn't on stable storage and is lost.) (I'm not a ZFS guy, so I can't suggest more w.r.t. ZFS.) rick > > cu > Gerrit >