From owner-freebsd-net@freebsd.org Fri Jun 26 09:56:17 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77BFE98C165 for ; Fri, 26 Jun 2015 09:56:17 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EA84D1899 for ; Fri, 26 Jun 2015 09:56:16 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 1348F20076E; Fri, 26 Jun 2015 11:56:13 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 0431C405881; Fri, 26 Jun 2015 11:56:13 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id 481E7405880; Fri, 26 Jun 2015 11:56:12 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([10.117.15.110]) by intranet.aei.uni-hannover.de (Lotus Domino Release 8.5.3FP6HF1691) with ESMTP id 2015062611560192-65989 ; Fri, 26 Jun 2015 11:56:01 +0200 Date: Fri, 26 Jun 2015 11:56:02 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Scott Larson Cc: freebsd-net@freebsd.org, "carsten.aulbert@aei.mpg.de" Subject: Re: NFS on 10G interface terribly slow Message-Id: <20150626115602.5ebcc82489bc9f3bf357b236@aei.mpg.de> In-Reply-To: References: <20150625145238.12cf9da3b368ef0b9a30f193@aei.mpg.de> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 8.5.3FP6HF1691 | May 7, 2015) at 26.06.2015 11:56:01, Serialize by Router on intranet/aei-hannover(Release 8.5.3FP6HF1691 | May 7, 2015) at 26.06.2015 11:56:12, Serialize complete at 26.06.2015 11:56:12 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2015.6.26.94517 X-PerlMx-Spam: Gauge=X, Probability=10%, Report=' URI_HOSTNAME_CONTAINS_EQUALS 0.4, MULTIPLE_RCPTS 0.1, HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_1900_1999 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, NO_URI_HTTPS 0, REFERENCES 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __MULTIPLE_RCPTS_CC_X2 0, __REFERENCES 0, __SANE_MSGID 0, __STOCK_PHRASE_6 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __URI_NO_PATH 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jun 2015 09:56:17 -0000 On Thu, 25 Jun 2015 12:56:36 -0700 Scott Larson wrote about Re: NFS on 10G interface terribly slow: SL> We've got 10.0 and 10.1 servers accessing Isilon and Nexenta via SL> NFS with Intel 10G gear and bursting to near wire speed with the stock SL> MTU/rsize/wsize works as expected. That sound promising. So we should be able to improve here, too. SL> TSO definitely needs to be enabled for that performance. Ok, I switched it back on. SL> Other things to look at: Are all the servers involved negotiating the SL> correct speed and duplex, with TSO? We have a direct link between the systems, with only one switch in-between acting as a transceiver to get from fibre to copper media. Both machines and the switch show a 10G full-duplex link, not a single error or collision to be spotted. The switch only carries these two lines, nothing else. SL> Does it need to have the network SL> stack tuned with whatever it's equivalent of maxsockbuf and SL> send/recvbuf are? On the FreeBSD side we set kern.ipc.maxsockbuf=33554432 net.inet.tcp.sendbuf_max=33554432 net.inet.tcp.recvbuf_max=33554432 I don't know what the equivalent for Solaris would be, still doing research on that. SL> Do the switch ports and NIC counters show any drops SL> or errors? No, nothing bad to be seen there. SL> On the FBSD servers you could also run 'netstat -i -w 1' SL> under load to see if drops are occurring locally, or 'systat -vmstat' SL> for resource contention problems. But again, a similar setup here and SL> no such issues have appeared. No errors, no collisions, no drops. I cannot spot any bottlenecks in netstat, either. One thing I just wonder about is that all IRQs (about 700 under load) are routed to only one quque on the ix interface (there seems to be one per core by default). Should the load spread, or is that expected behaviour? cu Gerrit