Date: Fri, 26 Jun 2015 11:56:02 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= <gerrit.kuehn@aei.mpg.de> To: Scott Larson <stl@wiredrive.com> Cc: freebsd-net@freebsd.org, "carsten.aulbert@aei.mpg.de" <carsten.aulbert@aei.mpg.de> Subject: Re: NFS on 10G interface terribly slow Message-ID: <20150626115602.5ebcc82489bc9f3bf357b236@aei.mpg.de> In-Reply-To: <CAFt8naF7xmZW8bgVrhrL=CaPXiVURqDLsNN5-NHDg=hiv-Qmtw@mail.gmail.com> References: <20150625145238.12cf9da3b368ef0b9a30f193@aei.mpg.de> <CAFt8naF7xmZW8bgVrhrL=CaPXiVURqDLsNN5-NHDg=hiv-Qmtw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 25 Jun 2015 12:56:36 -0700 Scott Larson <stl@wiredrive.com> wrote about Re: NFS on 10G interface terribly slow: SL> We've got 10.0 and 10.1 servers accessing Isilon and Nexenta via SL> NFS with Intel 10G gear and bursting to near wire speed with the stock SL> MTU/rsize/wsize works as expected. That sound promising. So we should be able to improve here, too. SL> TSO definitely needs to be enabled for that performance. Ok, I switched it back on. SL> Other things to look at: Are all the servers involved negotiating the SL> correct speed and duplex, with TSO? We have a direct link between the systems, with only one switch in-between acting as a transceiver to get from fibre to copper media. Both machines and the switch show a 10G full-duplex link, not a single error or collision to be spotted. The switch only carries these two lines, nothing else. SL> Does it need to have the network SL> stack tuned with whatever it's equivalent of maxsockbuf and SL> send/recvbuf are? On the FreeBSD side we set kern.ipc.maxsockbuf=33554432 net.inet.tcp.sendbuf_max=33554432 net.inet.tcp.recvbuf_max=33554432 I don't know what the equivalent for Solaris would be, still doing research on that. SL> Do the switch ports and NIC counters show any drops SL> or errors? No, nothing bad to be seen there. SL> On the FBSD servers you could also run 'netstat -i -w 1' SL> under load to see if drops are occurring locally, or 'systat -vmstat' SL> for resource contention problems. But again, a similar setup here and SL> no such issues have appeared. No errors, no collisions, no drops. I cannot spot any bottlenecks in netstat, either. One thing I just wonder about is that all IRQs (about 700 under load) are routed to only one quque on the ix interface (there seems to be one per core by default). Should the load spread, or is that expected behaviour? cu Gerrit
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150626115602.5ebcc82489bc9f3bf357b236>