From owner-freebsd-net@FreeBSD.ORG Wed Jul 3 13:39:05 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0AD794CF for ; Wed, 3 Jul 2013 13:39:05 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id 92E781CFC for ; Wed, 3 Jul 2013 13:39:04 +0000 (UTC) Received: from lstewart1.loshell.room52.net (ppp59-167-184-191.static.internode.on.net [59.167.184.191]) by lauren.room52.net (Postfix) with ESMTPSA id E44627E81E; Wed, 3 Jul 2013 23:39:02 +1000 (EST) Message-ID: <51D42976.9020206@freebsd.org> Date: Wed, 03 Jul 2013 23:39:02 +1000 From: Lawrence Stewart User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120613 Thunderbird/13.0 MIME-Version: 1.0 To: Outback Dingo Subject: Re: Terrible ix performance References: <51D3E5BC.1000604@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lauren.room52.net Cc: net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jul 2013 13:39:05 -0000 On 07/03/13 22:58, Outback Dingo wrote: > On Wed, Jul 3, 2013 at 4:50 AM, Lawrence Stewart > wrote: > > On 07/03/13 14:28, Outback Dingo wrote: > > Ive got a high end storage server here, iperf shows decent network io > > > > iperf -i 10 -t 20 -c 10.0.96.1 -w 2.5M -l 2.5M > > ------------------------------------------------------------ > > Client connecting to 10.0.96.1, TCP port 5001 > > TCP window size: 2.50 MByte (WARNING: requested 2.50 MByte) > > ------------------------------------------------------------ > > [ 3] local 10.0.96.2 port 34753 connected with 10.0.96.1 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 9.78 GBytes 8.40 Gbits/sec > > [ 3] 10.0-20.0 sec 8.95 GBytes 7.69 Gbits/sec > > [ 3] 0.0-20.0 sec 18.7 GBytes 8.05 Gbits/sec > > Given that iperf exercises the ixgbe driver (ix), network path and TCP, > I would suggest that your subject is rather misleading ;) > > > the card has a 3 meter twinax cable from cisco connected to it, going > > through a fujitsu switch. We have tweaked various networking, and > kernel > > sysctls, however from a sftp and nfs session i cant get better > then 100MBs > > from a zpool with 8 mirrored vdevs. We also have an identical box > that will > > get 1.4Gbs with a 1 meter cisco twinax cables that writes 2.4Gbs > compared > > to reads only 1.4Gbs... > > I take it the RTT between both hosts is very low i.e. sub 1ms? An answer to the above question would be useful. > > does anyone have an idea of what the bottle neck could be?? This is a > > shared storage array with dual LSI controllers connected to 32 > drives via > > an enclosure, local dd and other tests show the zpool performs > quite well. > > however as soon as we introduce any type of protocol, sftp, samba, nfs > > performance plummets. Im quite puzzled and have run out of ideas. > so now > > curiousity has me........ its loading the ix driver and working > but not up > > to speed, > > ssh (and sftp by extension) aren't often tuned for high speed operation. > Are you running with the HPN patch applied or a new enough FreeBSD that > has the patch included? Samba and NFS are both likely to need tuning for > multi-Gbps operation. > > > Running 9-STABLE as of 3 days ago, what are you referring to s i can > validate i dont need to apply it Ok so your SSH should have the HPN patch. > as for tuning for NFS/SAMBA sambas configured with AIO, and sendfile, > and there so much information > on tuninig these things that its a bit hard to decipher whats right and > not right Before looking at tuning, I'd suggest testing with a protocol that involves the disk but isn't as heavy weight as SSH/NFS/CIFS. FTP is the obvious choice. Set up an inetd-based FTP instance, serve a file large enough that it will take ~60s to transfer to the client and report back what data rates you get from 5 back-to-back transfer trials. Cheers, Lawrence