From owner-freebsd-net@FreeBSD.ORG Wed Jul 3 08:44:46 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3199918A for ; Wed, 3 Jul 2013 08:44:46 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm1-vm10.bullet.mail.gq1.yahoo.com (nm1-vm10.bullet.mail.gq1.yahoo.com [98.136.218.89]) by mx1.freebsd.org (Postfix) with ESMTP id D45271675 for ; Wed, 3 Jul 2013 08:44:45 +0000 (UTC) Received: from [98.137.12.56] by nm1.bullet.mail.gq1.yahoo.com with NNFMP; 03 Jul 2013 08:41:24 -0000 Received: from [208.71.42.200] by tm1.bullet.mail.gq1.yahoo.com with NNFMP; 03 Jul 2013 08:41:23 -0000 Received: from [127.0.0.1] by smtp211.mail.gq1.yahoo.com with NNFMP; 03 Jul 2013 08:41:23 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1372840883; bh=x9oAhiyf4herWuhktueebQqJnHpXdNqlRNGiXOjmJjA=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To:X-Mailer; b=P6+bkbrXiG452jP3+uWa/wdtrzylWBDuzL6FK348FT4KyvbTgm8w0HxUm46wOG/2wWyQ+Lr8fnExULRlP9eR6bncQUHsdTAbzOXIEc2rGFvfxiozjjo5Hf4MaBT3kOZICgdlwqGh3OZZSBoQp0l0bBiePne+js9d0VPu1jxTqcE= X-Yahoo-Newman-Id: 960422.49157.bm@smtp211.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: xG1m2tsVM1kuwLthGzICaLtNcQ4uSFNW8zPgz80EXp3Qmod .bqlWtU2rnPECjI1Rk2sU7JpatHXfzJictg2zMA4q3iGYYobFUuOr_qPIcIK IPYquRd5Cya5eeFzbEh66W4JYOxC3g_Qbqv1HV0Jp9rdkw9xJvqYJUfSf0G5 Kzkb4SIBgT6FfocYUXQwCNhxQxUPc3Ki1WYMVC.lBFxeOsYXTRnuM_SgTRdc u0ymPm7qjXuw_sdvdEGBb2bYR.9o5n37YVd7AZnXitbbEygtI2r3FAbhV79d 9A.Sqnn5yDfdGi15nWV4gMAGucSwXVMuKH6H0blzle9884iBdTRShQr1mXZD ikZuNEWJ2PQqJZHUnH87KXxaIoZqSa22V0bCans9.9CtBmh57TU_4PiET1qU iDJ1n4gtAA5QANNC350sBF6xasiTA9w51K_uP6ar6H7bP8xc47pmMrUQtXmn nbSHAA7GHeAUItaTrWjHZUP5_7Z0x8K8QO.aaHuLBgr0uiFMlXXkjV3144Wa JLcjofcfKRSQ8QhI0roXaCYs3GFoid9mV7BkUu67hsqcRGxKNrzKy.EEZRQ2 szA-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- X-Rocket-Received: from [10.64.26.20] (scott4long@69.53.237.126 with ) by smtp211.mail.gq1.yahoo.com with SMTP; 03 Jul 2013 01:41:23 -0700 PDT Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: Terrible ix performance From: Scott Long In-Reply-To: Date: Wed, 3 Jul 2013 02:41:23 -0600 Content-Transfer-Encoding: 7bit Message-Id: <97FD4CAC-4648-40F8-94CD-4B8D181C4BCC@yahoo.com> References: To: Outback Dingo X-Mailer: Apple Mail (2.1508) Cc: net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jul 2013 08:44:46 -0000 On Jul 2, 2013, at 10:28 PM, Outback Dingo wrote: > Ive got a high end storage server here, iperf shows decent network io > > > the card has a 3 meter twinax cable from cisco connected to it, going > through a fujitsu switch. We have tweaked various networking, and kernel > sysctls, however from a sftp and nfs session i cant get better then 100MBs > from a zpool with 8 mirrored vdevs. We also have an identical box that will > get 1.4Gbs with a 1 meter cisco twinax cables that writes 2.4Gbs compared > to reads only 1.4Gbs... > > does anyone have an idea of what the bottle neck could be?? This is a > shared storage array with dual LSI controllers connected to 32 drives via > an enclosure, local dd and other tests show the zpool performs quite well. > however as soon as we introduce any type of protocol, sftp, samba, nfs > performance plummets. Im quite puzzled and have run out of ideas. so now > curiousity has me........ its loading the ix driver and working but not up > to speed, > it is feasible it should be using the ixgbe driver?? Try turning off interrupt moderation. Add the following to /boot/loader.conf hw.ixgbe.enable_aim=0 NAS workloads are extremely sensitive to latency, and interrupt moderation only adds latency. We tune some other things as well at Netflix and manage to get quite good performance, though with a fairly different workload. Let me know if this does or does not work for you. Thanks, Scott