Date: Wed, 3 Jul 2013 02:41:23 -0600 From: Scott Long <scott4long@yahoo.com> To: Outback Dingo <outbackdingo@gmail.com> Cc: net@freebsd.org Subject: Re: Terrible ix performance Message-ID: <97FD4CAC-4648-40F8-94CD-4B8D181C4BCC@yahoo.com> In-Reply-To: <CAKYr3zyV74DPLsJRuDoRiYsYdAXs=EoqJ6%2B_k4hJiSnwq5zhUQ@mail.gmail.com> References: <CAKYr3zyV74DPLsJRuDoRiYsYdAXs=EoqJ6%2B_k4hJiSnwq5zhUQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jul 2, 2013, at 10:28 PM, Outback Dingo <outbackdingo@gmail.com> wrote: > Ive got a high end storage server here, iperf shows decent network io > > > the card has a 3 meter twinax cable from cisco connected to it, going > through a fujitsu switch. We have tweaked various networking, and kernel > sysctls, however from a sftp and nfs session i cant get better then 100MBs > from a zpool with 8 mirrored vdevs. We also have an identical box that will > get 1.4Gbs with a 1 meter cisco twinax cables that writes 2.4Gbs compared > to reads only 1.4Gbs... > > does anyone have an idea of what the bottle neck could be?? This is a > shared storage array with dual LSI controllers connected to 32 drives via > an enclosure, local dd and other tests show the zpool performs quite well. > however as soon as we introduce any type of protocol, sftp, samba, nfs > performance plummets. Im quite puzzled and have run out of ideas. so now > curiousity has me........ its loading the ix driver and working but not up > to speed, > it is feasible it should be using the ixgbe driver?? Try turning off interrupt moderation. Add the following to /boot/loader.conf hw.ixgbe.enable_aim=0 NAS workloads are extremely sensitive to latency, and interrupt moderation only adds latency. We tune some other things as well at Netflix and manage to get quite good performance, though with a fairly different workload. Let me know if this does or does not work for you. Thanks, Scott
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?97FD4CAC-4648-40F8-94CD-4B8D181C4BCC>