Date: Mon, 24 Aug 2015 16:13:57 -0700 From: Adrian Chadd <adrian.chadd@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Daniel Braniss <danny@cs.huji.ac.il>, Hans Petter Selasky <hps@selasky.org>, Yong-Hyeon Pyun <pyunyh@gmail.com>, FreeBSD stable <freebsd-stable@freebsd.org>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <CAJ-VmokJxtUJcHBVKCdKrpr1dvXFbqHg3SOpx-Wc=b34eFuG0Q@mail.gmail.com> In-Reply-To: <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> <55DAC623.60006@selasky.org> <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, Some hand-waving suggestions: * if you're running something before 10.2, please disable IXGBE_FDIR in sys/conf/options and sys/modules/ixgbe/Makefile . It's buggy and it caused a lot of issues. * It sounds like some extra latency is happening, so I'd fiddle around with interrupt settings. By default it does something called adaptive interrupt moderation and it may be getting in the way of what you're trying to do. There's a way to disable AIM in /boot/loader.conf and manually set the interrupt rate. * As others have said, TSO has been a bit of a problem - hps has been working on solidifying the TSO configuration side of things so NICs advertise to the stack what their maximum offload capability is so things like NFS and TCP don't exceed the segment count. I don't know if it's tunable without hacking the driver, but maybe hack the driver to reduce the count a little to make sure you're not overflowing things and causing it to fall back to a slower path (where it copies all the mbufs into a single larger one to send to the NIC.) * Disable software LRO and see if it helps. Since you're doing lots of little non-streaming operations, it may actually be hindering. HTH, -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmokJxtUJcHBVKCdKrpr1dvXFbqHg3SOpx-Wc=b34eFuG0Q>