From owner-freebsd-stable@freebsd.org Mon Aug 24 23:13:58 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 703C09C2BE9; Mon, 24 Aug 2015 23:13:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22f.google.com (mail-ig0-x22f.google.com [IPv6:2607:f8b0:4001:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3824C1B94; Mon, 24 Aug 2015 23:13:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igui7 with SMTP id i7so67696272igu.0; Mon, 24 Aug 2015 16:13:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=KGAF2xSeu2wUg7w3O4j+hiI26ZbFRHAN7BHuGt0n28A=; b=m9KreHrgrxFM9sPJ288MOlnrxkwTmNQ0Zc0TDWMSnUOVpPYosuX4T7AGlJx8y7T8Lm L8nvJhi+dgYdXmHmdGQRWAzoe9T0vF2bwnRdWw2paBCtOt+eR4MQLQh8l8HiQJIyU0r3 Mrvv+Omji2L8+JaplJZRinBp6PB9jVGyx1nAdV019xxsN35QR/NuVIm9Jni+YyGsrWYk yDnhG32k2ioIDT0cqnNk99mkkW8BXU2jbalfuQdbbOpJ5YyXqNHEXILfYbGc19XNIqOH +OKQHveBTjs7PkRsaSotadYF766MgFD0MflfiSQA3VO9/3Ddn9AXY4GxqUbW/etX5ehI admA== MIME-Version: 1.0 X-Received: by 10.50.128.169 with SMTP id np9mr17921927igb.37.1440458037693; Mon, 24 Aug 2015 16:13:57 -0700 (PDT) Received: by 10.36.28.208 with HTTP; Mon, 24 Aug 2015 16:13:57 -0700 (PDT) In-Reply-To: <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> <55DAC623.60006@selasky.org> <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> Date: Mon, 24 Aug 2015 16:13:57 -0700 Message-ID: Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance From: Adrian Chadd To: Rick Macklem Cc: Daniel Braniss , Hans Petter Selasky , Yong-Hyeon Pyun , FreeBSD stable , FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 23:13:58 -0000 Hi, Some hand-waving suggestions: * if you're running something before 10.2, please disable IXGBE_FDIR in sys/conf/options and sys/modules/ixgbe/Makefile . It's buggy and it caused a lot of issues. * It sounds like some extra latency is happening, so I'd fiddle around with interrupt settings. By default it does something called adaptive interrupt moderation and it may be getting in the way of what you're trying to do. There's a way to disable AIM in /boot/loader.conf and manually set the interrupt rate. * As others have said, TSO has been a bit of a problem - hps has been working on solidifying the TSO configuration side of things so NICs advertise to the stack what their maximum offload capability is so things like NFS and TCP don't exceed the segment count. I don't know if it's tunable without hacking the driver, but maybe hack the driver to reduce the count a little to make sure you're not overflowing things and causing it to fall back to a slower path (where it copies all the mbufs into a single larger one to send to the NIC.) * Disable software LRO and see if it helps. Since you're doing lots of little non-streaming operations, it may actually be hindering. HTH, -adrian