From owner-freebsd-net@freebsd.org Tue Aug 11 23:15:19 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B4E5799F1C2 for ; Tue, 11 Aug 2015 23:15:19 +0000 (UTC) (envelope-from cochard@gmail.com) Received: from mail-la0-x234.google.com (mail-la0-x234.google.com [IPv6:2a00:1450:4010:c03::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 28BA8F1E; Tue, 11 Aug 2015 23:15:19 +0000 (UTC) (envelope-from cochard@gmail.com) Received: by labd1 with SMTP id d1so216066lab.1; Tue, 11 Aug 2015 16:15:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=sZUgiWWtC6lyqUEYdBBP3TzTyjMNrQVPovKEdV9/DX0=; b=BgaCp7/Ea5REOQ8HJucy9Jye4elQIAAvnIB+Ay4X8kjXHXvdQGCFkvF3Gynxa/5eGG lY3TEedFUOHlRmRgZOsrtyDagDgN/2ffyiA674WoSyKIgPywIDFrtSejQitRv66w8nHI 9hOakWRHGAaFspg3K5NzmmEAaiN5lziAUQO6s3mi3CV/JbOSX6BukbTsRjpKflfRcx+3 xkwP8s2e+t/fyd2GVFZojbUKBdmFmoBk3uF93ifBiKzWzQVA7FpeEPMAeaIxd3VvUwAc y43gO6oUTaYyNFphzjk6xeuiAqjwkHJvm31F+mza+MNqovVP7Qje4cpPnBW9B2V22hfv Rwiw== X-Received: by 10.112.47.73 with SMTP id b9mr29202857lbn.46.1439334917144; Tue, 11 Aug 2015 16:15:17 -0700 (PDT) MIME-Version: 1.0 Sender: cochard@gmail.com Received: by 10.25.157.146 with HTTP; Tue, 11 Aug 2015 16:14:57 -0700 (PDT) In-Reply-To: References: From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Date: Wed, 12 Aug 2015 01:14:57 +0200 X-Google-Sender-Auth: yFh-H7CiJhxDVxV0eRVfIavIpVI Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 To: Maxim Sobolev Cc: FreeBSD Net , freebsd@intel.com, =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 23:15:19 -0000 On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev wrote= : > Hi folks, > > =E2=80=8BHi, =E2=80=8B > We've trying to migrate some of our high-PPS systems to a new hardware th= at > has four X540-AT2 10G NICs and observed that interrupt time goes through > roof after we cross around 200K PPS in and 200K out (two ports in LACP). > The previous hardware was stable up to about 350K PPS in and 350K out. I > believe the old one was equipped with the I350 and had the identical LACP > configuration. The new box also has better CPU with more cores (i.e. 24 > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fast= forwarding enabled) [1]. But my setup didn't use lagg(4): Can you disable lagg configuration and re-measure your performance without lagg ? Do you let Intel NIC drivers using 8 queues for port too? In my use case (forwarding smallest UDP packet size), I obtain better behaviour by limiting NIC queues to 4 (hw.ix.num_queues or hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this with Gigabit Intel[2] or Chelsio NIC [3]. Don't forget to disable TSO and LRO too. =E2=80=8BRegards, Olivier [1] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ib= m_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs [2] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_sup= erserver_5018a-ftn4#graph1 [3] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_= proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reduci= ng_nic_queues