FreeBSD Mail Archives

Date:      Wed, 12 Aug 2015 01:28:29 +0000 (UTC)
From:      Barney Cordoba <barney_cordoba@yahoo.com>
To:        =?UTF-8?Q?Olivier_Cochard-Labb=C3=A9?= <olivier@cochard.me>,  Maxim Sobolev <sobomax@freebsd.org>
Cc:        FreeBSD Net <freebsd-net@freebsd.org>,  "freebsd@intel.com" <freebsd@intel.com>,  =?UTF-8?Q?Jev_Bj=C3=B6rsell?= <jev@sippysoft.com>
Subject:   Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1
Message-ID:  <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
In-Reply-To: <CA%2Bq%2BTcqq1SGofhwA9PBmkcNrWsiyY32Ha9tfFM8ROwXmmvX%2BrA@mail.gmail.com>
References:  <CA%2Bq%2BTcqq1SGofhwA9PBmkcNrWsiyY32Ha9tfFM8ROwXmmvX%2BrA@mail.gmail.com>

index | next in thread | previous in thread | raw e-mail


Wow, this is really important! if this is a college project, I give you a D. Maybe a D- because it's almost useless information.
You ignore the most important aspect of "performance". Efficiency is arguably the most important aspect of performance. 
1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%. 
Why don't any of you understand this simple thing? Why does spreading equality really matter, unless you are hitting a wall with your cpus? I don't care which cpu processes which packet. If you weren't doing moronic things like binding to a cpu, then you'd never have to care about distribution unless it was extremely unbalanced.
BC 


     On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labbé <olivier@cochard.me> wrote:
   

 On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev <sobomax@freebsd.org> wrote:

> Hi folks,
>
> Hi,



> We've trying to migrate some of our high-PPS systems to a new hardware that
> has four X540-AT2 10G NICs and observed that interrupt time goes through
> roof after we cross around 200K PPS in and 200K out (two ports in LACP).
> The previous hardware was stable up to about 350K PPS in and 350K out. I
> believe the old one was equipped with the I350 and had the identical LACP
> configuration. The new box also has better CPU with more cores (i.e. 24
> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3.
>

200K PPS, and even 350K PPS are very low value indeed.
On a Intel Xeon L5630 (4 cores only) with one X540-AT2

(then 2 10Gigabit ports) I've reached about 1.8Mpps (fastforwarding
enabled) [1].
But my setup didn't use lagg(4): Can you disable lagg configuration and
re-measure your performance without lagg ?

Do you let Intel NIC drivers using 8 queues for port too?
In my use case (forwarding smallest UDP packet size), I obtain better
behaviour by limiting NIC queues to 4 (hw.ix.num_queues or
hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this
with Gigabit Intel[2] or Chelsio NIC [3].

Don't forget to disable TSO and LRO too.

Regards,

Olivier

[1]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs
[2]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4#graph1
[3]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reducing_nic_queues
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"


From owner-freebsd-net@freebsd.org  Wed Aug 12 01:40:17 2015
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2BA399E7B8
 for <freebsd-net@mailman.ysv.freebsd.org>;
 Wed, 12 Aug 2015 01:40:17 +0000 (UTC)
 (envelope-from barney_cordoba@yahoo.com)
Received: from nm1-vm0.bullet.mail.ne1.yahoo.com
 (nm1-vm0.bullet.mail.ne1.yahoo.com [98.138.91.74])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 6C5531CA
 for <freebsd-net@freebsd.org>; Wed, 12 Aug 2015 01:40:17 +0000 (UTC)
 (envelope-from barney_cordoba@yahoo.com)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t39343609; bh=g0kHJWoBbWWQcX7i17mr+ATL4vVJs831Ef4Jrp2PDGM=;
 h�te:From:Reply-To:To:Cc:In-Reply-To:References:Subject:From:Subject;
 b=KzoNi/BIo21zcbWtM9+t9ifIDGyIngCrf8aviCcyhAHyyS1vHq6Gk5bBW1m4cG7oipuWewJLCDsn3XR8gS5WRgsPCmhtUEtWKtfVdnJgLiQfVB3vx7zxYiWT89rkutAlWtbsDbofu2/TyDZi1rTiAdXkT31OauFieL5VbkupuI/uxZMbLPuLp9amcwrGHkjjUg+E7kd2afWkX3BD7uxHNPdq+1+UVqbHt1/OYlsH0X5aHcll1955WTJEy0UYcl4ZMEQctpFx3v9hEOBZURkWiUnubwqapE3/JWqST/Pw3TrU8ubYHrIhrjpl9h1Qg/bNtdkMR+kwne31oX/C3wxC9g=Received: from [98.138.100.115] by nm1.bullet.mail.ne1.yahoo.com with NNFMP;
 12 Aug 2015 01:40:09 -0000
Received: from [98.138.88.238] by tm106.bullet.mail.ne1.yahoo.com with NNFMP;
 12 Aug 2015 01:40:09 -0000
Received: from [127.0.0.1] by omp1038.mail.ne1.yahoo.com with NNFMP;
 12 Aug 2015 01:40:09 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 856165.77852.bm@omp1038.mail.ne1.yahoo.com
X-YMail-OSG: SpJcUVsVM1m_AJRXVKNA2xCb_5ie.ahITTyyihLhZWPsb8eYh1jqQ2f2RYlmQ_H
 XVPV7CPkYUaZi8xBJoLxb3NqnG_siFNKw8NlobR3ZQjLnNtSPlyUnDxV88btcOaNStVKN80ce29p
 w.5AWUsLzxKm1D7AbhYeEsN58xwvt7YE.8NR1WUapscewyleo_iMUsYm41YAPG1ye7tF3vFghpoS
 kOHq4_hNDKXtGrCwsOMsodjwwdn8aBiHKu7M7l6VEsq1XqjW1Lb4bujECLPblAXLa3Zl7z7QDNRj
 jYXsDM6w2PnBVvOZburgdL6rUto1Soem4xULXAUkEf9IukjC2cWvMBpgNOt0ZunZdFedQjt8IAJo
 YXRWGlt1cTDMGGSIrDfh1QJ0anDtCofCNa9J2f4jnRIe3Cy_Xyq8r4PKxKqYBJcRjwf4vpvx_WTa
 c8fQx20NirKd5QkZ6cUiRQtS_2x10dnPrxcHsiQz_Qdhwvo0NLD72sdP2pEbbHClGA1s4ipbgXHy
 .sau.9H4MIDgqGw--
Received: by 98.138.101.170; Wed, 12 Aug 2015 01:40:09 +0000
Date: Wed, 12 Aug 2015 01:40:09 +0000 (UTC)
From: Barney Cordoba <barney_cordoba@yahoo.com>
Reply-To: Barney Cordoba <barney_cordoba@yahoo.com>
To: Barney Cordoba <barney_cordoba@yahoo.com>,
 =?UTF-8?Q?Olivier_Cochard-Labbé?= <olivier@cochard.me>,
 Maxim Sobolev <sobomax@freebsd.org>
Cc: FreeBSD Net <freebsd-net@freebsd.org>,
 "freebsd@intel.com" <freebsd@intel.com>,
 =?UTF-8?Q?Jev_Björsell?= <jev@sippysoft.com>
Message-ID: <477444474.1722346.1439343609122.JavaMail.yahoo@mail.yahoo.com>
In-Reply-To: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
References: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in
 FreeBSD 10.1
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.20
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>;
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Aug 2015 01:40:17 -0000

Also, using a slow-ass cpu like the atom is completely absurd; first, no-one would ever use them. 
You have to test cpu usage under 60% cpu usage, because as you get to higher cpu usage levels the lock contention increases exponentially. You're increasing lock contention by having more queues; so more queues at higher cpu % usage will perform increasingly bad as usage increases.
You'd never run a system at 95% usage (ie totally hammering it) in real world usage, so why would you benchmark at such a high usage? Everything changes as cpu available become scarce. 
"What is the pps at 50% cpu usage" is a better question to ask than the one you're asking.
BC 


     On Tuesday, August 11, 2015 9:29 PM, Barney Cordoba via freebsd-net <freebsd-net@freebsd.org> wrote:
   

 Wow, this is really important! if this is a college project, I give you a D. Maybe a D- because it's almost useless information.
You ignore the most important aspect of "performance". Efficiency is arguably the most important aspect of performance. 
1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%. 
Why don't any of you understand this simple thing? Why does spreading equality really matter, unless you are hitting a wall with your cpus? I don't care which cpu processes which packet. If you weren't doing moronic things like binding to a cpu, then you'd never have to care about distribution unless it was extremely unbalanced.
BC 


    On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labbé <olivier@cochard.me> wrote:
  

 On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev <sobomax@freebsd.org> wrote:

> Hi folks,
>
> Hi,



> We've trying to migrate some of our high-PPS systems to a new hardware that
> has four X540-AT2 10G NICs and observed that interrupt time goes through
> roof after we cross around 200K PPS in and 200K out (two ports in LACP).
> The previous hardware was stable up to about 350K PPS in and 350K out. I
> believe the old one was equipped with the I350 and had the identical LACP
> configuration. The new box also has better CPU with more cores (i.e. 24
> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3.
>

200K PPS, and even 350K PPS are very low value indeed.
On a Intel Xeon L5630 (4 cores only) with one X540-AT2

(then 2 10Gigabit ports) I've reached about 1.8Mpps (fastforwarding
enabled) [1].
But my setup didn't use lagg(4): Can you disable lagg configuration and
re-measure your performance without lagg ?

Do you let Intel NIC drivers using 8 queues for port too?
In my use case (forwarding smallest UDP packet size), I obtain better
behaviour by limiting NIC queues to 4 (hw.ix.num_queues or
hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this
with Gigabit Intel[2] or Chelsio NIC [3].

Don't forget to disable TSO and LRO too.

Regards,

Olivier

[1]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs
[2]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4#graph1
[3]
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reducing_nic_queues
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"

  
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"


From owner-freebsd-net@freebsd.org  Wed Aug 12 01:46:45 2015
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE06499E92A
 for <freebsd-net@mailman.ysv.freebsd.org>;
 Wed, 12 Aug 2015 01:46:45 +0000 (UTC)
 (envelope-from sobomax@sippysoft.com)
Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com
 [209.85.212.181])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 64E90A07
 for <freebsd-net@freebsd.org>; Wed, 12 Aug 2015 01:46:44 +0000 (UTC)
 (envelope-from sobomax@sippysoft.com)
Received: by wicne3 with SMTP id ne3so82163661wic.0
 for <freebsd-net@freebsd.org>; Tue, 11 Aug 2015 18:46:42 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d100.net; s 130820;
 h=x-gm-message-state:mime-version:sender:in-reply-to:references:date
 :message-id:subject:from:to:cc:content-type;
 bh=QnUF8NPF+tTwauLQQi4jIFxu/EM5dzbuE3bv602TNLA=;
 b=IQ8EqxNP84p69cCmQf0TYp9Eu6SIO+xgXrpcKCvkNFOsC7HGCxFapbES6tsclhUlri
 xhCXNAcVKm1i2H5Aa4STLwTMoq3qrt0Gbb3i+E3WgsRjKgUBZyEsr1fnjXKzjaQNF6Ws
 0ALcvBOyGtDxtZRuIR7oGU6RIfPsCuI8auHhCNUPij/Ddl8if0kdRiIi4xtmUh1CrSgl
 gLxaEgTIJYrlZ7uvyxv8L+qpcwaaPXjVHVMc3P6Qur0pT7AJ6KzqlZqQfeIP0U3zY+xX
 fTSk11fZxds49Rozt8wZktf4f9d1kMA+8/CK4kQvXKODTFSWXjft+HRpzjxjTwgr0KRI
 HVqw=X-Gm-Message-State: ALoCoQn6BzD71Yqfk+Zxv5qXKPmBmIo21oSNzpCcMRTwYIeegAxqx08VRJ5W7JqfyDA5jJ+pYf1l
MIME-Version: 1.0
X-Received: by 10.180.78.166 with SMTP id c6mr21150419wix.8.1439344002764;
 Tue, 11 Aug 2015 18:46:42 -0700 (PDT)
Sender: sobomax@sippysoft.com
Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 18:46:42 -0700 (PDT)
In-Reply-To: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
References: <CA+q+Tcqq1SGofhwA9PBmkcNrWsiyY32Ha9tfFM8ROwXmmvX+rA@mail.gmail.com>
 <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com>
Date: Tue, 11 Aug 2015 18:46:42 -0700
X-Google-Sender-Auth: vYWQutNBf8V6QTWySNnYWFiUeYc
Message-ID: <CAH7qZft_MCdGc=2yCj+9-kgZ9Hf6SW1T8NR71ktSnUaP9c6f9g@mail.gmail.com>
Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in
 FreeBSD 10.1
From: Maxim Sobolev <sobomax@FreeBSD.org>
To: Barney Cordoba <barney_cordoba@yahoo.com>
Cc: =?UTF-8?Q?Olivier_Cochard-Labbé?= <olivier@cochard.me>,
 FreeBSD Net <freebsd-net@freebsd.org>, "freebsd@intel.com" <freebsd@intel.com>,
 =?UTF-8?Q?Jev_Björsell?= <jev@sippysoft.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.20
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>;
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Aug 2015 01:46:46 -0000

Thanks Barney for totally useless response and an attempted insult! And
yes, we are hitting CPU limit on 12 core  E5-2620 v2 systems running I350,
so yes, we do know a little bit how to distribute our application at least
with the igb. For some reason this does not work with ixgb and we are
trying to understand why is it so.

http://sobomax.sippysoft.com/ScreenShot388.png

On Tue, Aug 11, 2015 at 6:28 PM, Barney Cordoba <barney_cordoba@yahoo.com>
wrote:

> Wow, this is really important! if this is a college project, I give you a
> D. Maybe a D- because it's almost useless information.
>
> You ignore the most important aspect of "performance". Efficiency is
> arguably the most important aspect of performance.
>
> 1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%.
>
> Why don't any of you understand this simple thing? Why does spreading
> equality really matter, unless you are hitting a wall with your cpus? I
> don't care which cpu processes which packet. If you weren't doing moronic
> things like binding to a cpu, then you'd never have to care about
> distribution unless it was extremely unbalanced.
>
> BC
>
>
>
> On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labbé <
> olivier@cochard.me> wrote:
>
>
> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev <sobomax@freebsd.org>
> wrote:
>
> > Hi folks,
> >
> > Hi,
> 
>
>
> > We've trying to migrate some of our high-PPS systems to a new hardware
> that
> > has four X540-AT2 10G NICs and observed that interrupt time goes through
> > roof after we cross around 200K PPS in and 200K out (two ports in LACP).
> > The previous hardware was stable up to about 350K PPS in and 350K out. I
> > believe the old one was equipped with the I350 and had the identical LACP
> > configuration. The new box also has better CPU with more cores (i.e. 24
> > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3.
> >
>
> 200K PPS, and even 350K PPS are very low value indeed.
> On a Intel Xeon L5630 (4 cores only) with one X540-AT2
>
> (then 2 10Gigabit ports) I've reached about 1.8Mpps (fastforwarding
> enabled) [1].
> But my setup didn't use lagg(4): Can you disable lagg configuration and
> re-measure your performance without lagg ?
>
> Do you let Intel NIC drivers using 8 queues for port too?
> In my use case (forwarding smallest UDP packet size), I obtain better
> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or
> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this
> with Gigabit Intel[2] or Chelsio NIC [3].
>
> Don't forget to disable TSO and LRO too.
>
> Regards,
>
> Olivier
>
> [1]
>
> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs
> [2]
>
> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4#graph1
> [3]
>
> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reducing_nic_queues
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
>
>

help

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1931183673.3234634.1439342909940.JavaMail.yahoo>

Header And Logo

Peripheral Links

Site Navigation

Header And Logo

Peripheral Links

Search

Site Navigation