Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 17 Jul 2012 13:51:45 +0300
From:      Konstantin Belousov <kostikbel@gmail.com>
To:        "Alexander V. Chernikov" <melifaro@freebsd.org>
Cc:        Doug Barton <dougb@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it>, net@freebsd.org
Subject:   Re: FreeBSD 10G forwarding performance @Intel
Message-ID:  <20120717105145.GH2676@deviant.kiev.zoral.com.ua>
In-Reply-To: <50053F74.7040101@FreeBSD.org>
References:  <4FF3FB14.8020006@FreeBSD.org> <4FF402D1.4000505@FreeBSD.org> <20120704091241.GA99164@onelab2.iet.unipi.it> <4FF412B9.3000406@FreeBSD.org> <20120704154856.GC3680@onelab2.iet.unipi.it> <4FF59955.5090406@FreeBSD.org> <20120706061126.GA65432@onelab2.iet.unipi.it> <500452A5.3070501@FreeBSD.org> <20120716232352.GE2676@deviant.kiev.zoral.com.ua> <50053F74.7040101@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help

--D6IIOQQv2Iwyp54J
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Jul 17, 2012 at 02:33:24PM +0400, Alexander V. Chernikov wrote:
> On 17.07.2012 03:23, Konstantin Belousov wrote:
> >On Mon, Jul 16, 2012 at 09:43:01PM +0400, Alexander V. Chernikov wrote:
> >>On 06.07.2012 10:11, Luigi Rizzo wrote:
> >>>On Thu, Jul 05, 2012 at 05:40:37PM +0400, Alexander V. Chernikov wrote:
> >>>>On 04.07.2012 19:48, Luigi Rizzo wrote:
> >>>the thing discussed a few years ago (at least the one i took out of the
> >>>discussion) was that the counter fields in rules should hold the
> >>>index of a per-cpu counter associated to the rule. So CTR_INC(rule->ct=
r)
> >>>becomes something like pcpu->ipfw_ctrs[rule->ctr]++
> >>>Once you create a new rule you also grab one free index from ipfw_ctrs=
[],
> >>>and the same should go for dummynet counters.
> >>
> >>Old kernel from previous letters, same setup:
> >>
> >>net.inet.ip.fw.enable=3D0
> >>2.3 MPPS
> >>net.inet.ip.fw.update_counters=3D0
> >>net.inet.ip.fw.enable=3D1
> >>1.93MPPS
> >>net.inet.ip.fw.update_counters=3D1
> >>1.74MPPS
> >>
> >>Kernel with ipfw pcpu counters:
> >>
> >>net.inet.ip.fw.enable=3D0
> >>2.3 MPPS
> >>net.inet.ip.fw.update_counters=3D0
> >>net.inet.ip.fw.enable=3D1
> >>1.93MPPS
> >>net.inet.ip.fw.update_counters=3D1
> >>1.93MPPS
> >>
> >>Counters seems to be working without any (significant) overhead.
> >>(Maybe I'm wrong somewhere?)
>=20
> >I do not think that your 'per-cpu' counter are correct. The thread
> >migration or rescheduling causes the fetch or update of the wrong
> It is typical in networking stack to bind interface queues to CPUs.
> It is done by netisr code and some network drivers like ixgbe.
> (And usually you should do it by hand if you want to achieve maximum=20
> performance).
Binding is not enough to guarantee safe update. It elimminates some issues,
like changing curcpu, but does not prevents two threads from interwinding
the execution of read/incr/write sequences.

> >per-cpu structure. This allows parallel updates with undefined
> >consequences.
> Yes. However, current ipfw counters implementation is not protected by=20
> any lock either, so I'm not sure if we really need to provide _very_=20
> fine-graded counters.
> >
>=20
> >As a lowest thing to do, you need to disable preeemption around counter
> >structure dereference and increment.
> Same test with critical_enter() and critical_exit():
>=20
>    packets  errs idrops      bytes    packets  errs      bytes colls
>    2412016     0     0  159027422    2413575     0   98996894     0
>    2412603     0     0  159762580    2413196     0  343078548     0
>    2413501     0     0  159094602    2411561     0  159208034     0
>    2413818     0     0  158894876    2412041     0  159579354     0
>    2411331     0     0  159867612    2412699     0   98690770     0
>    2413578     0     0  159565910    2413256     0  220508472     0
> net.inet.ip.fw.update_counters=3D0
> net.inet.ip.fw.enable=3D1
>    2109719 200318     0  155246592    2101593     0  141714254     0
>    2043039 373932     0  159586476    2042970     0  135004566     0
>    2042629 371124     0  159429254    2042308     0   84790780     0
>    packets  errs idrops      bytes    packets  errs      bytes colls
>    2033687     0     0  134218324    2034435     0  134524224     0
>    2044290     0     0  134721534    2043947     0   85143014     0
>    2047714     0     0  135502190    2050383     0   85434406     0
> net.inet.ip.fw.update_counters=3D0
>    2008526     0     0  132737890    2009535     0  281671228     0
>    1977217 13550     0  132278298    1968571     0  130585076     0
>    1975823 43522     0  133355986    1973620     0  130319542     0
>    1974159 40715     0  133124772    1977259     0  130552612     0
>    1969801 42073     0  132911194    1969426     0  130451906     0
>    1964142 21919     0  131242870    1966925     0  129959256     0
>    1961748     0     0  129548688    1966168     0   82793086     0
>=20
> So, overhead (~80kpps) is now more observable. Not sure if this is=20
> reasonable.
>=20
>=20
> --=20
> WBR, Alexander

--D6IIOQQv2Iwyp54J
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)

iEYEARECAAYFAlAFQ8EACgkQC3+MBN1Mb4gH5ACg814l+k+mfbFT/LUaM4h461XY
6i8AoPNN7YqNn43Y2dusQOytppKGSmVx
=iAk7
-----END PGP SIGNATURE-----

--D6IIOQQv2Iwyp54J--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120717105145.GH2676>