Date: Tue, 17 Jul 2012 13:51:45 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: "Alexander V. Chernikov" <melifaro@freebsd.org> Cc: Doug Barton <dougb@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it>, net@freebsd.org Subject: Re: FreeBSD 10G forwarding performance @Intel Message-ID: <20120717105145.GH2676@deviant.kiev.zoral.com.ua> In-Reply-To: <50053F74.7040101@FreeBSD.org> References: <4FF3FB14.8020006@FreeBSD.org> <4FF402D1.4000505@FreeBSD.org> <20120704091241.GA99164@onelab2.iet.unipi.it> <4FF412B9.3000406@FreeBSD.org> <20120704154856.GC3680@onelab2.iet.unipi.it> <4FF59955.5090406@FreeBSD.org> <20120706061126.GA65432@onelab2.iet.unipi.it> <500452A5.3070501@FreeBSD.org> <20120716232352.GE2676@deviant.kiev.zoral.com.ua> <50053F74.7040101@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--D6IIOQQv2Iwyp54J Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 17, 2012 at 02:33:24PM +0400, Alexander V. Chernikov wrote: > On 17.07.2012 03:23, Konstantin Belousov wrote: > >On Mon, Jul 16, 2012 at 09:43:01PM +0400, Alexander V. Chernikov wrote: > >>On 06.07.2012 10:11, Luigi Rizzo wrote: > >>>On Thu, Jul 05, 2012 at 05:40:37PM +0400, Alexander V. Chernikov wrote: > >>>>On 04.07.2012 19:48, Luigi Rizzo wrote: > >>>the thing discussed a few years ago (at least the one i took out of the > >>>discussion) was that the counter fields in rules should hold the > >>>index of a per-cpu counter associated to the rule. So CTR_INC(rule->ct= r) > >>>becomes something like pcpu->ipfw_ctrs[rule->ctr]++ > >>>Once you create a new rule you also grab one free index from ipfw_ctrs= [], > >>>and the same should go for dummynet counters. > >> > >>Old kernel from previous letters, same setup: > >> > >>net.inet.ip.fw.enable=3D0 > >>2.3 MPPS > >>net.inet.ip.fw.update_counters=3D0 > >>net.inet.ip.fw.enable=3D1 > >>1.93MPPS > >>net.inet.ip.fw.update_counters=3D1 > >>1.74MPPS > >> > >>Kernel with ipfw pcpu counters: > >> > >>net.inet.ip.fw.enable=3D0 > >>2.3 MPPS > >>net.inet.ip.fw.update_counters=3D0 > >>net.inet.ip.fw.enable=3D1 > >>1.93MPPS > >>net.inet.ip.fw.update_counters=3D1 > >>1.93MPPS > >> > >>Counters seems to be working without any (significant) overhead. > >>(Maybe I'm wrong somewhere?) >=20 > >I do not think that your 'per-cpu' counter are correct. The thread > >migration or rescheduling causes the fetch or update of the wrong > It is typical in networking stack to bind interface queues to CPUs. > It is done by netisr code and some network drivers like ixgbe. > (And usually you should do it by hand if you want to achieve maximum=20 > performance). Binding is not enough to guarantee safe update. It elimminates some issues, like changing curcpu, but does not prevents two threads from interwinding the execution of read/incr/write sequences. > >per-cpu structure. This allows parallel updates with undefined > >consequences. > Yes. However, current ipfw counters implementation is not protected by=20 > any lock either, so I'm not sure if we really need to provide _very_=20 > fine-graded counters. > > >=20 > >As a lowest thing to do, you need to disable preeemption around counter > >structure dereference and increment. > Same test with critical_enter() and critical_exit(): >=20 > packets errs idrops bytes packets errs bytes colls > 2412016 0 0 159027422 2413575 0 98996894 0 > 2412603 0 0 159762580 2413196 0 343078548 0 > 2413501 0 0 159094602 2411561 0 159208034 0 > 2413818 0 0 158894876 2412041 0 159579354 0 > 2411331 0 0 159867612 2412699 0 98690770 0 > 2413578 0 0 159565910 2413256 0 220508472 0 > net.inet.ip.fw.update_counters=3D0 > net.inet.ip.fw.enable=3D1 > 2109719 200318 0 155246592 2101593 0 141714254 0 > 2043039 373932 0 159586476 2042970 0 135004566 0 > 2042629 371124 0 159429254 2042308 0 84790780 0 > packets errs idrops bytes packets errs bytes colls > 2033687 0 0 134218324 2034435 0 134524224 0 > 2044290 0 0 134721534 2043947 0 85143014 0 > 2047714 0 0 135502190 2050383 0 85434406 0 > net.inet.ip.fw.update_counters=3D0 > 2008526 0 0 132737890 2009535 0 281671228 0 > 1977217 13550 0 132278298 1968571 0 130585076 0 > 1975823 43522 0 133355986 1973620 0 130319542 0 > 1974159 40715 0 133124772 1977259 0 130552612 0 > 1969801 42073 0 132911194 1969426 0 130451906 0 > 1964142 21919 0 131242870 1966925 0 129959256 0 > 1961748 0 0 129548688 1966168 0 82793086 0 >=20 > So, overhead (~80kpps) is now more observable. Not sure if this is=20 > reasonable. >=20 >=20 > --=20 > WBR, Alexander --D6IIOQQv2Iwyp54J Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlAFQ8EACgkQC3+MBN1Mb4gH5ACg814l+k+mfbFT/LUaM4h461XY 6i8AoPNN7YqNn43Y2dusQOytppKGSmVx =iAk7 -----END PGP SIGNATURE----- --D6IIOQQv2Iwyp54J--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120717105145.GH2676>