Date: Tue, 17 Jul 2012 02:23:52 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: "Alexander V. Chernikov" <melifaro@freebsd.org> Cc: Doug Barton <dougb@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it>, net@freebsd.org Subject: Re: FreeBSD 10G forwarding performance @Intel Message-ID: <20120716232352.GE2676@deviant.kiev.zoral.com.ua> In-Reply-To: <500452A5.3070501@FreeBSD.org> References: <4FF36438.2030902@FreeBSD.org> <4FF3E2C4.7050701@FreeBSD.org> <4FF3FB14.8020006@FreeBSD.org> <4FF402D1.4000505@FreeBSD.org> <20120704091241.GA99164@onelab2.iet.unipi.it> <4FF412B9.3000406@FreeBSD.org> <20120704154856.GC3680@onelab2.iet.unipi.it> <4FF59955.5090406@FreeBSD.org> <20120706061126.GA65432@onelab2.iet.unipi.it> <500452A5.3070501@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--d8Lz2Tf5e5STOWUP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 16, 2012 at 09:43:01PM +0400, Alexander V. Chernikov wrote: > On 06.07.2012 10:11, Luigi Rizzo wrote: > >On Thu, Jul 05, 2012 at 05:40:37PM +0400, Alexander V. Chernikov wrote: > >>On 04.07.2012 19:48, Luigi Rizzo wrote: > >the thing discussed a few years ago (at least the one i took out of the > >discussion) was that the counter fields in rules should hold the > >index of a per-cpu counter associated to the rule. So CTR_INC(rule->ctr) > >becomes something like pcpu->ipfw_ctrs[rule->ctr]++ > >Once you create a new rule you also grab one free index from ipfw_ctrs[], > >and the same should go for dummynet counters. >=20 > Old kernel from previous letters, same setup: >=20 > net.inet.ip.fw.enable=3D0 > 2.3 MPPS > net.inet.ip.fw.update_counters=3D0 > net.inet.ip.fw.enable=3D1 > 1.93MPPS > net.inet.ip.fw.update_counters=3D1 > 1.74MPPS >=20 > Kernel with ipfw pcpu counters: >=20 > net.inet.ip.fw.enable=3D0 > 2.3 MPPS > net.inet.ip.fw.update_counters=3D0 > net.inet.ip.fw.enable=3D1 > 1.93MPPS > net.inet.ip.fw.update_counters=3D1 > 1.93MPPS >=20 > Counters seems to be working without any (significant) overhead. > (Maybe I'm wrong somewhere?) >=20 > Additionally, I've got (from my previous pcpu attempt) a small patch=20 > permitting ipfw to re-use rule map allocation instead of reallocating=20 > on every rule. This saves a bit of system time: >=20 > loading 20k rules with ipfw binary gives us: > 5.1s system time before and 4.1s system time after. >=20 I do not think that your 'per-cpu' counter are correct. The thread migration or rescheduling causes the fetch or update of the wrong per-cpu structure. This allows parallel updates with undefined consequences. As a lowest thing to do, you need to disable preeemption around counter structure dereference and increment. --d8Lz2Tf5e5STOWUP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlAEoogACgkQC3+MBN1Mb4j+WACfR/QhjNQeVUa/byYwoHT3lsHv WaMAnAkOPkpds/lMkJbshJVTZXn05Op6 =7n6q -----END PGP SIGNATURE----- --d8Lz2Tf5e5STOWUP--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120716232352.GE2676>