From owner-freebsd-net@FreeBSD.ORG Mon Jul 16 23:23:56 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CC060106566B; Mon, 16 Jul 2012 23:23:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 4DD228FC08; Mon, 16 Jul 2012 23:23:56 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q6GNO5cF093778; Tue, 17 Jul 2012 02:24:05 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q6GNNqVh074202; Tue, 17 Jul 2012 02:23:52 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q6GNNqYS074201; Tue, 17 Jul 2012 02:23:52 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 17 Jul 2012 02:23:52 +0300 From: Konstantin Belousov To: "Alexander V. Chernikov" Message-ID: <20120716232352.GE2676@deviant.kiev.zoral.com.ua> References: <4FF36438.2030902@FreeBSD.org> <4FF3E2C4.7050701@FreeBSD.org> <4FF3FB14.8020006@FreeBSD.org> <4FF402D1.4000505@FreeBSD.org> <20120704091241.GA99164@onelab2.iet.unipi.it> <4FF412B9.3000406@FreeBSD.org> <20120704154856.GC3680@onelab2.iet.unipi.it> <4FF59955.5090406@FreeBSD.org> <20120706061126.GA65432@onelab2.iet.unipi.it> <500452A5.3070501@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="d8Lz2Tf5e5STOWUP" Content-Disposition: inline In-Reply-To: <500452A5.3070501@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Doug Barton , Luigi Rizzo , net@freebsd.org Subject: Re: FreeBSD 10G forwarding performance @Intel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2012 23:23:57 -0000 --d8Lz2Tf5e5STOWUP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 16, 2012 at 09:43:01PM +0400, Alexander V. Chernikov wrote: > On 06.07.2012 10:11, Luigi Rizzo wrote: > >On Thu, Jul 05, 2012 at 05:40:37PM +0400, Alexander V. Chernikov wrote: > >>On 04.07.2012 19:48, Luigi Rizzo wrote: > >the thing discussed a few years ago (at least the one i took out of the > >discussion) was that the counter fields in rules should hold the > >index of a per-cpu counter associated to the rule. So CTR_INC(rule->ctr) > >becomes something like pcpu->ipfw_ctrs[rule->ctr]++ > >Once you create a new rule you also grab one free index from ipfw_ctrs[], > >and the same should go for dummynet counters. >=20 > Old kernel from previous letters, same setup: >=20 > net.inet.ip.fw.enable=3D0 > 2.3 MPPS > net.inet.ip.fw.update_counters=3D0 > net.inet.ip.fw.enable=3D1 > 1.93MPPS > net.inet.ip.fw.update_counters=3D1 > 1.74MPPS >=20 > Kernel with ipfw pcpu counters: >=20 > net.inet.ip.fw.enable=3D0 > 2.3 MPPS > net.inet.ip.fw.update_counters=3D0 > net.inet.ip.fw.enable=3D1 > 1.93MPPS > net.inet.ip.fw.update_counters=3D1 > 1.93MPPS >=20 > Counters seems to be working without any (significant) overhead. > (Maybe I'm wrong somewhere?) >=20 > Additionally, I've got (from my previous pcpu attempt) a small patch=20 > permitting ipfw to re-use rule map allocation instead of reallocating=20 > on every rule. This saves a bit of system time: >=20 > loading 20k rules with ipfw binary gives us: > 5.1s system time before and 4.1s system time after. >=20 I do not think that your 'per-cpu' counter are correct. The thread migration or rescheduling causes the fetch or update of the wrong per-cpu structure. This allows parallel updates with undefined consequences. As a lowest thing to do, you need to disable preeemption around counter structure dereference and increment. --d8Lz2Tf5e5STOWUP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlAEoogACgkQC3+MBN1Mb4j+WACfR/QhjNQeVUa/byYwoHT3lsHv WaMAnAkOPkpds/lMkJbshJVTZXn05Op6 =7n6q -----END PGP SIGNATURE----- --d8Lz2Tf5e5STOWUP--