From owner-freebsd-net@FreeBSD.ORG Tue Jul 17 10:51:52 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1FA01065674; Tue, 17 Jul 2012 10:51:52 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 7BFFF8FC15; Tue, 17 Jul 2012 10:51:52 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q6HApwL8065231; Tue, 17 Jul 2012 13:51:58 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q6HApjiA022129; Tue, 17 Jul 2012 13:51:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q6HApj6C022128; Tue, 17 Jul 2012 13:51:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 17 Jul 2012 13:51:45 +0300 From: Konstantin Belousov To: "Alexander V. Chernikov" Message-ID: <20120717105145.GH2676@deviant.kiev.zoral.com.ua> References: <4FF3FB14.8020006@FreeBSD.org> <4FF402D1.4000505@FreeBSD.org> <20120704091241.GA99164@onelab2.iet.unipi.it> <4FF412B9.3000406@FreeBSD.org> <20120704154856.GC3680@onelab2.iet.unipi.it> <4FF59955.5090406@FreeBSD.org> <20120706061126.GA65432@onelab2.iet.unipi.it> <500452A5.3070501@FreeBSD.org> <20120716232352.GE2676@deviant.kiev.zoral.com.ua> <50053F74.7040101@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="D6IIOQQv2Iwyp54J" Content-Disposition: inline In-Reply-To: <50053F74.7040101@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Doug Barton , Luigi Rizzo , net@freebsd.org Subject: Re: FreeBSD 10G forwarding performance @Intel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 10:51:53 -0000 --D6IIOQQv2Iwyp54J Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 17, 2012 at 02:33:24PM +0400, Alexander V. Chernikov wrote: > On 17.07.2012 03:23, Konstantin Belousov wrote: > >On Mon, Jul 16, 2012 at 09:43:01PM +0400, Alexander V. Chernikov wrote: > >>On 06.07.2012 10:11, Luigi Rizzo wrote: > >>>On Thu, Jul 05, 2012 at 05:40:37PM +0400, Alexander V. Chernikov wrote: > >>>>On 04.07.2012 19:48, Luigi Rizzo wrote: > >>>the thing discussed a few years ago (at least the one i took out of the > >>>discussion) was that the counter fields in rules should hold the > >>>index of a per-cpu counter associated to the rule. So CTR_INC(rule->ct= r) > >>>becomes something like pcpu->ipfw_ctrs[rule->ctr]++ > >>>Once you create a new rule you also grab one free index from ipfw_ctrs= [], > >>>and the same should go for dummynet counters. > >> > >>Old kernel from previous letters, same setup: > >> > >>net.inet.ip.fw.enable=3D0 > >>2.3 MPPS > >>net.inet.ip.fw.update_counters=3D0 > >>net.inet.ip.fw.enable=3D1 > >>1.93MPPS > >>net.inet.ip.fw.update_counters=3D1 > >>1.74MPPS > >> > >>Kernel with ipfw pcpu counters: > >> > >>net.inet.ip.fw.enable=3D0 > >>2.3 MPPS > >>net.inet.ip.fw.update_counters=3D0 > >>net.inet.ip.fw.enable=3D1 > >>1.93MPPS > >>net.inet.ip.fw.update_counters=3D1 > >>1.93MPPS > >> > >>Counters seems to be working without any (significant) overhead. > >>(Maybe I'm wrong somewhere?) >=20 > >I do not think that your 'per-cpu' counter are correct. The thread > >migration or rescheduling causes the fetch or update of the wrong > It is typical in networking stack to bind interface queues to CPUs. > It is done by netisr code and some network drivers like ixgbe. > (And usually you should do it by hand if you want to achieve maximum=20 > performance). Binding is not enough to guarantee safe update. It elimminates some issues, like changing curcpu, but does not prevents two threads from interwinding the execution of read/incr/write sequences. > >per-cpu structure. This allows parallel updates with undefined > >consequences. > Yes. However, current ipfw counters implementation is not protected by=20 > any lock either, so I'm not sure if we really need to provide _very_=20 > fine-graded counters. > > >=20 > >As a lowest thing to do, you need to disable preeemption around counter > >structure dereference and increment. > Same test with critical_enter() and critical_exit(): >=20 > packets errs idrops bytes packets errs bytes colls > 2412016 0 0 159027422 2413575 0 98996894 0 > 2412603 0 0 159762580 2413196 0 343078548 0 > 2413501 0 0 159094602 2411561 0 159208034 0 > 2413818 0 0 158894876 2412041 0 159579354 0 > 2411331 0 0 159867612 2412699 0 98690770 0 > 2413578 0 0 159565910 2413256 0 220508472 0 > net.inet.ip.fw.update_counters=3D0 > net.inet.ip.fw.enable=3D1 > 2109719 200318 0 155246592 2101593 0 141714254 0 > 2043039 373932 0 159586476 2042970 0 135004566 0 > 2042629 371124 0 159429254 2042308 0 84790780 0 > packets errs idrops bytes packets errs bytes colls > 2033687 0 0 134218324 2034435 0 134524224 0 > 2044290 0 0 134721534 2043947 0 85143014 0 > 2047714 0 0 135502190 2050383 0 85434406 0 > net.inet.ip.fw.update_counters=3D0 > 2008526 0 0 132737890 2009535 0 281671228 0 > 1977217 13550 0 132278298 1968571 0 130585076 0 > 1975823 43522 0 133355986 1973620 0 130319542 0 > 1974159 40715 0 133124772 1977259 0 130552612 0 > 1969801 42073 0 132911194 1969426 0 130451906 0 > 1964142 21919 0 131242870 1966925 0 129959256 0 > 1961748 0 0 129548688 1966168 0 82793086 0 >=20 > So, overhead (~80kpps) is now more observable. Not sure if this is=20 > reasonable. >=20 >=20 > --=20 > WBR, Alexander --D6IIOQQv2Iwyp54J Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlAFQ8EACgkQC3+MBN1Mb4gH5ACg814l+k+mfbFT/LUaM4h461XY 6i8AoPNN7YqNn43Y2dusQOytppKGSmVx =iAk7 -----END PGP SIGNATURE----- --D6IIOQQv2Iwyp54J--