From owner-freebsd-hackers@FreeBSD.ORG Tue Jul 3 17:38:49 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [69.147.83.53]) by hub.freebsd.org (Postfix) with ESMTP id 0E168106572A; Tue, 3 Jul 2012 17:38:48 +0000 (UTC) (envelope-from melifaro@FreeBSD.org) Received: from dhcp170-36-red.yandex.net (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx2.freebsd.org (Postfix) with ESMTP id AD66314E588; Tue, 3 Jul 2012 17:38:46 +0000 (UTC) Message-ID: <4FF32DE2.2010606@FreeBSD.org> Date: Tue, 03 Jul 2012 21:37:38 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:12.0) Gecko/20120511 Thunderbird/12.0.1 MIME-Version: 1.0 To: Luigi Rizzo References: <4FF319A2.6070905@FreeBSD.org> <20120703165506.GA90114@onelab2.iet.unipi.it> In-Reply-To: <20120703165506.GA90114@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: hackers@freebsd.org, performance@freebsd.org, net@freebsd.org Subject: Re: FreeBSD 10G forwarding performance @Intel X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jul 2012 17:38:49 -0000 On 03.07.2012 20:55, Luigi Rizzo wrote: > On Tue, Jul 03, 2012 at 08:11:14PM +0400, Alexander V. Chernikov wrote: >> Hello list! >> >> I'm quite stuck with bad forwarding performance on many FreeBSD boxes >> doing firewalling. > ... >> In most cases system can forward no more than 700 (or 1400) kpps which >> is quite a bad number (Linux does, say, 5MPPs on nearly the same hardware). > > among the many interesting tests you have run, i am curious > if you have tried to remove the update of the counters on route > entries. They might be another severe contention point. 21:47 [0] m@test15 netstat -I ix0 -w 1 input (ix0) output packets errs idrops bytes packets errs bytes colls 1785514 52785 0 121318340 1784650 0 117874854 0 1773126 52437 0 120701470 1772977 0 117584736 0 1781948 52154 0 121060126 1778271 0 75029554 0 1786169 52982 0 121451160 1787312 0 160967392 0 21:47 [0] test15# sysctl net.rt_count=0 net.rt_count: 1 -> 0 1814465 22546 0 121302076 1814291 0 76860092 0 1817769 14272 0 120984922 1816254 0 163643534 0 1815311 13113 0 120831970 1815340 0 120159118 0 1814059 13698 0 120799132 1813738 0 120172092 0 1818030 13513 0 120960140 1814578 0 120332662 0 1814169 14351 0 120836182 1814003 0 120164310 0 Thanks, another good point. I forgot to merge this option from andre's patch. Another 30-40-50kpps to win. +u_int rt_count = 1; +SYSCTL_INT(_net, OID_AUTO, rt_count, CTLFLAG_RW, &rt_count, 1, ""); @@ -601,17 +625,20 @@ passout: if (error != 0) IPSTAT_INC(ips_odropped); else { - ro.ro_rt->rt_rmx.rmx_pksent++; + if (rt_count) + ro.ro_rt->rt_rmx.rmx_pksent++; IPSTAT_INC(ips_forward); IPSTAT_INC(ips_fastforward); > > cheers > luigi > -- WBR, Alexander