From owner-freebsd-hackers@FreeBSD.ORG Tue Jul 3 21:08:52 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0390106564A; Tue, 3 Jul 2012 21:08:52 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 97C338FC0C; Tue, 3 Jul 2012 21:08:52 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 2AFCB73027; Tue, 3 Jul 2012 23:28:16 +0200 (CEST) Date: Tue, 3 Jul 2012 23:28:16 +0200 From: Luigi Rizzo To: "Alexander V. Chernikov" Message-ID: <20120703212816.GA92445@onelab2.iet.unipi.it> References: <4FF319A2.6070905@FreeBSD.org> <20120703165506.GA90114@onelab2.iet.unipi.it> <4FF32DE2.2010606@FreeBSD.org> <20120703202757.GA90741@onelab2.iet.unipi.it> <4FF356BC.2060306@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4FF356BC.2060306@FreeBSD.org> User-Agent: Mutt/1.4.2.3i Cc: hackers@freebsd.org, performance@freebsd.org, net@freebsd.org Subject: Re: FreeBSD 10G forwarding performance @Intel X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jul 2012 21:08:53 -0000 On Wed, Jul 04, 2012 at 12:31:56AM +0400, Alexander V. Chernikov wrote: > On 04.07.2012 00:27, Luigi Rizzo wrote: > >On Tue, Jul 03, 2012 at 09:37:38PM +0400, Alexander V. Chernikov wrote: > >... > >>Thanks, another good point. I forgot to merge this option from andre's > >>patch. > >> > >>Another 30-40-50kpps to win. > > > >not much gain though. > >What about the other IPSTAT_INC counters ? > Well, we should then remove all such counters (total, forwarded) and > per-interface statistics (at least for forwarded packets). I am not saying to remove them for good, but at least have a try at what we can hope to save by implementing them on a per-cpu basis. There is a chance that one will not see big gains util the majority of such shared counters are fixed (there are probably 3-4 at least on the non-error path for forwarded packets), plus the per-interface ones that are not even wrapped in macros (see if_ethersubr.c) > >I think the IPSTAT_INC macros were introduced (by rwatson ?) > >following a discussion on how to make the counters per-cpu > >and avoid the contention on cache lines. > >But they are still implemented as a single instance, > >and neither volatile nor atomic, so it is not even clear > >that they can give reliable results, let alone the fact > >that you are likely to get some cache misses. > > > >the relevant macro is in ip_var.h. > Hm. This seems to be just per-vnet structure instance. yes but essentially they are still shared by all threads within a vnet (besides you probably ran your tests in the main instance) > We've got some more real DPCPU stuff (sys/pcpu.h && kern/subr_pcpu.c) > which can be used for global ipstat structure, however since it is > allocated from single area without possibility to free we can't use it > for per-interface counters. yes, those should be moved to a private, dynamically allocated region of the ifnet (the number of CPUs is known at driver init time, i hope). But again for a quick test disabling the if_{i|o}{bytesC|packets} should do the job, if you can count the received rate by some other means. > I'll try to run tests without any possibly contested counters and report > the results on Thursday. great, that would be really useful info. cheers luigi