Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 15 Feb 2006 22:01:06 +0100
From:      Andre Oppermann <andre@freebsd.org>
To:        Luigi Rizzo <rizzo@icir.org>
Cc:        Marcos Bedinelli <bedinelli@madhaus.cns.utoronto.ca>, Max Laier <max@love2party.net>, Julian Elischer <julian@elischer.org>, freebsd-net@freebsd.org
Subject:   Re: Network performance in a dual CPU system
Message-ID:  <43F39692.7A3228BA@freebsd.org>
References:  <7bb8f24157080b6aaacb897a99259df9@madhaus.cns.utoronto.ca> <711b7ec873f31bc5be50ce477313fac3@madhaus.cns.utoronto.ca> <200602110002.21275.max@love2party.net> <43F38CF5.71C326C1@freebsd.org> <20060215123043.A29559@xorpc.icir.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Luigi Rizzo wrote:
> 
> On Wed, Feb 15, 2006 at 09:20:05PM +0100, Andre Oppermann wrote:
> ...
> > >From my profiling with the Agilent tester there seem to be two areas where
> > the packet filters (ipfw in my test case) burn a lot of CPU per packet.
> > That is a) setup of lots of packet variables unconditionally at the entry
> > of ip_fw_chk() no matter whether they get looked at later or not, and b)
> > the switch() going through all the packet inspection options is for some
> > reason not optimized by the compiler and burns even more CPU.  Some sort
> > of JIT (as in the new bpf code) which replaces the case testing and jumps
> > directly to the proper place in the switch statement would go a long way
> > of making it way more performant.
> 
> i was expecting some overhead in the initial setting of
> variables but the cost of the switch() surprises me a bit.
> did you look at the assembly code produced, or otherwise
> could you explain a bit more how you think the switch
> affects performance ?
> Maybe one could make it cheaper through an indirect function call ?
> (in the end, instructions are already indexes for a jump table).

I didn't look at the assembler code as I can't do assembler.

In my testing (on UP) the peak forwarding rate on this particular hardware
with fastforwarding enabled dropped from 580kpps to 476kpps (ipfw allow all)
to 357kpps (30 non-matching rules on IP address).

The number of CPU instructions and branches per packet is as follows:

			maxkpps	instr.	branch	mispred	dcache	icfetch	icmiss
fastfwd			580	2238	300	 3.8	1429	 812	0.06
fastfwd+ipfw		476	2573	329	17.2	1721	1005	4.31
fastfwd+ipfw30		357	3493	508	15.2	2129	1500	3.35

The setup of the packet variables only happens once per packet. The overhead
thus must come from the micro-op evaluation.

-- 
Andre



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43F39692.7A3228BA>