From owner-freebsd-net@FreeBSD.ORG  Wed Mar 15 10:48:36 2006
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
X-Original-To: freebsd-net@freebsd.org
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 727FE16A401
	for <freebsd-net@freebsd.org>; Wed, 15 Mar 2006 10:48:36 +0000 (UTC)
	(envelope-from thompsa@freebsd.org)
Received: from dbmail-mx4.orcon.co.nz (loadbalancer-VIP.orcon.net.nz
	[219.88.242.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id BFF8C43D66
	for <freebsd-net@freebsd.org>; Wed, 15 Mar 2006 10:48:24 +0000 (GMT)
	(envelope-from thompsa@freebsd.org)
Received: from heff.fud.org.nz (60-234-149-201.bitstream.orcon.net.nz
	[60.234.149.201])
	by dbmail-mx4.orcon.co.nz (8.13.5/8.13.5/Debian-3) with SMTP id
	k2FAm0oR010754; Wed, 15 Mar 2006 23:48:01 +1300
Received: by heff.fud.org.nz (Postfix, from userid 1001)
	id 70A481CCC2; Wed, 15 Mar 2006 23:48:07 +1300 (NZDT)
Date: Wed, 15 Mar 2006 23:48:07 +1300
From: Andrew Thompson <thompsa@freebsd.org>
To: Chih-Chang Hsieh <cch@kmu.edu.tw>
Message-ID: <20060315104807.GA20361@heff.fud.org.nz>
References: <4417E97D.7050202@kmu.edu.tw>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4417E97D.7050202@kmu.edu.tw>
User-Agent: Mutt/1.5.11
X-Virus-Scanned: ClamAV 0.88/1333/Wed Mar 15 19:57:53 2006 on
	dbmail-mx4.orcon.co.nz
X-Virus-Status: Clean
Cc: freebsd-net@freebsd.org
Subject: Re: if_bridge + polling get lower througphts
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 15 Mar 2006 10:48:36 -0000

On Wed, Mar 15, 2006 at 06:16:29PM +0800, Chih-Chang Hsieh wrote:
> We have a FreeBSD 6.1-PRELEASE box.
> 
> It runs with 2 em NICs and uses if_bridge + IPFW + ipfilter + pf.
> 
> This box usually gets a very high interrupt rate >90%.
> 
> By using netstat -I em0 1, we see:
> 
> input (em0) output
> packets errs bytes packets errs bytes colls
> 15164 0 9549410 16842 0 12132262 0
> 15331 0 9220723 17466 0 12354486 0
> 16486 0 10089165 17926 0 13256014 0
> 14936 10 8363433 17090 0 13367271 0
> 14712 0 8449705 16701 0 13443079 0
> 
> 
> When we enable the polling on em0 and em1 by ifconfig, netstat -I em0 shows:
> 
> input (em0) output
> packets errs bytes packets errs bytes colls
> 10910 0 6629176 11467 0 7681346 0
> 10951 0 6916042 11397 0 7954205 0
> 11281 0 7437813 10804 0 7390386 0
> 10711 0 6308033 11605 0 8400405 0
> 10807 0 6270716 10973 0 7700748 0
> 
> the interrupt rate decreases (<30%) but we get lower throuphts.
> 
> net.link.bridge.ipfw: 1
> net.link.bridge.pfil_member: 1
> net.link.bridge.pfil_bridge: 1
> net.link.bridge.pfil_onlyip: 0
> 
> How could we tune the box to reach higher throughputs

In the above setup you are running four packet filters, ipfilter, pf and
IPFW twice (ipfw=1 is an extra layer2 run). More specifically the packet filters
are actually invoked _10_ times per packet, in_if -> bridge_if -> out_if
per pfil_hook, plus the extra ipfw layer2.

This explains your high cpu usage, you may want to reorganise the
filtering to optimise it.


Andrew