From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 09:27:43 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4028516A403; Sat, 31 Mar 2007 09:27:43 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 0FD8D13C4BA; Sat, 31 Mar 2007 09:27:42 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l2V9Rfan095305; Sat, 31 Mar 2007 01:27:41 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l2V9Rf54095304; Sat, 31 Mar 2007 02:27:41 -0700 (PDT) (envelope-from rizzo) Date: Sat, 31 Mar 2007 02:27:41 -0700 From: Luigi Rizzo To: Andre Oppermann Message-ID: <20070331022741.A94927@xorpc.icir.org> References: <460D75CE.70804@elischer.org> <20070330145938.A88154@xorpc.icir.org> <460DA258.2030402@elischer.org> <460E19EE.3020700@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <460E19EE.3020700@freebsd.org>; from andre@freebsd.org on Sat, Mar 31, 2007 at 10:21:02AM +0200 Cc: FreeBSD Net , ipfw@freebsd.org, Julian Elischer Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 09:27:43 -0000 On Sat, Mar 31, 2007 at 10:21:02AM +0200, Andre Oppermann wrote: > Julian Elischer wrote: > > Luigi Rizzo wrote: > >> On Fri, Mar 30, 2007 at 01:40:46PM -0700, Julian Elischer wrote: > >>> I have been looking at the IPFW code recently, especially with > >>> respect to locking. > >>> There are some things that could be done to improve IPFW's behaviour ... > The locking overhead per packet in ipfw is by no means its limiting i think you and Julian are looking at different issues. if i understand julian's comment, the problem is that the list is protected by a single lock, so no hope of parallelising the work, and if one kernel thread is busy processing a packet in the filter, others might be blocked for a long time (in your case, the set of 30 rules is 765ns for ipfw and 1198ns for pf). Your tests presumably have little if any contention on the lock. Specifically, if you compute the difference of the inverses of those pps rates you see the following: +pfil_pass 45.3 ns per packet +ipfw_allow +253.4 ns/packet (setup and first rule) +ipfw_30 +17.67 ns/(packet * extra rule) +pf_pass +376.9 ns/packet (setup and first rule) +pf_30 +28.34 ns/(packet * extra rule) the lock acquisition cost is in the 'setup' part but i cannot tell how expensive it is. Julian's suggested change (and surely the one i described) replaces the lock/unlock pair on the rule list with a refcount add/dec pair (with uncontested locks the cost should be similar), but especially makes the operation non-blocking allowing running the input and output paths in parallel. > factor. Actually it's a very small part and pretty much any work on > it is lost love. It would be much better spent time to optimize the > main rule loop of ipfw to speed things up. I was profiling ipfw early > last year with an Agilent packet generator and hwpmc. In the meantime > the packet forwarding path (w/o ipfw) has been improved but relative > to each other the number are still correct. actually your numbers show that at least the rule setup (and the processing of simple rules) is significantly faster (50% or so) in ipfw2 than in pf. I know that the setup time is expensive, but i am not sure that one can save much - in both cases, you need to fetching a lot of information, which is scattered in variable locations in the mbuf and packet headers. cheers luigi