From owner-freebsd-net@FreeBSD.ORG Wed Jan 20 19:07:41 2010 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2482A106566C for ; Wed, 20 Jan 2010 19:07:41 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id E18508FC16 for ; Wed, 20 Jan 2010 19:07:40 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id F080F730A1; Wed, 20 Jan 2010 20:15:59 +0100 (CET) Date: Wed, 20 Jan 2010 20:15:59 +0100 From: Luigi Rizzo To: net@freebsd.org Message-ID: <20100120191559.GA71192@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: best way to optimize ipfw-dummynet interaction ? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2010 19:07:41 -0000 Hi, I am trying to solve the following problem: when ipfw passes a packet to dummynet, the packet is tagged with a 'pipe number' which dummynet needs to look up to find the pointer to the correct object -- something like this: IPFW_LOCK() rule = find_matching_rule(); 1. ... pkt.tag.pipeNR = rule->pipe_number; IPFW_UNLOCK() ... // now ipfw_pfil_hook calls dummynet: DUMMYNET_LOCK() 2. pipePTR = lookup(pkt.tag.pipeNR) ... dummynet processing DUMMYNET_UNLOCK() // ... after some time the packet is delivered 3. ip_output(pkt) It would be nice to cache the result of a lookup in the ipfw rule or in the table entry so we don't have to pay the cost all the times (the result would be stored as a tuple same as ipfw rule pointers, so you can detect changes and do a lookup if the generation number changes.) But i cannot think of a good way of doing it -- i have two options in mind: A) right after #1, call into dummynet while still holding the IPFW LOCK: if (rule->pipePTR.generation != dummynet.generation) { DUMMYNET_LOCK() rule->pipePTR = lookup(pkt.tag.pipeNR) DUMMYNET_UNLOCK() } We can probably do the check outside the lock because the cached value is only advisory and if wrong we will detect it some other time (as long as the dummynet.generation variable does not go away). B) + right after #1, store a reference to the rule (in the usual way involving a generation number) in the packet tag; + right after #2, store a reference to the pipePTR in the packet tag; + after #3, if the packet happens to enter IPFW again (or if it does not, do it on purpose) and the rulePTR is still valid, then store the result in rulePTR. Solution A is somewhat simpler because the code is exactly the one i wrote above, but i am not particularly fond of the nesting of the locks. Solution B solves the lock nesting problem by piggybacking the message onto the packet, but it might require a bit of extra work for copying and checking those "safe references". Finally, we should make sure that the 'reinjection' of information occurs even if packets are systematically dropped (not unlikely to configure an ipfw rule that points to a non-existing pipe, in which case we have useless lookups; we could cache a negative info and save the lookup). Comments or suggestions anyone ? With the recent optimization work on ipfw and dummynet this is probably the last piece that uses a suboptimal algorithm (the lookup is through a hash table but constants do matter here). cheers luigi