From owner-freebsd-net@FreeBSD.ORG Thu May 4 01:40:10 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 89F9516A403 for ; Thu, 4 May 2006 01:40:10 +0000 (UTC) (envelope-from tpeixoto@widesoft.com.br) Received: from srv1.netconsultoria.com.br (srv1.netconsultoria.com.br [200.230.201.252]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C1D243D4C for ; Thu, 4 May 2006 01:40:08 +0000 (GMT) (envelope-from tpeixoto@widesoft.com.br) Received: from [192.168.0.1] (mailgw.netconsultoria.com.br [200.230.201.249]) by srv1.netconsultoria.com.br (8.13.6/8.13.3) with ESMTP id k441dfuw081699; Wed, 3 May 2006 22:39:44 -0300 (BRT) (envelope-from tpeixoto@widesoft.com.br) Message-ID: <44595B76.9010901@widesoft.com.br> Date: Wed, 03 May 2006 22:40:06 -0300 From: tpeixoto@widesoft.com.br User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: .@babolo.ru References: <1146645702.297895.80691.nullmailer@cicuta.babolo.ru> In-Reply-To: <1146645702.297895.80691.nullmailer@cicuta.babolo.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.2/1438/Wed May 3 15:40:02 2006 on srv1.netconsultoria.com.br X-Virus-Status: Clean Cc: Lee Johnston , freebsd-net@freebsd.org, Julian Elischer , mihai@duras.ro Subject: Re: Packet loss with traffic shaper and routing X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 May 2006 01:40:10 -0000 Very good. You're right! I inserted a rule to match all non-layer2 packets on the top of the ruleset and interrupts dropped 10~20% immediately. Given that, I went to apply Julian's idea of grouping 'in' and 'out' pipe rules to reduce the searching on the firewall and that gave me a little bit more of performance. As interrupts were still hitting 60% mark, I did some more experiences: Test 1: I changed all 'pipe' rules to 'allow' rules, so all packets were allowed and no shaping was done. The pipes were still there, but there were no rules pointing packets to them. Result: No difference. Interrupts are the same as before. Conclusion: It's not the shaping itself that slows the system. Test 2: With the same ruleset of test 1, I just removed all pipes (ipfw pipe flush). Result: Interrupts were only 20%! Conclusion: Lots of pipes bother the system. I didn't figure out why, but it's not a coincidence. I tested several times to make sure. Test 3: I applied Michael's idea of using 'mask src-ip' and 'mask dst-ip' in the pipes to use them as a template for dynamic generated pipes. Result: Worked like a charm. Now I have only 18 pipes instead of 3200. Interrupts are ~30%. Conclusion: The reduced number of pipes generated less system interrupts. The only problem I noticed (so far) with this method is that if we have more than 1 IP address to a single MAC address, each IP will be shaped individually instead of share the same speed of the other(s) IP(s) with the same MAC. Anyway, I am very curious about the result of test 2. Why do the pipes have influence on system performance if there is nothing passing through them? Thank you very much everyone. "."@babolo.ru wrote: [...] > In your example each packet walk through the rule set 4 times > 1 mac input - abount half a ruleset average > 2 ip input - all ruleset, not succesfull > 3 ip output - all ruleset, not succesfull > 4 mac output - abount half a ruleset average > > allow all ip level packets on the ruleset begin and > down proc usage 3 times down. > [...]