From owner-freebsd-pf@FreeBSD.ORG Thu Feb 17 16:58:47 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35C811065672; Thu, 17 Feb 2011 16:58:47 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 952778FC16; Thu, 17 Feb 2011 16:58:46 +0000 (UTC) Received: by ewy24 with SMTP id 24so1165116ewy.13 for ; Thu, 17 Feb 2011 08:58:45 -0800 (PST) Received: by 10.204.59.72 with SMTP id k8mr1939725bkh.208.1297961924670; Thu, 17 Feb 2011 08:58:44 -0800 (PST) Received: from dfleuriot-at-hi-media.com ([83.167.62.196]) by mx.google.com with ESMTPS id 12sm795520bki.7.2011.02.17.08.58.42 (version=SSLv3 cipher=OTHER); Thu, 17 Feb 2011 08:58:43 -0800 (PST) Message-ID: <4D5D53C2.3010707@my.gd> Date: Thu, 17 Feb 2011 17:58:42 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jack Vogel References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> <4D41D7BE.3030208@my.gd> <20110127205845.GA41537@icarus.home.lan> <4D429A9F.8040307@my.gd> In-Reply-To: <4D429A9F.8040307@my.gd> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , Jeremy Chadwick , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Feb 2011 16:58:47 -0000 On 1/28/11 11:29 AM, Damien Fleuriot wrote: > On 1/27/11 10:44 PM, Jack Vogel wrote: >> >> The 8.X kernel is NOT single-threaded. Anything but. And the stack has >> also been improved, I believe there are still bottlenecks but its far better >> than the old days. >> >> The igb driver in 8.2 creates up to 8 queues on the right hardware, they >> are each auto-bound to a particular CPU. >> >> The older version you are running had issues and hence multiqueue was >> not enabled. So, do upgrade once 8.2 is finalized :) >> >> Cheers, >> >> Jack >> > > Going to push for us to install 8.2 as soon as the release hits, thanks > for your feedback Jack :) Hello guys, list, This is a quick headsup regarding this issue. We have now swapped our PF firewalls to active-active and observed, as one would expect, approx. 50% drop of traffic, seeing it's now balanced between 2 machines :) We have also disabled pfsync (which also resulted in a massive drop of interrupts). One of the hosts is running 8.2-PRERELEASE , and this is the one for which I'm providing stats now. For completeness, also find the graphs: http://my.gd/fw_graphs/ # vmstat -i interrupt total rate irq16: mpt0 320899 0 irq21: atapci1 35 0 irq22: ehci0 ehci1 1992267 2 cpu0: timer 1330310985 1979 irq258: igb0:que 0 829898 1 irq259: igb0:que 1 3255 0 irq260: igb0:que 2 2059 0 irq261: igb0:que 3 1060 0 irq262: igb0:link 2 0 irq263: igb1:que 0 2676083520 3981 irq264: igb1:que 1 2676853656 3982 irq265: igb1:que 2 2682493388 3990 irq266: igb1:que 3 2688637571 3999 irq267: igb1:link 2 0 irq273: igb3:que 0 2654678899 3949 irq274: igb3:que 1 2648682488 3940 irq275: igb3:que 2 2650599952 3943 irq276: igb3:que 3 2657367887 3953 irq277: igb3:link 2 0 cpu1: timer 1330301807 1978 cpu2: timer 1330301315 1978 cpu3: timer 1330301347 1978 Total 26659762294 39659 # pfctl -si Status: Enabled for 7 days 18:43:34 Debug: Urgent Interface Stats for igb3 IPv4 IPv6 Bytes In 1585211309166 0 Bytes Out 2044715081803 0 Packets In Passed 6238056055 0 Blocked 15350206 0 Packets Out Passed 6300823415 0 Blocked 1223577 0 State Table Total Rate current entries 37627 searches 25108284353 37351.6/s inserts 2157108574 3209.0/s removals 2157070947 3208.9/s Counters match 2175657232 3236.6/s bad-offset 0 0.0/s fragment 104 0.0/s short 5 0.0/s normalize 557 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 6 0.0/s proto-cksum 52649 0.1/s state-mismatch 340029 0.5/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 90 0.0/s igb0@pci0:7:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82575GB Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xdabc0000, size 131072, enabled bar [14] = type Memory, range 32, base 0xdac00000, size 2097152, enabled bar [18] = type I/O Port, range 32, base 0xdcc0, size 32, enabled bar [1c] = type Memory, range 32, base 0xdabb8000, size 16384, enabled cap 01[40] = powerspec 2 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit cap 11[60] = MSI-X supports 10 messages in map 0x1c enabled cap 10[a0] = PCI-Express 2 endpoint max data 256(256) link x4(x4) ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 001b21ffff12f438 synproxy 0 0.0/s (there are 4 of these, it's a quad port card)