From owner-freebsd-pf@FreeBSD.ORG Sun Jan 23 08:24:37 2011 Return-Path: Delivered-To: freebsd-pf@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B911106564A for ; Sun, 23 Jan 2011 08:24:37 +0000 (UTC) (envelope-from mohacsi@niif.hu) Received: from mail.ki.iif.hu (mail.ki.iif.hu [IPv6:2001:738:0:411::241]) by mx1.freebsd.org (Postfix) with ESMTP id DE1BB8FC0A for ; Sun, 23 Jan 2011 08:24:36 +0000 (UTC) Received: from bolha.lvs.iif.hu (bolha.lvs.iif.hu [193.225.14.181]) by mail.ki.iif.hu (Postfix) with ESMTP id 1E5BF8722C; Sun, 23 Jan 2011 09:24:35 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at bolha.lvs.iif.hu Received: from mail.ki.iif.hu ([IPv6:::ffff:193.6.222.241]) by bolha.lvs.iif.hu (bolha.lvs.iif.hu [::ffff:193.225.14.72]) (amavisd-new, port 10024) with ESMTP id Xo4-Awh5WDjK; Sun, 23 Jan 2011 09:24:19 +0100 (CET) Received: by mail.ki.iif.hu (Postfix, from userid 9002) id 9646387220; Sun, 23 Jan 2011 09:24:19 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mail.ki.iif.hu (Postfix) with ESMTP id 91573871EB; Sun, 23 Jan 2011 09:24:19 +0100 (CET) Date: Sun, 23 Jan 2011 09:24:19 +0100 (CET) From: Mohacsi Janos X-X-Sender: mohacsi@mignon.ki.iif.hu To: Christian Laursen In-Reply-To: <4D3ADF52.1020205@borderworlds.dk> Message-ID: References: <4D3ADF52.1020205@borderworlds.dk> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-pf@FreeBSD.org Subject: Re: NAT64 support in pf? X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jan 2011 08:24:37 -0000 Hi, nat64 code for OpenBSD has been developed: http://ecdysis.viagenie.ca/download.html This should be ported to FreeBSD, since uses BSD license. Janos Mohacsi Head of HBONE+ project Network Engineer, Deputy Director of Network Planning and Projects NIIF/HUNGARNET, HUNGARY Key 70EF9882: DEC2 C685 1ED4 C95A 145F 4300 6F64 7B00 70EF 9882 On Sat, 22 Jan 2011, Christian Laursen wrote: > Hello, > > Are there any plans to add NAT64 functionality to the pf implementation in > FreeBSD? > > It looks like BIND 9.8 will have DNS64 support and together with FreeBSD/pf > that would make a nice combination. > > Thanks. > > -- > Christian Laursen > _______________________________________________ > freebsd-pf@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > From owner-freebsd-pf@FreeBSD.ORG Mon Jan 24 11:07:06 2011 Return-Path: Delivered-To: freebsd-pf@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 570311065793 for ; Mon, 24 Jan 2011 11:07:06 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 456368FC17 for ; Mon, 24 Jan 2011 11:07:06 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p0OB76dI077878 for ; Mon, 24 Jan 2011 11:07:06 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p0OB754j077876 for freebsd-pf@FreeBSD.org; Mon, 24 Jan 2011 11:07:05 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Jan 2011 11:07:05 GMT Message-Id: <201101241107.p0OB754j077876@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-pf@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-pf@FreeBSD.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jan 2011 11:07:06 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/153307 pf [pf] Bug with PF firewall o kern/148290 pf [pf] "sticky-address" option of Packet Filter (PF) blo o kern/148260 pf [pf] [patch] pf rdr incompatible with dummynet o kern/147789 pf [pf] Firewall PF no longer drops connections by sendin o kern/146832 pf [pf] "(self)" not always matching all local IPv6 addre o kern/143543 pf [pf] [panic] PF route-to causes kernel panic o bin/143504 pf [patch] outgoing states are not killed by authpf(8) o conf/142961 pf [pf] No way to adjust pidfile in pflogd o conf/142817 pf [patch] etc/rc.d/pf: silence pfctl o kern/141905 pf [pf] [panic] pf kernel panic on 7.2-RELEASE with empty o kern/140697 pf [pf] pf behaviour changes - must be documented o kern/137982 pf [pf] when pf can hit state limits, random IP failures o kern/136781 pf [pf] Packets appear to drop with pf scrub and if_bridg o kern/135948 pf [pf] [gre] pf not natting gre protocol o kern/135162 pf [pfsync] pfsync(4) not usable with GENERIC kernel o kern/134996 pf [pf] Anchor tables not included when pfctl(8) is run w o kern/133732 pf [pf] max-src-conn issue o kern/132769 pf [pf] [lor] 2 LOR's with pf task mtx / ifnet and rtent f kern/132176 pf [pf] pf stalls connection when using route-to [regress o conf/130381 pf [rc.d] [pf] [ip6] ipv6 not fully configured when pf st o kern/129861 pf [pf] [patch] Argument names reversed in pf_table.c:_co o kern/127920 pf [pf] ipv6 and synproxy don't play well together o conf/127814 pf [pf] The flush in pf_reload in /etc/rc.d/pf does not w o kern/127439 pf [pf] deadlock in pf f kern/127345 pf [pf] Problem with PF on FreeBSD7.0 [regression] o kern/127121 pf [pf] [patch] pf incorrect log priority o kern/127042 pf [pf] [patch] pf recursion panic if interface group is o kern/125467 pf [pf] pf keep state bug while handling sessions between s kern/124933 pf [pf] [ip6] pf does not support (drops) IPv6 fragmented o kern/124364 pf [pf] [panic] Kernel panic with pf + bridge o kern/122773 pf [pf] pf doesn't log uid or pid when configured to o kern/122014 pf [pf] [panic] FreeBSD 6.2 panic in pf o kern/120281 pf [pf] [request] lost returning packets to PF for a rdr o kern/120057 pf [pf] [patch] Allow proper settings of ALTQ_HFSC. The c o bin/118355 pf [pf] [patch] pfctl(8) help message options order false o kern/114567 pf [pf] [lor] pf_ioctl.c + if.c o kern/114095 pf [carp] carp+pf delay with high state limit o kern/111220 pf [pf] repeatable hangs while manipulating pf tables s conf/110838 pf [pf] tagged parameter on nat not working on FreeBSD 5. o kern/103283 pf pfsync fails to sucessfully transfer some sessions o kern/103281 pf pfsync reports bulk update failures o kern/93825 pf [pf] pf reply-to doesn't work o sparc/93530 pf [pf] Incorrect checksums when using pf's route-to on s o kern/92949 pf [pf] PF + ALTQ problems with latency o bin/86635 pf [patch] pfctl(8): allow new page character (^L) in pf. o kern/82271 pf [pf] cbq scheduler cause bad latency 46 problems total. From owner-freebsd-pf@FreeBSD.ORG Wed Jan 26 13:18:17 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1B4C1065679 for ; Wed, 26 Jan 2011 13:18:17 +0000 (UTC) (envelope-from vision2020-bounces@moscow.com) Received: from trumpet.fsr.net (trumpet.fsr.net [64.126.132.9]) by mx1.freebsd.org (Postfix) with ESMTP id 930828FC21 for ; Wed, 26 Jan 2011 13:18:17 +0000 (UTC) Received: from trumpet.fsr.net (localhost [127.0.0.1]) by trumpet.fsr.net (8.14.3/8.13.6) with ESMTP id p0QCqWcH089388 for ; Wed, 26 Jan 2011 04:52:32 -0800 (PST) (envelope-from vision2020-bounces@moscow.com) MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit From: vision2020-bounces@moscow.com To: freebsd-pf@freebsd.org Message-ID: Date: Wed, 26 Jan 2011 04:52:29 -0800 Precedence: bulk X-BeenThere: vision2020@moscow.com X-Mailman-Version: 2.1.9 X-List-Administrivia: yes Sender: vision2020-bounces@moscow.com Errors-To: vision2020-bounces@moscow.com Subject: Your message to Vision2020 awaits moderator approval X-BeenThere: freebsd-pf@freebsd.org List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jan 2011 13:18:17 -0000 Your mail to 'Vision2020' with the subject test Is being held until the list moderator can review it for approval. The reason it is being held: Post by non-member to a members-only list Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL: http://mailman.fsr.com/mailman/confirm/vision2020/32d8b9dc0e841e9d3e1549b148b821c6d5398e46 From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 09:57:26 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AAAEE1065673; Thu, 27 Jan 2011 09:57:26 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 02DD88FC0C; Thu, 27 Jan 2011 09:57:24 +0000 (UTC) Received: by bwz12 with SMTP id 12so2249634bwz.13 for ; Thu, 27 Jan 2011 01:57:23 -0800 (PST) Received: by 10.204.56.3 with SMTP id w3mr1368730bkg.60.1296122242330; Thu, 27 Jan 2011 01:57:22 -0800 (PST) Received: from dfleuriot.technique-admin.paris.hi-media-techno.com ([83.167.62.196]) by mx.google.com with ESMTPS id 12sm7986486bki.19.2011.01.27.01.57.16 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 01:57:18 -0800 (PST) Message-ID: <4D41417A.20904@my.gd> Date: Thu, 27 Jan 2011 10:57:14 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-pf@freebsd.org Subject: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 09:57:26 -0000 Hello list, I have a problem with interrupts, network cards, and PF performance. We have 2 firewalls running FreeBSD 8.0 for the current master and FreeBSD 8.1 for the backup host, which I upgraded just yesterday. The servers use CARP for redundancy. These are rather busy boxes which run PF and nginx as a reverse proxy. As you will see below, we're getting a "high" %interrupt CPU usage, which seems to come mostly from the NICs. I'm wondering if there is any way to optimize the box's performance and reduce the interrupts rate or the CPU usage ? Also, we've noticed a sharp drop in CPU usage since we've disabled pfsync, but we'd rather keep it now wouldn't we ? Last, we seem to get input errors on the NICs, although the switch ports report not a single layer 2 error in over a year. I'm wondering what counts as a NIC input error ? Hardware is as follows: CPU -- CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2496.25-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Features=0xbfebfbff Features2=0xce3bd AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 MEM -- real memory = 2147483648 (2048 MB) avail memory = 2057293824 (1961 MB) NICs -- bce0: mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7 bce1: mem 0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3 igb0: port 0xdce0-0xdcff mem 0xfd0e0000-0xfd0fffff,0xfce00000-0xfcffffff,0xfd0dc000-0xfd0dffff irq 18 at device 0.0 on pci14 igb0: Using MSIX interrupts with 3 vectors Find below different outputs from the current master running FreeBSD 8.0-RELEASE-p2 systat -v --- 3 users Load 0.41 0.31 0.29 Jan 26 18:59 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 143036 8152 836392 11188 1262556 count All 168224 10420 1074653k 31172 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 36163 total 47 105k 76 2077 28k 223 zfod ata0 irq14 ozfod mfi0 irq16 4.3%Sys 28.1%Intr 3.0%User 0.0%Nice 64.7%Idle %ozfod uhci0 uhci | | | | | | | | | | | daefr 1998 cpu0: time ==++++++++++++++>> prcfr 9428 bce0 256 33 dtbuf totfr 12931 igb0 257 Namei Name-cache Dir-cache 100000 desvn react 5791 igb0 258 Calls hits % hits % 70448 numvn pdwak igb0 259 24988 frevn pdpgs igb1 260 intrn 1 igb1 261 Disks mfid0 372392 wire igb1 262 KB/t 0.00 62336 act 20 bce1 269 tps 0 323720 inact 1998 cpu1: time MB/s 0.00 292 cache 1998 cpu2: time %busy 0 1262264 free 1998 cpu3: time 218272 buf vmstat -i --- interrupt total rate irq14: ata0 36 0 irq16: mfi0 353244 1 irq21: uhci0 uhci+ 461504 1 cpu0: timer 615183815 1996 irq256: bce0 1015412475 3295 irq257: igb0 1067318584 3464 irq258: igb0 695648752 2258 irq259: igb0 2 0 irq260: igb1 11503857 37 irq261: igb1 506598 1 irq262: igb1 69 0 irq269: bce1 790820 2 cpu1: timer 615183757 1996 cpu2: timer 615197165 1996 cpu3: timer 615197165 1996 Total 5252757843 17050 pf status (159 filter rules, 17 nat/rdr rules) --- # pfctl -si Status: Enabled for 3 days 13:34:56 Debug: Urgent Interface Stats for igb0 IPv4 IPv6 Bytes In 487209136643 384 Bytes Out 687158173727 0 Packets In Passed 1967249106 0 Blocked 6183860 6 Packets Out Passed 2018192359 0 Blocked 686901 0 State Table Total Rate current entries 25428 searches 9006187476 29231.8/s inserts 679746853 2206.3/s removals 679721425 2206.2/s Counters match 686988143 2229.8/s bad-offset 0 0.0/s fragment 56 0.0/s short 0 0.0/s normalize 171 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 1 0.0/s proto-cksum 13916 0.0/s state-mismatch 220169 0.7/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 1812 0.0/s synproxy 0 0.0/s Regards, -- dfl From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 10:28:32 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB8B2106566B for ; Thu, 27 Jan 2011 10:28:32 +0000 (UTC) (envelope-from bartosz.stec@it4pro.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id 5FB128FC1A for ; Thu, 27 Jan 2011 10:28:32 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1PiOhv-000BZ8-2q; Thu, 27 Jan 2011 11:04:10 +0100 Message-ID: <4D414304.3090905@it4pro.pl> Date: Thu, 27 Jan 2011 11:03:48 +0100 From: Bartosz Stec Organization: IT4Pro User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Damien Fleuriot References: <4D41417A.20904@my.gd> In-Reply-To: <4D41417A.20904@my.gd> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-User: bartosz.stec@it4pro.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.1 X-Spam-Score-Int: -80 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-01-27 11:04:10 X-Connected-IP: 78.8.144.74:63193 X-Message-Linecount: 27 X-Body-Linecount: 12 X-Message-Size: 877 X-Body-Size: 254 X-Received-Count: 1 X-Recipient-Count: 3 X-Local-Recipient-Count: 3 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Cc: "freebsd-stable@freebsd.org" , freebsd-pf@freebsd.org Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 10:28:32 -0000 W dniu 2011-01-27 10:57, Damien Fleuriot pisze: > Hello list, > > I have a problem with interrupts, network cards, and PF performance. > I think you should try with polling(4) enabled and probably increase kernel.hz i sysctl.conf :) -- Bartosz Stec From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 10:48:38 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DD17106564A; Thu, 27 Jan 2011 10:48:38 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1E15A8FC0A; Thu, 27 Jan 2011 10:48:37 +0000 (UTC) Received: by gxk8 with SMTP id 8so578466gxk.13 for ; Thu, 27 Jan 2011 02:48:37 -0800 (PST) Received: by 10.100.105.16 with SMTP id d16mr461544anc.219.1296125316863; Thu, 27 Jan 2011 02:48:36 -0800 (PST) Received: from dfleuriot.technique-admin.paris.hi-media-techno.com ([83.167.62.196]) by mx.google.com with ESMTPS id f10sm20287328anh.5.2011.01.27.02.48.34 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 02:48:35 -0800 (PST) Message-ID: <4D414D80.3060706@my.gd> Date: Thu, 27 Jan 2011 11:48:32 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Bartosz Stec References: <4D41417A.20904@my.gd> <4D414304.3090905@it4pro.pl> In-Reply-To: <4D414304.3090905@it4pro.pl> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-stable@freebsd.org" , freebsd-pf@freebsd.org Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 10:48:38 -0000 On 1/27/11 11:03 AM, Bartosz Stec wrote: > W dniu 2011-01-27 10:57, Damien Fleuriot pisze: >> Hello list, >> >> I have a problem with interrupts, network cards, and PF performance. >> > I think you should try with polling(4) enabled and probably increase > kernel.hz i sysctl.conf :) > As a matter of fact, we tried polling on the backup firewall yesterday with the following kernel options: options DEVICE_POLLING options HZ=1000 This had disastrous results. First, our LAN and DMZ interfaces (bce0 and 1) do not support polling, so no change here. Second, the WAN interface (igb0) supports polling but that caused problems with carp0 and the physical interface resetting itself for god knows what reason: carp0: link state changed to DOWN carp0: INIT -> BACKUP igb0: link state changed to UP carp0: link state changed to DOWN carp0: link state changed to UP carp0: MASTER -> BACKUP (more frequent advertisement received) carp0: link state changed to DOWN carp0: link state changed to UP igb0: Watchdog timeout -- resetting igb0: Queue(1) tdh = 57, hw tdt = 57 igb0: TX(1) desc avail = 967,Next TX to Clean = 0 igb0: link state changed to DOWN carp0: link state changed to DOWN carp0: INIT -> BACKUP igb0: link state changed to UP carp0: link state changed to DOWN carp0: link state changed to UP carp0: link state changed to DOWN igb0: Watchdog timeout -- resetting igb0: Queue(3) tdh = 5, hw tdt = 5 igb0: TX(3) desc avail = 1019,Next TX to Clean = 0 igb0: link state changed to DOWN igb0: link state changed to UP igb0: Watchdog timeout -- resetting igb0: Queue(2) tdh = 53, hw tdt = 53 igb0: TX(2) desc avail = 971,Next TX to Clean = 0 igb0: link state changed to DOWN igb0: link state changed to UP igb0: Watchdog timeout -- resetting igb0: Queue(2) tdh = 19, hw tdt = 19 igb0: TX(2) desc avail = 1005,Next TX to Clean = 0 igb0: link state changed to DOWN igb0: link state changed to UP From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 17:00:06 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E5CA1065693 for ; Thu, 27 Jan 2011 17:00:06 +0000 (UTC) (envelope-from kevin.wilcox@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id F3E9A8FC26 for ; Thu, 27 Jan 2011 17:00:05 +0000 (UTC) Received: by gyf3 with SMTP id 3so745390gyf.13 for ; Thu, 27 Jan 2011 09:00:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=nuIe/5NQ18bAQSpMgNbYdj5wfuenmlbMhBb89Tm10Go=; b=A1MGzp2fZCVexefCp0Ow+zMZpGTJQtLJfe0lp/Fh6blTCvyg5MkmmUZDfu5qFsdYay jNtYhGPSjfydUmKrHsTBmqc6MeqTTdT2NsQy9FY5brdYDV4Al41az/btgpg4c6gDMUa0 mMMiMAEdH7l9qRjPGZGPajRS3npLmLNqFkkjA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=wLXjuRyqtQEp9cJKy37D5UjTRSUKx1VV9pGctfw8jtZJB157w724FLl0t9Yjm0VWKG uPuxX69RtvaLvY75N06nVAY6b3hqTQQLSIZR0BknyOcveYrv2BAXiVARait5E3IbgJVo n3A9GDVEWdrlln6VuLYvQJrcjD8fl1B/8MOlI= MIME-Version: 1.0 Received: by 10.90.100.6 with SMTP id x6mr2491521agb.33.1296146031312; Thu, 27 Jan 2011 08:33:51 -0800 (PST) Received: by 10.90.79.10 with HTTP; Thu, 27 Jan 2011 08:33:51 -0800 (PST) Date: Thu, 27 Jan 2011 11:33:51 -0500 Message-ID: From: Kevin Wilcox To: freebsd-pf@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: log NAT translations X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 17:00:06 -0000 Hello all. I've been using FreeBSD 7.x and 8.x for bridged firewalls and logging hasn't been an issue. Now I'm moving one of them to NAT and I suddenly realise I have a major problem - I can't log the actual translations. Consider the following: Client A - 10.1.1.1 Client B - 10.1.2.2 Remote server C - some IP out on the Internet Inside firewall interface: 10.1.2.254 Outside firewall interface: 192.168.1.1 The sysadmin for C comes to me and says, "hey, someone from 192.168.1.1, source port 12345, is banging on my server on port 80." I go to the logs for my firewall, logging on both interfaces. The log for the inside interface shows connections from clients A and B going to C on port 80 with source ports 30000 and 40000. I go to the log for the outside interface and see connections going from 192.168.1.1 to server C, destination port 80, source ports 12345 and 23456. My problem is that I can't tie the inside IP:port to the translated IP:port, so while I can narrow it down to a couple of internal IPs, I can't pinpoint which client is being civil and which one is causing the problem. Before I write something to interpret state changes from pfsync, can anyone offer guidance on how to pull those translations? Thanks! kmw From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 17:31:57 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 52A1C1065670; Thu, 27 Jan 2011 17:31:57 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id BCBEE8FC17; Thu, 27 Jan 2011 17:31:56 +0000 (UTC) Received: by fxm16 with SMTP id 16so2505281fxm.13 for ; Thu, 27 Jan 2011 09:31:55 -0800 (PST) Received: by 10.223.73.206 with SMTP id r14mr1169451faj.126.1296149492773; Thu, 27 Jan 2011 09:31:32 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id r24sm6084315fax.27.2011.01.27.09.31.30 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 09:31:31 -0800 (PST) Message-ID: <4D41ABF1.1010405@my.gd> Date: Thu, 27 Jan 2011 18:31:29 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D41417A.20904@my.gd> <20110127172724.GA36587@icarus.home.lan> In-Reply-To: <20110127172724.GA36587@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , freebsd-pf@freebsd.org Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 17:31:57 -0000 On 1/27/11 6:27 PM, Jeremy Chadwick wrote: > On Thu, Jan 27, 2011 at 10:57:14AM +0100, Damien Fleuriot wrote: >> Hello list, >> >> I have a problem with interrupts, network cards, and PF performance. >> >> We have 2 firewalls running FreeBSD 8.0 for the current master and >> FreeBSD 8.1 for the backup host, which I upgraded just yesterday. >> >> [...] >> >> vmstat -i >> --- >> interrupt total rate >> irq14: ata0 36 0 >> irq16: mfi0 353244 1 >> irq21: uhci0 uhci+ 461504 1 >> cpu0: timer 615183815 1996 >> irq256: bce0 1015412475 3295 >> irq257: igb0 1067318584 3464 >> irq258: igb0 695648752 2258 >> irq259: igb0 2 0 >> irq260: igb1 11503857 37 >> irq261: igb1 506598 1 >> irq262: igb1 69 0 >> irq269: bce1 790820 2 >> cpu1: timer 615183757 1996 >> cpu2: timer 615197165 1996 >> cpu3: timer 615197165 1996 >> Total 5252757843 17050 > > There are changes to the igb(4) driver which are in RELENG_8 (8-STABLE), > and some which will be in the upcoming 8.2-RELEASE, which may address > this. Jack Vogel of Intel would be able to confirm for sure; CC'ing him > here. > > Could you please provide output from the following commands? > > * pciconf -lvcb (only include igbX entries, thanks) > * sysctl -a | grep msi > > Thanks. > > I can't help with the CARP-related issues or other stuff you're > experiencing. These issues may all be separate problems, hard to say. > igb0@pci0:14:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 igb1@pci0:14:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 igb2@pci0:15:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 igb3@pci0:15:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 hw.bce.msi_enable: 1 hw.pci.honor_msi_blacklist: 1 hw.pci.enable_msix: 1 hw.pci.enable_msi: 1 igb0: flags=8943 metric 0 mtu 1500 options=13b ether 00:1b:21:12:ec:38 inet [snip] netmask 0xffffffc0 broadcast [snip] media: Ethernet autoselect (1000baseT ) status: active igb1: flags=8843 metric 0 mtu 1500 options=13b ether 00:1b:21:12:ec:39 inet 10.0.0.252 netmask 0xffffff00 broadcast 10.0.0.255 media: Ethernet autoselect (1000baseT ) status: active Here you go :) Note that the igb2 and 3 interfaces are unused, unplugged, unconfigured From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 17:39:50 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4217106566C; Thu, 27 Jan 2011 17:39:50 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 1AC4E8FC0C; Thu, 27 Jan 2011 17:39:49 +0000 (UTC) Received: by wwf26 with SMTP id 26so2264259wwf.31 for ; Thu, 27 Jan 2011 09:39:48 -0800 (PST) Received: by 10.227.182.68 with SMTP id cb4mr1369901wbb.218.1296149988825; Thu, 27 Jan 2011 09:39:48 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id y29sm6835608wbd.16.2011.01.27.09.39.36 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 09:39:38 -0800 (PST) Message-ID: <4D41ADD4.6050507@my.gd> Date: Thu, 27 Jan 2011 18:39:32 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D41417A.20904@my.gd> <20110127172724.GA36587@icarus.home.lan> <4D41ABF1.1010405@my.gd> <20110127173723.GA36846@icarus.home.lan> In-Reply-To: <20110127173723.GA36846@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , freebsd-pf@freebsd.org Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 17:39:50 -0000 On 1/27/11 6:37 PM, Jeremy Chadwick wrote: > On Thu, Jan 27, 2011 at 06:31:29PM +0100, Damien Fleuriot wrote: >> >> >> On 1/27/11 6:27 PM, Jeremy Chadwick wrote: >>> On Thu, Jan 27, 2011 at 10:57:14AM +0100, Damien Fleuriot wrote: >>>> Hello list, >>>> >>>> I have a problem with interrupts, network cards, and PF performance. >>>> >>>> We have 2 firewalls running FreeBSD 8.0 for the current master and >>>> FreeBSD 8.1 for the backup host, which I upgraded just yesterday. >>>> >>>> [...] >>>> >>>> vmstat -i >>>> --- >>>> interrupt total rate >>>> irq14: ata0 36 0 >>>> irq16: mfi0 353244 1 >>>> irq21: uhci0 uhci+ 461504 1 >>>> cpu0: timer 615183815 1996 >>>> irq256: bce0 1015412475 3295 >>>> irq257: igb0 1067318584 3464 >>>> irq258: igb0 695648752 2258 >>>> irq259: igb0 2 0 >>>> irq260: igb1 11503857 37 >>>> irq261: igb1 506598 1 >>>> irq262: igb1 69 0 >>>> irq269: bce1 790820 2 >>>> cpu1: timer 615183757 1996 >>>> cpu2: timer 615197165 1996 >>>> cpu3: timer 615197165 1996 >>>> Total 5252757843 17050 >>> >>> There are changes to the igb(4) driver which are in RELENG_8 (8-STABLE), >>> and some which will be in the upcoming 8.2-RELEASE, which may address >>> this. Jack Vogel of Intel would be able to confirm for sure; CC'ing him >>> here. >>> >>> Could you please provide output from the following commands? >>> >>> * pciconf -lvcb (only include igbX entries, thanks) >>> * sysctl -a | grep msi >>> >>> Thanks. >>> >>> I can't help with the CARP-related issues or other stuff you're >>> experiencing. These issues may all be separate problems, hard to say. >>> >> > > What you did here was "pciconf -lvcb | grep igb", or something > equivalent. Indeed, my bad. igb0@pci0:14:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82575GB Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xfd0e0000, size 131072, enabled bar [14] = type Memory, range 32, base 0xfce00000, size 2097152, enabled bar [18] = type I/O Port, range 32, base 0xdce0, size 32, enabled bar [1c] = type Memory, range 32, base 0xfd0dc000, size 16384, enabled cap 01[40] = powerspec 2 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit cap 11[60] = MSI-X supports 10 messages in map 0x1c enabled cap 10[a0] = PCI-Express 2 endpoint max data 256(256) link x4(x4) igb1@pci0:14:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82575GB Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xfd0a0000, size 131072, enabled bar [14] = type Memory, range 32, base 0xfcc00000, size 2097152, enabled bar [18] = type I/O Port, range 32, base 0xdcc0, size 32, enabled bar [1c] = type Memory, range 32, base 0xfd0d8000, size 16384, enabled cap 01[40] = powerspec 2 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit cap 11[60] = MSI-X supports 10 messages in map 0x1c enabled cap 10[a0] = PCI-Express 2 endpoint max data 256(256) link x4(x4) From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 17:40:39 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 29B8A106564A for ; Thu, 27 Jan 2011 17:40:39 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta08.emeryville.ca.mail.comcast.net (qmta08.emeryville.ca.mail.comcast.net [76.96.30.80]) by mx1.freebsd.org (Postfix) with ESMTP id 0F1178FC19 for ; Thu, 27 Jan 2011 17:40:37 +0000 (UTC) Received: from omta05.emeryville.ca.mail.comcast.net ([76.96.30.43]) by qmta08.emeryville.ca.mail.comcast.net with comcast id 0hMq1g0070vp7WLA8hTShA; Thu, 27 Jan 2011 17:27:26 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta05.emeryville.ca.mail.comcast.net with comcast id 0hTQ1g00K2tehsa8RhTQJ3; Thu, 27 Jan 2011 17:27:26 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 3E4DB9B422; Thu, 27 Jan 2011 09:27:24 -0800 (PST) Date: Thu, 27 Jan 2011 09:27:24 -0800 From: Jeremy Chadwick To: Damien Fleuriot Message-ID: <20110127172724.GA36587@icarus.home.lan> References: <4D41417A.20904@my.gd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D41417A.20904@my.gd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , freebsd-pf@freebsd.org Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 17:40:39 -0000 On Thu, Jan 27, 2011 at 10:57:14AM +0100, Damien Fleuriot wrote: > Hello list, > > I have a problem with interrupts, network cards, and PF performance. > > We have 2 firewalls running FreeBSD 8.0 for the current master and > FreeBSD 8.1 for the backup host, which I upgraded just yesterday. > > [...] > > vmstat -i > --- > interrupt total rate > irq14: ata0 36 0 > irq16: mfi0 353244 1 > irq21: uhci0 uhci+ 461504 1 > cpu0: timer 615183815 1996 > irq256: bce0 1015412475 3295 > irq257: igb0 1067318584 3464 > irq258: igb0 695648752 2258 > irq259: igb0 2 0 > irq260: igb1 11503857 37 > irq261: igb1 506598 1 > irq262: igb1 69 0 > irq269: bce1 790820 2 > cpu1: timer 615183757 1996 > cpu2: timer 615197165 1996 > cpu3: timer 615197165 1996 > Total 5252757843 17050 There are changes to the igb(4) driver which are in RELENG_8 (8-STABLE), and some which will be in the upcoming 8.2-RELEASE, which may address this. Jack Vogel of Intel would be able to confirm for sure; CC'ing him here. Could you please provide output from the following commands? * pciconf -lvcb (only include igbX entries, thanks) * sysctl -a | grep msi Thanks. I can't help with the CARP-related issues or other stuff you're experiencing. These issues may all be separate problems, hard to say. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 17:51:06 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CE02106566B for ; Thu, 27 Jan 2011 17:51:06 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.westchester.pa.mail.comcast.net (qmta14.westchester.pa.mail.comcast.net [76.96.59.212]) by mx1.freebsd.org (Postfix) with ESMTP id D14778FC08 for ; Thu, 27 Jan 2011 17:51:05 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by qmta14.westchester.pa.mail.comcast.net with comcast id 0hag1g0050bG4ec5Ehdqs1; Thu, 27 Jan 2011 17:37:50 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta03.westchester.pa.mail.comcast.net with comcast id 0hdS1g01s2tehsa3PhdVHT; Thu, 27 Jan 2011 17:37:45 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 28F6C9B422; Thu, 27 Jan 2011 09:37:23 -0800 (PST) Date: Thu, 27 Jan 2011 09:37:23 -0800 From: Jeremy Chadwick To: Damien Fleuriot Message-ID: <20110127173723.GA36846@icarus.home.lan> References: <4D41417A.20904@my.gd> <20110127172724.GA36587@icarus.home.lan> <4D41ABF1.1010405@my.gd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D41ABF1.1010405@my.gd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , freebsd-pf@freebsd.org Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 17:51:06 -0000 On Thu, Jan 27, 2011 at 06:31:29PM +0100, Damien Fleuriot wrote: > > > On 1/27/11 6:27 PM, Jeremy Chadwick wrote: > > On Thu, Jan 27, 2011 at 10:57:14AM +0100, Damien Fleuriot wrote: > >> Hello list, > >> > >> I have a problem with interrupts, network cards, and PF performance. > >> > >> We have 2 firewalls running FreeBSD 8.0 for the current master and > >> FreeBSD 8.1 for the backup host, which I upgraded just yesterday. > >> > >> [...] > >> > >> vmstat -i > >> --- > >> interrupt total rate > >> irq14: ata0 36 0 > >> irq16: mfi0 353244 1 > >> irq21: uhci0 uhci+ 461504 1 > >> cpu0: timer 615183815 1996 > >> irq256: bce0 1015412475 3295 > >> irq257: igb0 1067318584 3464 > >> irq258: igb0 695648752 2258 > >> irq259: igb0 2 0 > >> irq260: igb1 11503857 37 > >> irq261: igb1 506598 1 > >> irq262: igb1 69 0 > >> irq269: bce1 790820 2 > >> cpu1: timer 615183757 1996 > >> cpu2: timer 615197165 1996 > >> cpu3: timer 615197165 1996 > >> Total 5252757843 17050 > > > > There are changes to the igb(4) driver which are in RELENG_8 (8-STABLE), > > and some which will be in the upcoming 8.2-RELEASE, which may address > > this. Jack Vogel of Intel would be able to confirm for sure; CC'ing him > > here. > > > > Could you please provide output from the following commands? > > > > * pciconf -lvcb (only include igbX entries, thanks) > > * sysctl -a | grep msi > > > > Thanks. > > > > I can't help with the CARP-related issues or other stuff you're > > experiencing. These issues may all be separate problems, hard to say. > > > > > igb0@pci0:14:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 > rev=0x02 hdr=0x00 > igb1@pci0:14:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 > rev=0x02 hdr=0x00 > igb2@pci0:15:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 > rev=0x02 hdr=0x00 > igb3@pci0:15:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 > rev=0x02 hdr=0x00 What you did here was "pciconf -lvcb | grep igb", or something equivalent. There is output after each of these entries which is highly relevant. Example for an emX device: em1@pci0:15:0:0: class=0x020000 card=0x109a15d9 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = 'Intel PRO/1000 PL Network Adaptor (82573L)' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xdc300000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x3000, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) This is the sort of output we're looking for. > hw.bce.msi_enable: 1 > hw.pci.honor_msi_blacklist: 1 > hw.pci.enable_msix: 1 > hw.pci.enable_msi: 1 > > > igb0: flags=8943 metric > 0 mtu 1500 > options=13b > ether 00:1b:21:12:ec:38 > inet [snip] netmask 0xffffffc0 broadcast [snip] > media: Ethernet autoselect (1000baseT ) > status: active > > igb1: flags=8843 metric 0 mtu 1500 > options=13b > ether 00:1b:21:12:ec:39 > inet 10.0.0.252 netmask 0xffffff00 broadcast 10.0.0.255 > media: Ethernet autoselect (1000baseT ) > status: active > > > Here you go :) > > Note that the igb2 and 3 interfaces are unused, unplugged, unconfigured -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 17:55:53 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE795106566B; Thu, 27 Jan 2011 17:55:53 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 52D388FC18; Thu, 27 Jan 2011 17:55:52 +0000 (UTC) Received: by wwf26 with SMTP id 26so2279257wwf.31 for ; Thu, 27 Jan 2011 09:55:52 -0800 (PST) Received: by 10.227.128.21 with SMTP id i21mr1404334wbs.219.1296150952014; Thu, 27 Jan 2011 09:55:52 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id f35sm11983913wbf.14.2011.01.27.09.55.39 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 09:55:41 -0800 (PST) Message-ID: <4D41B197.6070308@my.gd> Date: Thu, 27 Jan 2011 18:55:35 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: "Vogel, Jack" References: <4D41417A.20904@my.gd> <20110127172724.GA36587@icarus.home.lan> <4D41ABF1.1010405@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> In-Reply-To: <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-stable@freebsd.org" , Jeremy Chadwick , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 17:55:54 -0000 On 1/27/11 6:41 PM, Vogel, Jack wrote: > Jeremy is right, if you have a problem the first step is to try the latest code. > > However, when I look at the interrupts below I don't see what the problem is? > The Broadcom seems to have about the same rate, it just doesn't have MSIX (multiple vectors). > > Jack > > My main concern is that the CPU %interrupt is quite high, also, we seem to be experiencing input errors on the interfaces. See for yourself the following munin graphs: http://my.gd/fw_graphs/ igb0 = WAN interf bce0 = LAN Obviously we've had quite a traffic increase since the beginning of the year, as shown by the PF statistics. But jeez, the CPU %interrupt doubled or tripled... You'll notice a drop in graphs between 23 and 25 january, this is when we switched the CARP master to the backup firewall. From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 18:10:02 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84030106566B for ; Thu, 27 Jan 2011 18:10:02 +0000 (UTC) (envelope-from jack.vogel@intel.com) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by mx1.freebsd.org (Postfix) with ESMTP id 5CDB98FC17 for ; Thu, 27 Jan 2011 18:10:02 +0000 (UTC) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP; 27 Jan 2011 09:41:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.60,386,1291622400"; d="scan'208";a="881583920" Received: from orsmsx603.amr.corp.intel.com ([10.22.226.49]) by fmsmga001.fm.intel.com with ESMTP; 27 Jan 2011 09:41:49 -0800 Received: from orsmsx601.amr.corp.intel.com (10.22.226.213) by orsmsx603.amr.corp.intel.com (10.22.226.49) with Microsoft SMTP Server (TLS) id 8.2.254.0; Thu, 27 Jan 2011 09:41:49 -0800 Received: from orsmsx508.amr.corp.intel.com ([10.22.226.46]) by orsmsx601.amr.corp.intel.com ([10.22.226.213]) with mapi; Thu, 27 Jan 2011 09:41:49 -0800 From: "Vogel, Jack" To: Damien Fleuriot , Jeremy Chadwick Date: Thu, 27 Jan 2011 09:41:48 -0800 Thread-Topic: High interrupt rate on a PF box + performance Thread-Index: Acu+SBzCXp13vjqUT1K7RqlhPscCTAAAJ4FA Message-ID: <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> References: <4D41417A.20904@my.gd> <20110127172724.GA36587@icarus.home.lan> <4D41ABF1.1010405@my.gd> In-Reply-To: <4D41ABF1.1010405@my.gd> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: RE: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 18:10:02 -0000 Jeremy is right, if you have a problem the first step is to try the latest = code. However, when I look at the interrupts below I don't see what the problem i= s? The Broadcom seems to have about the same rate, it just doesn't have MSIX (= multiple vectors). Jack -----Original Message----- From: Damien Fleuriot [mailto:ml@my.gd]=20 Sent: Thursday, January 27, 2011 9:31 AM To: Jeremy Chadwick Cc: freebsd-stable@freebsd.org; freebsd-pf@freebsd.org; Vogel, Jack Subject: Re: High interrupt rate on a PF box + performance On 1/27/11 6:27 PM, Jeremy Chadwick wrote: > On Thu, Jan 27, 2011 at 10:57:14AM +0100, Damien Fleuriot wrote: >> Hello list, >> >> I have a problem with interrupts, network cards, and PF performance. >> >> We have 2 firewalls running FreeBSD 8.0 for the current master and >> FreeBSD 8.1 for the backup host, which I upgraded just yesterday. >> >> [...] >> >> vmstat -i >> --- >> interrupt total rate >> irq14: ata0 36 0 >> irq16: mfi0 353244 1 >> irq21: uhci0 uhci+ 461504 1 >> cpu0: timer 615183815 1996 >> irq256: bce0 1015412475 3295 >> irq257: igb0 1067318584 3464 >> irq258: igb0 695648752 2258 >> irq259: igb0 2 0 >> irq260: igb1 11503857 37 >> irq261: igb1 506598 1 >> irq262: igb1 69 0 >> irq269: bce1 790820 2 >> cpu1: timer 615183757 1996 >> cpu2: timer 615197165 1996 >> cpu3: timer 615197165 1996 >> Total 5252757843 17050 >=20 > There are changes to the igb(4) driver which are in RELENG_8 (8-STABLE), > and some which will be in the upcoming 8.2-RELEASE, which may address > this. Jack Vogel of Intel would be able to confirm for sure; CC'ing him > here. >=20 > Could you please provide output from the following commands? >=20 > * pciconf -lvcb (only include igbX entries, thanks) > * sysctl -a | grep msi >=20 > Thanks. >=20 > I can't help with the CARP-related issues or other stuff you're > experiencing. These issues may all be separate problems, hard to say. >=20 igb0@pci0:14:0:0: class=3D0x020000 card=3D0x145a8086 chip=3D0x10d68086 rev=3D0x02 hdr=3D0x00 igb1@pci0:14:0:1: class=3D0x020000 card=3D0x145a8086 chip=3D0x10d68086 rev=3D0x02 hdr=3D0x00 igb2@pci0:15:0:0: class=3D0x020000 card=3D0x145a8086 chip=3D0x10d68086 rev=3D0x02 hdr=3D0x00 igb3@pci0:15:0:1: class=3D0x020000 card=3D0x145a8086 chip=3D0x10d68086 rev=3D0x02 hdr=3D0x00 hw.bce.msi_enable: 1 hw.pci.honor_msi_blacklist: 1 hw.pci.enable_msix: 1 hw.pci.enable_msi: 1 igb0: flags=3D8943 metric 0 mtu 1500 options=3D13b ether 00:1b:21:12:ec:38 inet [snip] netmask 0xffffffc0 broadcast [snip] media: Ethernet autoselect (1000baseT ) status: active igb1: flags=3D8843 metric 0 mtu 150= 0 options=3D13b ether 00:1b:21:12:ec:39 inet 10.0.0.252 netmask 0xffffff00 broadcast 10.0.0.255 media: Ethernet autoselect (1000baseT ) status: active Here you go :) Note that the igb2 and 3 interfaces are unused, unplugged, unconfigured From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 19:39:49 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C87F3106564A; Thu, 27 Jan 2011 19:39:49 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 4139D8FC0A; Thu, 27 Jan 2011 19:39:46 +0000 (UTC) Received: by wwf26 with SMTP id 26so2388332wwf.31 for ; Thu, 27 Jan 2011 11:39:46 -0800 (PST) Received: by 10.227.11.143 with SMTP id t15mr1612016wbt.27.1296157186115; Thu, 27 Jan 2011 11:39:46 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id u9sm1966078wbg.0.2011.01.27.11.39.42 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 11:39:43 -0800 (PST) Message-ID: <4D41C9FC.10503@my.gd> Date: Thu, 27 Jan 2011 20:39:40 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Sergey Lobanov References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> In-Reply-To: <201101280146.57028.wmn@siberianet.ru> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 19:39:49 -0000 On 1/27/11 7:46 PM, Sergey Lobanov wrote: > В сообщении от Пятница 28 января 2011 00:55:35 автор Damien Fleuriot написал: >> On 1/27/11 6:41 PM, Vogel, Jack wrote: >>> Jeremy is right, if you have a problem the first step is to try the >>> latest code. >>> >>> However, when I look at the interrupts below I don't see what the problem >>> is? The Broadcom seems to have about the same rate, it just doesn't have >>> MSIX (multiple vectors). >>> >>> Jack >> >> My main concern is that the CPU %interrupt is quite high, also, we seem >> to be experiencing input errors on the interfaces. > Would you show igb tuning which is done in loader.conf and output of sysctl > dev.igb.0? > Did you rise number of igb descriptors such as: > hw.igb.rxd=4096 > hw.igb.txd=4096 ? There is no tuning at all on our part in the loader's conf. Find below the sysctls: # sysctl -a |grep igb dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 dev.igb.0.%driver: igb dev.igb.0.%location: slot=0 function=0 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 subdevice=0x145a class=0x020000 dev.igb.0.%parent: pci14 dev.igb.0.debug: -1 dev.igb.0.stats: -1 dev.igb.0.flow_control: 3 dev.igb.0.enable_aim: 1 dev.igb.0.low_latency: 128 dev.igb.0.ave_latency: 450 dev.igb.0.bulk_latency: 1200 dev.igb.0.rx_processing_limit: 100 dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 dev.igb.1.%driver: igb dev.igb.1.%location: slot=0 function=1 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 subdevice=0x145a class=0x020000 dev.igb.1.%parent: pci14 dev.igb.1.debug: -1 dev.igb.1.stats: -1 dev.igb.1.flow_control: 3 dev.igb.1.enable_aim: 1 dev.igb.1.low_latency: 128 dev.igb.1.ave_latency: 450 dev.igb.1.bulk_latency: 1200 dev.igb.1.rx_processing_limit: 100 From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 19:57:44 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 933601065670 for ; Thu, 27 Jan 2011 19:57:44 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta05.emeryville.ca.mail.comcast.net (qmta05.emeryville.ca.mail.comcast.net [76.96.30.48]) by mx1.freebsd.org (Postfix) with ESMTP id 718968FC19 for ; Thu, 27 Jan 2011 19:57:43 +0000 (UTC) Received: from omta04.emeryville.ca.mail.comcast.net ([76.96.30.35]) by qmta05.emeryville.ca.mail.comcast.net with comcast id 0jvo1g0010lTkoCA5jxjk7; Thu, 27 Jan 2011 19:57:43 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta04.emeryville.ca.mail.comcast.net with comcast id 0jxi1g0062tehsa8Qjxi5v; Thu, 27 Jan 2011 19:57:43 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id CD2A09B422; Thu, 27 Jan 2011 11:57:41 -0800 (PST) Date: Thu, 27 Jan 2011 11:57:41 -0800 From: Jeremy Chadwick To: Damien Fleuriot Message-ID: <20110127195741.GA40449@icarus.home.lan> References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <4D41C9FC.10503@my.gd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 19:57:44 -0000 On Thu, Jan 27, 2011 at 08:39:40PM +0100, Damien Fleuriot wrote: > > > On 1/27/11 7:46 PM, Sergey Lobanov wrote: > > В сообщении от Пятница 28 января 2011 00:55:35 автор Damien Fleuriot написал: > >> On 1/27/11 6:41 PM, Vogel, Jack wrote: > >>> Jeremy is right, if you have a problem the first step is to try the > >>> latest code. > >>> > >>> However, when I look at the interrupts below I don't see what the problem > >>> is? The Broadcom seems to have about the same rate, it just doesn't have > >>> MSIX (multiple vectors). > >>> > >>> Jack > >> > >> My main concern is that the CPU %interrupt is quite high, also, we seem > >> to be experiencing input errors on the interfaces. > > Would you show igb tuning which is done in loader.conf and output of sysctl > > dev.igb.0? > > Did you rise number of igb descriptors such as: > > hw.igb.rxd=4096 > > hw.igb.txd=4096 ? > > There is no tuning at all on our part in the loader's conf. > > Find below the sysctls: > > # sysctl -a |grep igb > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 > subdevice=0x145a class=0x020000 > dev.igb.0.%parent: pci14 > dev.igb.0.debug: -1 > dev.igb.0.stats: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.low_latency: 128 > dev.igb.0.ave_latency: 450 > dev.igb.0.bulk_latency: 1200 > dev.igb.0.rx_processing_limit: 100 > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 > subdevice=0x145a class=0x020000 > dev.igb.1.%parent: pci14 > dev.igb.1.debug: -1 > dev.igb.1.stats: -1 > dev.igb.1.flow_control: 3 > dev.igb.1.enable_aim: 1 > dev.igb.1.low_latency: 128 > dev.igb.1.ave_latency: 450 > dev.igb.1.bulk_latency: 1200 > dev.igb.1.rx_processing_limit: 100 I'm not aware of how to tune igb(4), so the advice Sergey gave you may be applicable. You'll need to schedule downtime to adjust those tunables however (since a reboot will be requried). I also reviewed the munin graphs. I don't see anything necessarily wrong. However, you omitted yearly graphs for the network interfaces. Why I care about that: The pf state table (yearly) graph basically correlates with the CPU usage (yearly) graph, and I expect that the yearly network graphs would show a similar trend: an increase in your overall traffic over the course of a year. What I'm trying to figure out is what you're concerned about. You are in fact pushing anywhere between 60-120MBytes/sec across these interfaces. Given those numbers, I'm not surprised by the ""high"" interrupt usage. Graphs of this nature usually indicate that you're hitting a "bottleneck" (for lack of better word) where you're simply doing "too much" with a single machine (given its network throughput). The machine is spending a tremendous amount of CPU time handling network traffic, and equally as much with regards to the pf usage. If you want my opinion based on the information I have so far, it's this: you need to scale your infrastructure. You can no longer rely on a single machine to handle this amount of traffic. As for the network errors you see -- to get low-level NIC and driver statistics, you'll need to run "sysctl dev.igb.X.stats=1" then run "dmesg" and look at the numbers shown (the sysctl command won't output anything itself). This may help indicate where the packets are being lost. You should also check the interface counters on the switch which these interfaces are connected to. I sure hope it's a managed switch which can give you those statistics. Hope this helps, or at least acts as food for thought. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 20:29:21 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EEF21065670 for ; Thu, 27 Jan 2011 20:29:21 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id D67368FC19 for ; Thu, 27 Jan 2011 20:29:20 +0000 (UTC) Received: by yie19 with SMTP id 19so807336yie.13 for ; Thu, 27 Jan 2011 12:29:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=pGeSt05Fm5mPVvgFYr5VNoHugJlyM+K8wOkLiefnLCs=; b=HH5wfaqQYg8qNgTignRT/wE5LAACJ+O4OZ4hjec+J1kZaqyXp7XELyoo+ENCM5VSXR WZWCvIWVs//dTlNMeEowVK6CyB69yYiBes0DQsoXGmSBsT7SOQOlPrykAOW9wNjFaoYB lFLWMcHTghEhxKHWIX85iOhqKxP2WN3dkBdTg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=xAEfzd//q+dPNZLx3hqnxWEivdJCV1jtbRfG+f7cXxOFZN66srwbFnDO9qxPoE5nXx sGvGETFUEeQejyDOCmtQc2wAPh7FWVJz/U2l1y7RudDlTa10KKSCn+mS2AF4QUJw9Wx8 PozhWhRbw6VUawzRp77O44SChLnvV1aArsYNQ= MIME-Version: 1.0 Received: by 10.236.108.177 with SMTP id q37mr3091801yhg.11.1296158538636; Thu, 27 Jan 2011 12:02:18 -0800 (PST) Received: by 10.147.171.17 with HTTP; Thu, 27 Jan 2011 12:02:18 -0800 (PST) In-Reply-To: <20110127195741.GA40449@icarus.home.lan> References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> Date: Thu, 27 Jan 2011 12:02:18 -0800 Message-ID: From: Jack Vogel To: Jeremy Chadwick Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 20:29:21 -0000 If you go to 8.2 and the latest driver you will get better stats also, ahem... Jack On Thu, Jan 27, 2011 at 11:57 AM, Jeremy Chadwick wrote: > On Thu, Jan 27, 2011 at 08:39:40PM +0100, Damien Fleuriot wrote: > > > > > > On 1/27/11 7:46 PM, Sergey Lobanov wrote: > > > =F7 =D3=CF=CF=C2=DD=C5=CE=C9=C9 =CF=D4 =F0=D1=D4=CE=C9=C3=C1 28 =D1= =CE=D7=C1=D2=D1 2011 00:55:35 =C1=D7=D4=CF=D2 Damien Fleuriot > =CE=C1=D0=C9=D3=C1=CC: > > >> On 1/27/11 6:41 PM, Vogel, Jack wrote: > > >>> Jeremy is right, if you have a problem the first step is to try the > > >>> latest code. > > >>> > > >>> However, when I look at the interrupts below I don't see what the > problem > > >>> is? The Broadcom seems to have about the same rate, it just doesn't > have > > >>> MSIX (multiple vectors). > > >>> > > >>> Jack > > >> > > >> My main concern is that the CPU %interrupt is quite high, also, we > seem > > >> to be experiencing input errors on the interfaces. > > > Would you show igb tuning which is done in loader.conf and output of > sysctl > > > dev.igb.0? > > > Did you rise number of igb descriptors such as: > > > hw.igb.rxd=3D4096 > > > hw.igb.txd=3D4096 ? > > > > There is no tuning at all on our part in the loader's conf. > > > > Find below the sysctls: > > > > # sysctl -a |grep igb > > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 > > dev.igb.0.%driver: igb > > dev.igb.0.%location: slot=3D0 function=3D0 > > dev.igb.0.%pnpinfo: vendor=3D0x8086 device=3D0x10d6 subvendor=3D0x8086 > > subdevice=3D0x145a class=3D0x020000 > > dev.igb.0.%parent: pci14 > > dev.igb.0.debug: -1 > > dev.igb.0.stats: -1 > > dev.igb.0.flow_control: 3 > > dev.igb.0.enable_aim: 1 > > dev.igb.0.low_latency: 128 > > dev.igb.0.ave_latency: 450 > > dev.igb.0.bulk_latency: 1200 > > dev.igb.0.rx_processing_limit: 100 > > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 > > dev.igb.1.%driver: igb > > dev.igb.1.%location: slot=3D0 function=3D1 > > dev.igb.1.%pnpinfo: vendor=3D0x8086 device=3D0x10d6 subvendor=3D0x8086 > > subdevice=3D0x145a class=3D0x020000 > > dev.igb.1.%parent: pci14 > > dev.igb.1.debug: -1 > > dev.igb.1.stats: -1 > > dev.igb.1.flow_control: 3 > > dev.igb.1.enable_aim: 1 > > dev.igb.1.low_latency: 128 > > dev.igb.1.ave_latency: 450 > > dev.igb.1.bulk_latency: 1200 > > dev.igb.1.rx_processing_limit: 100 > > I'm not aware of how to tune igb(4), so the advice Sergey gave you may > be applicable. You'll need to schedule downtime to adjust those > tunables however (since a reboot will be requried). > > I also reviewed the munin graphs. I don't see anything necessarily > wrong. However, you omitted yearly graphs for the network interfaces. > Why I care about that: > > The pf state table (yearly) graph basically correlates with the CPU > usage (yearly) graph, and I expect that the yearly network graphs would > show a similar trend: an increase in your overall traffic over the > course of a year. > > What I'm trying to figure out is what you're concerned about. You are > in fact pushing anywhere between 60-120MBytes/sec across these > interfaces. Given those numbers, I'm not surprised by the ""high"" > interrupt usage. > > Graphs of this nature usually indicate that you're hitting a > "bottleneck" (for lack of better word) where you're simply doing "too > much" with a single machine (given its network throughput). The machine > is spending a tremendous amount of CPU time handling network traffic, > and equally as much with regards to the pf usage. > > If you want my opinion based on the information I have so far, it's > this: you need to scale your infrastructure. You can no longer rely on > a single machine to handle this amount of traffic. > > As for the network errors you see -- to get low-level NIC and driver > statistics, you'll need to run "sysctl dev.igb.X.stats=3D1" then run > "dmesg" and look at the numbers shown (the sysctl command won't output > anything itself). This may help indicate where the packets are being > lost. You should also check the interface counters on the switch which > these interfaces are connected to. I sure hope it's a managed switch > which can give you those statistics. > > Hope this helps, or at least acts as food for thought. > > -- > | Jeremy Chadwick jdc@parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP 4BD6C0CB | > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 20:38:25 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3B751065670; Thu, 27 Jan 2011 20:38:25 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 24BF08FC0C; Thu, 27 Jan 2011 20:38:24 +0000 (UTC) Received: by wwf26 with SMTP id 26so2444071wwf.31 for ; Thu, 27 Jan 2011 12:38:24 -0800 (PST) Received: by 10.227.144.12 with SMTP id x12mr1662478wbu.102.1296160703918; Thu, 27 Jan 2011 12:38:23 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id o6sm441232wbo.15.2011.01.27.12.38.22 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 12:38:23 -0800 (PST) Message-ID: <4D41D7BE.3030208@my.gd> Date: Thu, 27 Jan 2011 21:38:22 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> In-Reply-To: <20110127195741.GA40449@icarus.home.lan> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 20:38:25 -0000 On 1/27/11 8:57 PM, Jeremy Chadwick wrote: > On Thu, Jan 27, 2011 at 08:39:40PM +0100, Damien Fleuriot wrote: >> >> >> On 1/27/11 7:46 PM, Sergey Lobanov wrote: >>> В сообщении от Пятница 28 января 2011 00:55:35 автор Damien Fleuriot написал: >>>> On 1/27/11 6:41 PM, Vogel, Jack wrote: >>>>> Jeremy is right, if you have a problem the first step is to try the >>>>> latest code. >>>>> >>>>> However, when I look at the interrupts below I don't see what the problem >>>>> is? The Broadcom seems to have about the same rate, it just doesn't have >>>>> MSIX (multiple vectors). >>>>> >>>>> Jack >>>> >>>> My main concern is that the CPU %interrupt is quite high, also, we seem >>>> to be experiencing input errors on the interfaces. >>> Would you show igb tuning which is done in loader.conf and output of sysctl >>> dev.igb.0? >>> Did you rise number of igb descriptors such as: >>> hw.igb.rxd=4096 >>> hw.igb.txd=4096 ? >> >> There is no tuning at all on our part in the loader's conf. >> >> Find below the sysctls: >> >> # sysctl -a |grep igb >> dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 >> dev.igb.0.%driver: igb >> dev.igb.0.%location: slot=0 function=0 >> dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 >> subdevice=0x145a class=0x020000 >> dev.igb.0.%parent: pci14 >> dev.igb.0.debug: -1 >> dev.igb.0.stats: -1 >> dev.igb.0.flow_control: 3 >> dev.igb.0.enable_aim: 1 >> dev.igb.0.low_latency: 128 >> dev.igb.0.ave_latency: 450 >> dev.igb.0.bulk_latency: 1200 >> dev.igb.0.rx_processing_limit: 100 >> dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 >> dev.igb.1.%driver: igb >> dev.igb.1.%location: slot=0 function=1 >> dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 >> subdevice=0x145a class=0x020000 >> dev.igb.1.%parent: pci14 >> dev.igb.1.debug: -1 >> dev.igb.1.stats: -1 >> dev.igb.1.flow_control: 3 >> dev.igb.1.enable_aim: 1 >> dev.igb.1.low_latency: 128 >> dev.igb.1.ave_latency: 450 >> dev.igb.1.bulk_latency: 1200 >> dev.igb.1.rx_processing_limit: 100 > > I'm not aware of how to tune igb(4), so the advice Sergey gave you may > be applicable. You'll need to schedule downtime to adjust those > tunables however (since a reboot will be requried). > > I also reviewed the munin graphs. I don't see anything necessarily > wrong. However, you omitted yearly graphs for the network interfaces. Indeed I have, the reason is because the yearly graphs are fucked up, for some reason that eludes me munin recorded a 2petabyte spike sometime during september or so. So this makes the whole graph flatlined for the year -.- However, we clearly have an increase in traffic, as we may also see from our nginx requests graphs. > Why I care about that: > > The pf state table (yearly) graph basically correlates with the CPU > usage (yearly) graph, and I expect that the yearly network graphs would > show a similar trend: an increase in your overall traffic over the > course of a year. > > What I'm trying to figure out is what you're concerned about. You are > in fact pushing anywhere between 60-120MBytes/sec across these > interfaces. Given those numbers, I'm not surprised by the ""high"" > interrupt usage. > I'm worried we may hit a bottleneck soon. I was also hoping for some kind of magical way to diminish the interrupts so we could get more performance from the machines. > Graphs of this nature usually indicate that you're hitting a > "bottleneck" (for lack of better word) where you're simply doing "too > much" with a single machine (given its network throughput). The machine > is spending a tremendous amount of CPU time handling network traffic, > and equally as much with regards to the pf usage. > We've indeed been thinking about moving to an active-active setup for some time already, guess it'll have to happen sooner rather than later :) > If you want my opinion based on the information I have so far, it's > this: you need to scale your infrastructure. You can no longer rely on > a single machine to handle this amount of traffic. > > As for the network errors you see -- to get low-level NIC and driver > statistics, you'll need to run "sysctl dev.igb.X.stats=1" then run > "dmesg" and look at the numbers shown (the sysctl command won't output > anything itself). This may help indicate where the packets are being > lost. You should also check the interface counters on the switch which > these interfaces are connected to. I sure hope it's a managed switch > which can give you those statistics. > > Hope this helps, or at least acts as food for thought. > Aye, will try that. We're also considering moving to faster machines but I don't think that will help much with our problem. I suppose additional CPU cores will be of no help at all, considering the kernel is single threaded and runs on cpu0 only ? Actually, I assume it might even be detrimental to us to add more cores, since they'll spend more time interrupting each other ? Thanks for sharing your thoughts :) From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 20:39:54 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0100010656C0; Thu, 27 Jan 2011 20:39:54 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6A87E8FC29; Thu, 27 Jan 2011 20:39:50 +0000 (UTC) Received: by wyf19 with SMTP id 19so2525180wyf.13 for ; Thu, 27 Jan 2011 12:39:50 -0800 (PST) Received: by 10.227.134.206 with SMTP id k14mr1733548wbt.5.1296160790088; Thu, 27 Jan 2011 12:39:50 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id w25sm1044533wbd.5.2011.01.27.12.39.49 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 12:39:49 -0800 (PST) Message-ID: <4D41D814.90305@my.gd> Date: Thu, 27 Jan 2011 21:39:48 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jack Vogel References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , Jeremy Chadwick , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 20:39:54 -0000 On 1/27/11 9:02 PM, Jack Vogel wrote: > If you go to 8.2 and the latest driver you will get better stats also, > ahem... > > Jack > We'll be doing that as soon as 8.2 hits release, as opposed to prerelease/rc. Can never be too careful with this one project, outtages would be costly -.- From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 20:58:47 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 59854106564A for ; Thu, 27 Jan 2011 20:58:47 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.emeryville.ca.mail.comcast.net (qmta07.emeryville.ca.mail.comcast.net [76.96.30.64]) by mx1.freebsd.org (Postfix) with ESMTP id 379D48FC08 for ; Thu, 27 Jan 2011 20:58:47 +0000 (UTC) Received: from omta10.emeryville.ca.mail.comcast.net ([76.96.30.28]) by qmta07.emeryville.ca.mail.comcast.net with comcast id 0ks11g0040cQ2SLA7kymnH; Thu, 27 Jan 2011 20:58:46 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta10.emeryville.ca.mail.comcast.net with comcast id 0kyl1g00d2tehsa8WkylyL; Thu, 27 Jan 2011 20:58:46 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 44AB09B422; Thu, 27 Jan 2011 12:58:45 -0800 (PST) Date: Thu, 27 Jan 2011 12:58:45 -0800 From: Jeremy Chadwick To: Damien Fleuriot Message-ID: <20110127205845.GA41537@icarus.home.lan> References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> <4D41D7BE.3030208@my.gd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D41D7BE.3030208@my.gd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 20:58:47 -0000 On Thu, Jan 27, 2011 at 09:38:22PM +0100, Damien Fleuriot wrote: > On 1/27/11 8:57 PM, Jeremy Chadwick wrote: > <...snipping out stuff...> > We're also considering moving to faster machines but I don't think that > will help much with our problem. > > I suppose additional CPU cores will be of no help at all, considering > the kernel is single threaded and runs on cpu0 only ? Kernel folks should be able to talk about this in detail, but my understanding is that the kernel itself supports multiple threads, but the question is whether or not the drivers or relevant "pieces" (e.g. igb(4) driver, pf, TCP stack, etc.) support SMP (multi-core/threading) or not. I think this is referred to as something being "MPSAFE" or not. The things you see during boot -- [ITHREAD], [FILTER], and [GIANT-LOCKED] play a role as well, but I think those indicate what style of locking is used (since some drivers/features might not work properly in a multiprocessor environment). I'm trying to avoid correlating "multiprocessor safe" with "makes use of multiple processors". I'm an old 65xxx CPU guy, this SMP stuff is still "new technology" to me when it comes to actual operations/mechanics. Regarding TCP and SMP, this is regularly touched on in the FreeBSD Status Reports that go out (always worth reading). See "TCP SMP scalability project": http://www.freebsd.org/news/status/report-2010-10-2010-12.html I know all this information is technical of course and doesn't answer your question directly. I wish there was something more authoritative when it came to this question. > Actually, I assume it might even be detrimental to us to add more cores, > since they'll spend more time interrupting each other ? > > Thanks for sharing your thoughts :) -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Thu Jan 27 21:44:15 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F20531065670; Thu, 27 Jan 2011 21:44:15 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9555C8FC14; Thu, 27 Jan 2011 21:44:15 +0000 (UTC) Received: by gxk8 with SMTP id 8so866560gxk.13 for ; Thu, 27 Jan 2011 13:44:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=v9F2tGocRlB8Nz3Py3RzDC7SNAJqwFYdvTlJ3U5vm8Q=; b=W+hZEEFw3vdqULowdQindVfqPVNpxLVT/3JeRCH1WxAaA3VnG6xnyq2+vJTTigqEYf vy9sqb4XP+4Yz6C6DmAlu7ye9ohNidR/6TNToIKgGkgVs/c/vieCOFxWgH4fz7S8C0Ud 1oiG/CDiYDNXrm6URcUsY+vHDn9C5gOUXILSI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=dXwWKi6/Ec7oq2H4Zmm7fwK+Ov6y9e1uuG1Osyqtr/GpvQIBB7op5WPNt18rV6BpRm b+4pHERg9pDST3Rc3NQC3QsYpnpO2v+SHFz/MWqoQngPxvCZSJ/qXBBEXH3yRkVlxIK8 wdjPdowJPRbg5roD8HLeOHljSKzQuMjovwJp0= MIME-Version: 1.0 Received: by 10.151.46.10 with SMTP id y10mr3432226ybj.22.1296164654437; Thu, 27 Jan 2011 13:44:14 -0800 (PST) Received: by 10.147.171.17 with HTTP; Thu, 27 Jan 2011 13:44:14 -0800 (PST) In-Reply-To: <20110127205845.GA41537@icarus.home.lan> References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> <4D41D7BE.3030208@my.gd> <20110127205845.GA41537@icarus.home.lan> Date: Thu, 27 Jan 2011 13:44:14 -0800 Message-ID: From: Jack Vogel To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 21:44:16 -0000 On Thu, Jan 27, 2011 at 12:58 PM, Jeremy Chadwick wrote: > > On Thu, Jan 27, 2011 at 09:38:22PM +0100, Damien Fleuriot wrote: > > On 1/27/11 8:57 PM, Jeremy Chadwick wrote: > > > > <...snipping out stuff...> > > > We're also considering moving to faster machines but I don't think that > > will help much with our problem. > > > > I suppose additional CPU cores will be of no help at all, considering > > the kernel is single threaded and runs on cpu0 only ? > > Kernel folks should be able to talk about this in detail, but my > understanding is that the kernel itself supports multiple threads, but > the question is whether or not the drivers or relevant "pieces" (e.g. > igb(4) driver, pf, TCP stack, etc.) support SMP (multi-core/threading) > or not. I think this is referred to as something being "MPSAFE" or not. > > The 8.X kernel is NOT single-threaded. Anything but. And the stack has also been improved, I believe there are still bottlenecks but its far better than the old days. The igb driver in 8.2 creates up to 8 queues on the right hardware, they are each auto-bound to a particular CPU. The older version you are running had issues and hence multiqueue was not enabled. So, do upgrade once 8.2 is finalized :) Cheers, Jack From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 09:06:03 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D28D1065674 for ; Fri, 28 Jan 2011 09:06:03 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Received: from mail.time-domain.co.uk (81-179-248-237.static.dsl.pipex.com [81.179.248.237]) by mx1.freebsd.org (Postfix) with ESMTP id B5E088FC0C for ; Fri, 28 Jan 2011 09:06:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.time-domain.co.uk (8.14.3/8.14.3) with ESMTP id p0S8nRnn021163 for ; Fri, 28 Jan 2011 08:49:27 GMT Date: Fri, 28 Jan 2011 08:49:27 +0000 (GMT) From: andy thomas X-X-Sender: andy-tds@mail.time-domain.co.uk To: freebsd-pf@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: clamav-milter 0.96.5 at mail X-Virus-Status: Clean Subject: PF port forward problem with Sonicwall VPN X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 09:06:03 -0000 I'm maintaining some OpenBSD-based firewalls and have been really stumped with a problem when trying to add a Sonicwall VPN appliance behind the firewall, and thought I'd ask here for help. The Sonicwall device uses SSL on port 443 for it's external VPN traffic and listens on other ports for internal LAN traffic and it uses a single network interface for this. On our installation, there is a webmail server behind the firewall listening on port 443 and the existing PF rule for this is (abbreviated for clarity): ext_if="vr0" int_if="vr1" webmail="192.168.30.14" rdr pass log on $ext_if proto tcp from any to $ext_if port 443 -> $webmail port 443 This works fine so as external port 443 is already in use for webmail, I decided to use external port 444 for the Sonicwall and added these two extra rules: sonicwall="192.168.30.28" rdr pass log on $ext_if proto tcp from any to $ext_if port 444 -> $sonicwall port 443 However, the Sonicwall cannot be accessed from the external port 444 although it can be accessed internall on port 443 of course. I have tested this rule by changing it to point to the webmail server like this: rdr pass log on $ext_if proto tcp from any to $ext_if port 444 -> $webmail port 443 and this works fine as I can access webmail on port 444. But why can't I access the Sonicwall on port 444? Does anyone know if the Sonicwall uses additional ports or has anyone got this device to with with a PF-based firewall? Thanks in advance for any suggestions, Andy From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 09:13:04 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F2DE1065694 for ; Fri, 28 Jan 2011 09:13:04 +0000 (UTC) (envelope-from artem@aws-net.org.ua) Received: from lazy.aws-net.org.ua (lazy.aws-net.org.ua [IPv6:2a00:1db0:20::828:140]) by mx1.freebsd.org (Postfix) with ESMTP id E1A7B8FC19 for ; Fri, 28 Jan 2011 09:13:03 +0000 (UTC) Received: from rainbow.vl.net.ua (rainbow.vl.net.ua [188.230.120.215]) (authenticated bits=0) by lazy.aws-net.org.ua (8.14.3/8.14.3) with ESMTP id p0S9CtRl006916 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=OK); Fri, 28 Jan 2011 11:13:02 +0200 (EET) (envelope-from artem@aws-net.org.ua) Message-ID: <4D428897.4030505@aws-net.org.ua> Date: Fri, 28 Jan 2011 11:12:55 +0200 From: Artyom Viklenko Organization: Art&Co. User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.11) Gecko/20101025 Thunderbird/3.1.5 MIME-Version: 1.0 To: andy thomas References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.5 (lazy.aws-net.org.ua [188.230.120.140]); Fri, 28 Jan 2011 11:13:02 +0200 (EET) Cc: freebsd-pf@freebsd.org Subject: Re: PF port forward problem with Sonicwall VPN X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 09:13:04 -0000 28.01.2011 10:49, andy thomas пишет: > I'm maintaining some OpenBSD-based firewalls and have been really > stumped with a problem when trying to add a Sonicwall VPN appliance > behind the firewall, and thought I'd ask here for help. > > The Sonicwall device uses SSL on port 443 for it's external VPN traffic > and listens on other ports for internal LAN traffic and it uses a single > network interface for this. On our installation, there is a webmail > server behind the firewall listening on port 443 and the existing PF > rule for this is (abbreviated for clarity): > > ext_if="vr0" > int_if="vr1" > > webmail="192.168.30.14" > > rdr pass log on $ext_if proto tcp from any to $ext_if port 443 -> > $webmail port 443 > > This works fine so as external port 443 is already in use for webmail, I > decided to use external port 444 for the Sonicwall and added these two > extra rules: > > sonicwall="192.168.30.28" > > rdr pass log on $ext_if proto tcp from any to $ext_if port 444 -> > $sonicwall port 443 > > However, the Sonicwall cannot be accessed from the external port 444 > although it can be accessed internall on port 443 of course. I have Check your filtering rules on internal interface, may be you have 'pass' for trafic to webmail host and doesn't for sonicwall? > tested this rule by changing it to point to the webmail server like this: > > rdr pass log on $ext_if proto tcp from any to $ext_if port 444 -> > $webmail port 443 > > and this works fine as I can access webmail on port 444. But why can't I > access the Sonicwall on port 444? Does anyone know if the Sonicwall uses > additional ports or has anyone got this device to with with a PF-based > firewall? > > Thanks in advance for any suggestions, > > Andy > _______________________________________________ > freebsd-pf@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" -- Sincerely yours, Artyom Viklenko. ------------------------------------------------------- artem@aws-net.org.ua | http://www.aws-net.org.ua/~artem artem@viklenko.net | JID: artem@jabber.aws-net.org.ua FreeBSD: The Power to Serve - http://www.freebsd.org From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 09:28:59 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E1E51065672 for ; Fri, 28 Jan 2011 09:28:59 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Received: from mail.time-domain.co.uk (81-179-248-237.static.dsl.pipex.com [81.179.248.237]) by mx1.freebsd.org (Postfix) with ESMTP id 984738FC20 for ; Fri, 28 Jan 2011 09:28:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.time-domain.co.uk (8.14.3/8.14.3) with ESMTP id p0S9SqqT021383; Fri, 28 Jan 2011 09:28:53 GMT Date: Fri, 28 Jan 2011 09:28:52 +0000 (GMT) From: andy thomas X-X-Sender: andy-tds@mail.time-domain.co.uk To: Artyom Viklenko In-Reply-To: <4D428897.4030505@aws-net.org.ua> Message-ID: References: <4D428897.4030505@aws-net.org.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: clamav-milter 0.96.5 at mail X-Virus-Status: Clean Cc: freebsd-pf@freebsd.org Subject: Re: PF port forward problem with Sonicwall VPN X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 09:28:59 -0000 On Fri, 28 Jan 2011, Artyom Viklenko wrote: > 28.01.2011 10:49, andy thomas : >> I'm maintaining some OpenBSD-based firewalls and have been really >> stumped with a problem when trying to add a Sonicwall VPN appliance >> behind the firewall, and thought I'd ask here for help. >> >> The Sonicwall device uses SSL on port 443 for it's external VPN traffic >> and listens on other ports for internal LAN traffic and it uses a single >> network interface for this. On our installation, there is a webmail >> server behind the firewall listening on port 443 and the existing PF >> rule for this is (abbreviated for clarity): >> >> ext_if="vr0" >> int_if="vr1" >> >> webmail="192.168.30.14" >> >> rdr pass log on $ext_if proto tcp from any to $ext_if port 443 -> >> $webmail port 443 >> >> This works fine so as external port 443 is already in use for webmail, I >> decided to use external port 444 for the Sonicwall and added these two >> extra rules: >> >> sonicwall="192.168.30.28" >> >> rdr pass log on $ext_if proto tcp from any to $ext_if port 444 -> >> $sonicwall port 443 >> >> However, the Sonicwall cannot be accessed from the external port 444 >> although it can be accessed internall on port 443 of course. I have > > Check your filtering rules on internal interface, may be you have 'pass' > for trafic to webmail host and doesn't for sonicwall? Thanks for the quick response - here are the existing internal interface rules: # int_if pass in on $int_if proto carp keep state pass out on vr1 proto carp keep state pass in on $int_if proto tcp from any to any flags S/SA keep state pass in on $int_if proto { udp, icmp } from any to any keep state pass out on $int_if all keep state label "int_net:$if:out" I should add this firewall also handles IPsec VPN with other rules using port 500 and the enc0 interface but this should not affect Sonicwall on port 443, should it? If it's of any help I can post the entire pf.conf but it's very long - I didn't write this and have only recently taken over responsibility for the firewalls along with the FreeBSD-based server infrastructure. (The IPsec VPN doesn't work very well with ShrewSoft VPN clients under Windows 7 so the company bought the Sonicwall as a solution for this problem although IPsec/Shrewsoft VPN works fine with machines running Windows XP). thanks, Andy From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 09:44:54 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23EE61065672 for ; Fri, 28 Jan 2011 09:44:54 +0000 (UTC) (envelope-from mlmichael70@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id ADBCB8FC17 for ; Fri, 28 Jan 2011 09:44:53 +0000 (UTC) Received: by wyf19 with SMTP id 19so3069445wyf.13 for ; Fri, 28 Jan 2011 01:44:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:message-id:date:from:user-agent:mime-version:to :subject:content-type:content-transfer-encoding; bh=AkZ4Nw9ybk3/onaDVoOxtMG60v+Y8o33qUikCzFqG9U=; b=VZf0UCpyv+SelzomlRxJ3i3n0GLZLCVIgLLMCSz556lUi30qYyJHsk/ivDaN6XGMi8 IpgyH/RLZO8b6ZP782CByYzI7s07dbSfN991diJm/5SQKkvYt3W5Na5Eysaoyvi+NAbU 6dNd43BnyU8bnZvdb7KAfthTNHZFAOenQukFE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; b=WgHuZSuffoLBiPDlRluo3bJqsT4lGfOaqe9irEGRwTfurGt7AZJva2T3Mjv83bxpsx zheKT2CdQR+OykuMpCyHcbgd6lirdDAa6opZsNuO7S+xGdyjdBpoQL4L+G0AzjNcn8bo JDX2r9Rd8u/5KumQYKkuRsOPQMczq/psLnRyo= Received: by 10.227.138.15 with SMTP id y15mr2306202wbt.186.1296206386164; Fri, 28 Jan 2011 01:19:46 -0800 (PST) Received: from prime.nonspace ([82.132.211.49]) by mx.google.com with ESMTPS id f35sm12522128wbf.8.2011.01.28.01.19.45 (version=SSLv3 cipher=RC4-MD5); Fri, 28 Jan 2011 01:19:45 -0800 (PST) Message-ID: <4D428A38.8000609@gmail.com> Date: Fri, 28 Jan 2011 09:19:52 +0000 From: Michael User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.13) Gecko/20101215 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-pf@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: why "block quick on wlan0" doesn't stop DHCP? X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 09:44:54 -0000 Hello, Here is my simple rule set: set loginterface wlan0 block log block quick on wlan0 Now I'm booting my 8.1-R box. After it's up and running with pf I'm powering on my wireless access point. After couple seconds my wlan0 is associated and receives it's IP address. I don't understand why was it not stopped by pf? And how can I tune my rules to be able to control DHCP conversation? Michael From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 09:58:20 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB59E106566B for ; Fri, 28 Jan 2011 09:58:20 +0000 (UTC) (envelope-from Greg.Hennessy@nviz.net) Received: from mail2.jellyfishnet.co.uk (mail2.jellyfishnet.co.uk [93.91.20.10]) by mx1.freebsd.org (Postfix) with ESMTP id 6F2ED8FC1A for ; Fri, 28 Jan 2011 09:58:20 +0000 (UTC) Received: from pemexhub01.jellyfishnet.co.uk.local (93.91.20.2) by mail2.jellyfishnet.co.uk (93.91.20.10) with Microsoft SMTP Server (TLS) id 8.1.436.0; Fri, 28 Jan 2011 09:47:26 +0000 Received: from PEMEXMBXVS02.jellyfishnet.co.uk.local ([192.168.65.37]) by pemexhub01.jellyfishnet.co.uk.local ([192.168.65.7]) with mapi; Fri, 28 Jan 2011 09:47:26 +0000 From: Greg Hennessy To: "freebsd-pf@freebsd.org" Date: Fri, 28 Jan 2011 09:47:23 +0000 Thread-Topic: why "block quick on wlan0" doesn't stop DHCP? Thread-Index: Acu+0BFhQgC+yZMBShSo+P8FT0JxZAAAA/9w Message-ID: <9E8D76EC267C9444AC737F649CBBAD9027BC4023C4@PEMEXMBXVS02.jellyfishnet.co.uk.local> References: <4D428A38.8000609@gmail.com> In-Reply-To: <4D428A38.8000609@gmail.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Subject: RE: why "block quick on wlan0" doesn't stop DHCP? X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 09:58:20 -0000 Q291bGQgYmUgdGFsa2luZyBjb21wbGV0ZSBub25zZW5zZSBoZXJlLCBidXQuLi4uDQoNCklJUkMg QlBGIHNlZXMgYWxsIHRyYWZmaWMgYmVmb3JlIFBGLiBESENQIGhvb2tzIGF0IHRoZSBCUEYgbGF5 ZXIsIHNvIGl0J2xsIGJlIHNlcnZpY2VkIGJlZm9yZSBhbnkgZmlsdGVyaW5nIHBvbGljeSBhcHBs aWVzLiANCg0KDQpHcmVnDQogDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJv bTogb3duZXItZnJlZWJzZC1wZkBmcmVlYnNkLm9yZyBbbWFpbHRvOm93bmVyLWZyZWVic2QtDQo+ IHBmQGZyZWVic2Qub3JnXSBPbiBCZWhhbGYgT2YgTWljaGFlbA0KPiBTZW50OiAyOCBKYW51YXJ5 IDIwMTEgOToyMCBBTQ0KPiBUbzogZnJlZWJzZC1wZkBmcmVlYnNkLm9yZw0KPiBTdWJqZWN0OiB3 aHkgImJsb2NrIHF1aWNrIG9uIHdsYW4wIiBkb2Vzbid0IHN0b3AgREhDUD8NCj4gDQo+IEhlbGxv LA0KPiANCj4gSGVyZSBpcyBteSBzaW1wbGUgcnVsZSBzZXQ6DQo+IA0KPiBzZXQgbG9naW50ZXJm YWNlIHdsYW4wDQo+IGJsb2NrIGxvZw0KPiBibG9jayBxdWljayBvbiB3bGFuMA0KPiANCj4gTm93 IEknbSBib290aW5nIG15IDguMS1SIGJveC4gQWZ0ZXIgaXQncyB1cCBhbmQgcnVubmluZyB3aXRo IHBmIEknbQ0KPiBwb3dlcmluZyBvbiBteSB3aXJlbGVzcyBhY2Nlc3MgcG9pbnQuDQo+IA0KPiBB ZnRlciBjb3VwbGUgc2Vjb25kcyBteSB3bGFuMCBpcyBhc3NvY2lhdGVkIGFuZCByZWNlaXZlcyBp dCdzIElQDQo+IGFkZHJlc3MuIEkgZG9uJ3QgdW5kZXJzdGFuZCB3aHkgd2FzIGl0IG5vdCBzdG9w cGVkIGJ5IHBmPw0KPiBBbmQgaG93IGNhbiBJIHR1bmUgbXkgcnVsZXMgdG8gYmUgYWJsZSB0byBj b250cm9sIERIQ1AgY29udmVyc2F0aW9uPw0KPiANCj4gTWljaGFlbA0KPiBfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBmcmVlYnNkLXBmQGZyZWVic2Qu b3JnIG1haWxpbmcgbGlzdA0KPiBodHRwOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0 aW5mby9mcmVlYnNkLXBmDQo+IFRvIHVuc3Vic2NyaWJlLCBzZW5kIGFueSBtYWlsIHRvICJmcmVl YnNkLXBmLXVuc3Vic2NyaWJlQGZyZWVic2Qub3JnIg0K From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 10:29:58 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48940106564A; Fri, 28 Jan 2011 10:29:58 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 72DDF8FC15; Fri, 28 Jan 2011 10:29:55 +0000 (UTC) Received: by fxm16 with SMTP id 16so3295231fxm.13 for ; Fri, 28 Jan 2011 02:29:54 -0800 (PST) Received: by 10.223.70.142 with SMTP id d14mr2163596faj.110.1296210594389; Fri, 28 Jan 2011 02:29:54 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id 11sm1556061faw.44.2011.01.28.02.29.52 (version=SSLv3 cipher=RC4-MD5); Fri, 28 Jan 2011 02:29:52 -0800 (PST) Message-ID: <4D429A9F.8040307@my.gd> Date: Fri, 28 Jan 2011 11:29:51 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jack Vogel References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> <4D41D7BE.3030208@my.gd> <20110127205845.GA41537@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , Jeremy Chadwick , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 10:29:58 -0000 On 1/27/11 10:44 PM, Jack Vogel wrote: > > The 8.X kernel is NOT single-threaded. Anything but. And the stack has > also been improved, I believe there are still bottlenecks but its far better > than the old days. > > The igb driver in 8.2 creates up to 8 queues on the right hardware, they > are each auto-bound to a particular CPU. > > The older version you are running had issues and hence multiqueue was > not enabled. So, do upgrade once 8.2 is finalized :) > > Cheers, > > Jack > Going to push for us to install 8.2 as soon as the release hits, thanks for your feedback Jack :) From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 10:32:22 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C2B0F106566B; Fri, 28 Jan 2011 10:32:22 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2EB648FC0A; Fri, 28 Jan 2011 10:32:21 +0000 (UTC) Received: by fxm16 with SMTP id 16so3297475fxm.13 for ; Fri, 28 Jan 2011 02:32:21 -0800 (PST) Received: by 10.223.87.5 with SMTP id u5mr411048fal.48.1296210741264; Fri, 28 Jan 2011 02:32:21 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id o17sm6355535fal.25.2011.01.28.02.32.20 (version=SSLv3 cipher=RC4-MD5); Fri, 28 Jan 2011 02:32:20 -0800 (PST) Message-ID: <4D429B33.2010402@my.gd> Date: Fri, 28 Jan 2011 11:32:19 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> <4D41D7BE.3030208@my.gd> <20110127205845.GA41537@icarus.home.lan> In-Reply-To: <20110127205845.GA41537@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 10:32:22 -0000 On 1/27/11 9:58 PM, Jeremy Chadwick wrote: > > Kernel folks should be able to talk about this in detail, but my > understanding is that the kernel itself supports multiple threads, but > the question is whether or not the drivers or relevant "pieces" (e.g. > igb(4) driver, pf, TCP stack, etc.) support SMP (multi-core/threading) > or not. I think this is referred to as something being "MPSAFE" or not. > > The things you see during boot -- [ITHREAD], [FILTER], and > [GIANT-LOCKED] play a role as well, but I think those indicate what > style of locking is used (since some drivers/features might not work > properly in a multiprocessor environment). > > I'm trying to avoid correlating "multiprocessor safe" with "makes use of > multiple processors". I'm an old 65xxx CPU guy, this SMP stuff is still > "new technology" to me when it comes to actual operations/mechanics. > > Regarding TCP and SMP, this is regularly touched on in the FreeBSD > Status Reports that go out (always worth reading). See "TCP SMP > scalability project": > > http://www.freebsd.org/news/status/report-2010-10-2010-12.html > > I know all this information is technical of course and doesn't answer > your question directly. I wish there was something more authoritative > when it came to this question. > Thanks for your time explaining all this, I'll have a look at your link, even if it may or may not apply directly to our case, it'll still be interesting material. I'll have to poke around for information about the kernel and how it works with multithreading :) From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 12:05:14 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 814E11065679 for ; Fri, 28 Jan 2011 12:05:14 +0000 (UTC) (envelope-from inigoortizdeurbina@gmail.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id 3B93D8FC1C for ; Fri, 28 Jan 2011 12:05:13 +0000 (UTC) Received: by yie19 with SMTP id 19so1083898yie.13 for ; Fri, 28 Jan 2011 04:05:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=PxGZpwNtjKOHJ9KyxD+r6sXY7zhiJT2mBsdEBUsr9Y8=; b=W0ENZjQOzlxZhxa5yGZBwn5gM6jhKTbcO9RG5obQEP50YLJYOWSCy1QyGSELScmsx8 068esQeqYcwXWUKv43VkmBKN8I2k8AFxY6jyOcodtmD2JyXRxlgyHmkRFylpI+GDMRCi zapopC98y8B/rpTVTBfKdJSdGs/N8YEEqYJ6o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=CXRZas7qc/VfZKFLhWsSuOMT7x7WfVWEVGFklrouk7nFAQb++Msqk8pigsNHpbLJv+ QajSYL93OSKM3xhbBTN3qh5VNoE5TB95jvziHHDWdMwSTkocoz9kGUCGcD6p1O6cwz4a eS0uz2VXvfNMgoyq+67oaoISC2oMWE+sx5BtY= MIME-Version: 1.0 Received: by 10.100.167.1 with SMTP id p1mr1482310ane.136.1296214452079; Fri, 28 Jan 2011 03:34:12 -0800 (PST) Received: by 10.101.188.16 with HTTP; Fri, 28 Jan 2011 03:34:12 -0800 (PST) In-Reply-To: <9E8D76EC267C9444AC737F649CBBAD9027BC4023C4@PEMEXMBXVS02.jellyfishnet.co.uk.local> References: <4D428A38.8000609@gmail.com> <9E8D76EC267C9444AC737F649CBBAD9027BC4023C4@PEMEXMBXVS02.jellyfishnet.co.uk.local> Date: Fri, 28 Jan 2011 12:34:12 +0100 Message-ID: From: =?UTF-8?Q?I=C3=B1igo_Ortiz_de_Urbina?= To: Greg Hennessy , "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: why "block quick on wlan0" doesn't stop DHCP? X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 12:05:14 -0000 And it makes perfect sense only if you can trust your dhcp server (runs chrooted and privilege separated :) On 1/28/11, Greg Hennessy wrote: > Could be talking complete nonsense here, but.... > > IIRC BPF sees all traffic before PF. DHCP hooks at the BPF layer, so it'l= l > be serviced before any filtering policy applies. > > > Greg > > >> -----Original Message----- >> From: owner-freebsd-pf@freebsd.org [mailto:owner-freebsd- >> pf@freebsd.org] On Behalf Of Michael >> Sent: 28 January 2011 9:20 AM >> To: freebsd-pf@freebsd.org >> Subject: why "block quick on wlan0" doesn't stop DHCP? >> >> Hello, >> >> Here is my simple rule set: >> >> set loginterface wlan0 >> block log >> block quick on wlan0 >> >> Now I'm booting my 8.1-R box. After it's up and running with pf I'm >> powering on my wireless access point. >> >> After couple seconds my wlan0 is associated and receives it's IP >> address. I don't understand why was it not stopped by pf? >> And how can I tune my rules to be able to control DHCP conversation? >> >> Michael >> _______________________________________________ >> freebsd-pf@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-pf >> To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > --=20 I=C3=B1igo Ortiz de Urbina Cazenave http://www.twitter.com/ioc32 From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 15:25:28 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB666106567A for ; Fri, 28 Jan 2011 15:25:28 +0000 (UTC) (envelope-from mlmichael70@gmail.com) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3B3628FC1A for ; Fri, 28 Jan 2011 15:25:27 +0000 (UTC) Received: by wwi17 with SMTP id 17so1116800wwi.1 for ; Fri, 28 Jan 2011 07:25:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=C3PnlDydypry0BVuWe2tIv8XYlZDYP/UCRIoYsyQ67Q=; b=iubbcRO/+Q5W5ihrwmdsGNiVyJ/rn2PQbkGR9edU3xDuiPnjGMy/a48pZbBPYsyFpQ JXKheb/cYQbOSMjnIDC5d0yC42ZWmigy6BvOQIKOUsRuhjGP2GrqcBUJxfS3mq5uXO0P RgzVpLmRNoi4m2eXNMNv7RRPjHGmnI3hZIKrI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; b=KvHz/SV5Jl6oD7rAtbf8cTKsBfLDFi/Im0RL6dUg288Fmlgbi4aKn3wJWv5gFPDvcU p2z4FZuO0b51Psx4iqH7sn7HxZQYWxijTHs23EJ2pRwC9L6hZKMojPl2LwJ4Ur4PHY4T JfGAnM+lP4kbrBlKP9PqFEbHf4xCYEj+lC2h0= Received: by 10.227.138.15 with SMTP id y15mr2785773wbt.186.1296228327174; Fri, 28 Jan 2011 07:25:27 -0800 (PST) Received: from prime.nonspace ([82.132.211.68]) by mx.google.com with ESMTPS id f27sm1902575wbf.7.2011.01.28.07.25.24 (version=SSLv3 cipher=RC4-MD5); Fri, 28 Jan 2011 07:25:25 -0800 (PST) Message-ID: <4D42DFEA.3020003@gmail.com> Date: Fri, 28 Jan 2011 15:25:30 +0000 From: Michael User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.13) Gecko/20101215 Thunderbird/3.1.7 MIME-Version: 1.0 To: Greg Hennessy References: <4D428A38.8000609@gmail.com> <9E8D76EC267C9444AC737F649CBBAD9027BC4023C4@PEMEXMBXVS02.jellyfishnet.co.uk.local> In-Reply-To: <9E8D76EC267C9444AC737F649CBBAD9027BC4023C4@PEMEXMBXVS02.jellyfishnet.co.uk.local> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-pf@freebsd.org" Subject: Re: why "block quick on wlan0" doesn't stop DHCP? X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 15:25:28 -0000 On 28/01/2011 09:47, Greg Hennessy wrote: > > IIRC BPF sees all traffic before PF. DHCP hooks at the BPF layer, so it'll be serviced before any filtering policy applies. > Now that's not cool man.. ;) So is it like there's nothing I can do about it? Thanks a lot for your explanation, I was not aware of that. Michael From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 16:37:08 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05714106564A for ; Fri, 28 Jan 2011 16:37:08 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 9A5DC8FC19 for ; Fri, 28 Jan 2011 16:37:06 +0000 (UTC) Received: by wwf26 with SMTP id 26so3360532wwf.31 for ; Fri, 28 Jan 2011 08:37:05 -0800 (PST) Received: by 10.227.141.147 with SMTP id m19mr2918300wbu.208.1296232625461; Fri, 28 Jan 2011 08:37:05 -0800 (PST) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id o6sm1166438wbo.21.2011.01.28.08.37.03 (version=SSLv3 cipher=RC4-MD5); Fri, 28 Jan 2011 08:37:03 -0800 (PST) Message-ID: <4D42F0AE.7010009@my.gd> Date: Fri, 28 Jan 2011 17:37:02 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-pf@freebsd.org References: <4D428A38.8000609@gmail.com> <9E8D76EC267C9444AC737F649CBBAD9027BC4023C4@PEMEXMBXVS02.jellyfishnet.co.uk.local> <4D42DFEA.3020003@gmail.com> In-Reply-To: <4D42DFEA.3020003@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: why "block quick on wlan0" doesn't stop DHCP? X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 16:37:08 -0000 On 1/28/11 4:25 PM, Michael wrote: > On 28/01/2011 09:47, Greg Hennessy wrote: >> >> IIRC BPF sees all traffic before PF. DHCP hooks at the BPF layer, so >> it'll be serviced before any filtering policy applies. >> > > Now that's not cool man.. ;) So is it like there's nothing I can do > about it? > Thanks a lot for your explanation, I was not aware of that. > > Michael Dirty workaround, in rc.conf ifconfig_wlan0="down" Or just give it a static IP. From owner-freebsd-pf@FreeBSD.ORG Fri Jan 28 16:38:45 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A44F1065673 for ; Fri, 28 Jan 2011 16:38:45 +0000 (UTC) (envelope-from Greg.Hennessy@nviz.net) Received: from mail1.jellyfishnet.co.uk (mail1.jellyfishnet.co.uk [93.91.20.9]) by mx1.freebsd.org (Postfix) with ESMTP id EE9B78FC18 for ; Fri, 28 Jan 2011 16:38:44 +0000 (UTC) Received: from pemexhub02.jellyfishnet.co.uk.local (93.91.20.2) by mail1.jellyfishnet.co.uk (93.91.20.9) with Microsoft SMTP Server (TLS) id 8.1.393.1; Fri, 28 Jan 2011 16:38:44 +0000 Received: from PEMEXMBXVS02.jellyfishnet.co.uk.local ([192.168.65.37]) by pemexhub02.jellyfishnet.co.uk.local ([192.168.65.8]) with mapi; Fri, 28 Jan 2011 16:38:42 +0000 From: Greg Hennessy To: =?utf-8?B?ScOxaWdvIE9ydGl6IGRlIFVyYmluYQ==?= , "freebsd-pf@freebsd.org" Date: Fri, 28 Jan 2011 16:38:40 +0000 Thread-Topic: why "block quick on wlan0" doesn't stop DHCP? Thread-Index: Acu+30qj+KGsnWwES1GkRMmSb0xBtQAKoLCA Message-ID: <9E8D76EC267C9444AC737F649CBBAD9027BC4027AE@PEMEXMBXVS02.jellyfishnet.co.uk.local> References: <4D428A38.8000609@gmail.com> <9E8D76EC267C9444AC737F649CBBAD9027BC4023C4@PEMEXMBXVS02.jellyfishnet.co.uk.local> In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Cc: Subject: RE: why "block quick on wlan0" doesn't stop DHCP? X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 16:38:45 -0000 VG9vIHRydWUuIA0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IEnDsWln byBPcnRpeiBkZSBVcmJpbmEgW21haWx0bzppbmlnb29ydGl6ZGV1cmJpbmFAZ21haWwuY29tXQ0K PiBTZW50OiAyOCBKYW51YXJ5IDIwMTEgMTE6MzQgQU0NCj4gVG86IEdyZWcgSGVubmVzc3k7IGZy ZWVic2QtcGZAZnJlZWJzZC5vcmcNCj4gU3ViamVjdDogUmU6IHdoeSAiYmxvY2sgcXVpY2sgb24g d2xhbjAiIGRvZXNuJ3Qgc3RvcCBESENQPw0KPiANCj4gQW5kIGl0IG1ha2VzIHBlcmZlY3Qgc2Vu c2Ugb25seSBpZiB5b3UgY2FuIHRydXN0IHlvdXIgZGhjcCBzZXJ2ZXINCj4gKHJ1bnMgY2hyb290 ZWQgYW5kIHByaXZpbGVnZSBzZXBhcmF0ZWQgOikNCj4gDQo+IE9uIDEvMjgvMTEsIEdyZWcgSGVu bmVzc3kgPEdyZWcuSGVubmVzc3lAbnZpei5uZXQ+IHdyb3RlOg0KPiA+IENvdWxkIGJlIHRhbGtp bmcgY29tcGxldGUgbm9uc2Vuc2UgaGVyZSwgYnV0Li4uLg0KPiA+DQo+ID4gSUlSQyBCUEYgc2Vl cyBhbGwgdHJhZmZpYyBiZWZvcmUgUEYuIERIQ1AgaG9va3MgYXQgdGhlIEJQRiBsYXllciwgc28N Cj4gaXQnbGwNCj4gPiBiZSBzZXJ2aWNlZCBiZWZvcmUgYW55IGZpbHRlcmluZyBwb2xpY3kgYXBw bGllcy4NCj4gPg0KPiA+DQo+ID4gR3JlZw0KPiA+DQo+ID4NCj4gPj4gLS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCj4gPj4gRnJvbTogb3duZXItZnJlZWJzZC1wZkBmcmVlYnNkLm9yZyBbbWFp bHRvOm93bmVyLWZyZWVic2QtDQo+ID4+IHBmQGZyZWVic2Qub3JnXSBPbiBCZWhhbGYgT2YgTWlj aGFlbA0KPiA+PiBTZW50OiAyOCBKYW51YXJ5IDIwMTEgOToyMCBBTQ0KPiA+PiBUbzogZnJlZWJz ZC1wZkBmcmVlYnNkLm9yZw0KPiA+PiBTdWJqZWN0OiB3aHkgImJsb2NrIHF1aWNrIG9uIHdsYW4w IiBkb2Vzbid0IHN0b3AgREhDUD8NCj4gPj4NCj4gPj4gSGVsbG8sDQo+ID4+DQo+ID4+IEhlcmUg aXMgbXkgc2ltcGxlIHJ1bGUgc2V0Og0KPiA+Pg0KPiA+PiBzZXQgbG9naW50ZXJmYWNlIHdsYW4w DQo+ID4+IGJsb2NrIGxvZw0KPiA+PiBibG9jayBxdWljayBvbiB3bGFuMA0KPiA+Pg0KPiA+PiBO b3cgSSdtIGJvb3RpbmcgbXkgOC4xLVIgYm94LiBBZnRlciBpdCdzIHVwIGFuZCBydW5uaW5nIHdp dGggcGYgSSdtDQo+ID4+IHBvd2VyaW5nIG9uIG15IHdpcmVsZXNzIGFjY2VzcyBwb2ludC4NCj4g Pj4NCj4gPj4gQWZ0ZXIgY291cGxlIHNlY29uZHMgbXkgd2xhbjAgaXMgYXNzb2NpYXRlZCBhbmQg cmVjZWl2ZXMgaXQncyBJUA0KPiA+PiBhZGRyZXNzLiBJIGRvbid0IHVuZGVyc3RhbmQgd2h5IHdh cyBpdCBub3Qgc3RvcHBlZCBieSBwZj8NCj4gPj4gQW5kIGhvdyBjYW4gSSB0dW5lIG15IHJ1bGVz IHRvIGJlIGFibGUgdG8gY29udHJvbCBESENQIGNvbnZlcnNhdGlvbj8NCj4gPj4NCj4gPj4gTWlj aGFlbA0KPiA+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Xw0KPiA+PiBmcmVlYnNkLXBmQGZyZWVic2Qub3JnIG1haWxpbmcgbGlzdA0KPiA+PiBodHRwOi8v bGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLXBmDQo+ID4+IFRvIHVu c3Vic2NyaWJlLCBzZW5kIGFueSBtYWlsIHRvICJmcmVlYnNkLXBmLQ0KPiB1bnN1YnNjcmliZUBm cmVlYnNkLm9yZyINCj4gPg0KPiANCj4gDQo+IC0tDQo+IEnDsWlnbyBPcnRpeiBkZSBVcmJpbmEg Q2F6ZW5hdmUNCj4gaHR0cDovL3d3dy50d2l0dGVyLmNvbS9pb2MzMg0K