Date: Thu, 17 Feb 2011 17:58:42 +0100 From: Damien Fleuriot <ml@my.gd> To: Jack Vogel <jfvogel@gmail.com> Cc: Sergey Lobanov <wmn@siberianet.ru>, "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org>, Jeremy Chadwick <freebsd@jdc.parodius.com>, "freebsd-pf@freebsd.org" <freebsd-pf@freebsd.org> Subject: Re: High interrupt rate on a PF box + performance Message-ID: <4D5D53C2.3010707@my.gd> In-Reply-To: <4D429A9F.8040307@my.gd> References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> <4D41D7BE.3030208@my.gd> <20110127205845.GA41537@icarus.home.lan> <AANLkTi=JYFNkXBGdmrt%2B0pq4PJZS=9gG-iV1OMOSkew3@mail.gmail.com> <4D429A9F.8040307@my.gd>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/28/11 11:29 AM, Damien Fleuriot wrote: > On 1/27/11 10:44 PM, Jack Vogel wrote: >> >> The 8.X kernel is NOT single-threaded. Anything but. And the stack has >> also been improved, I believe there are still bottlenecks but its far better >> than the old days. >> >> The igb driver in 8.2 creates up to 8 queues on the right hardware, they >> are each auto-bound to a particular CPU. >> >> The older version you are running had issues and hence multiqueue was >> not enabled. So, do upgrade once 8.2 is finalized :) >> >> Cheers, >> >> Jack >> > > Going to push for us to install 8.2 as soon as the release hits, thanks > for your feedback Jack :) Hello guys, list, This is a quick headsup regarding this issue. We have now swapped our PF firewalls to active-active and observed, as one would expect, approx. 50% drop of traffic, seeing it's now balanced between 2 machines :) We have also disabled pfsync (which also resulted in a massive drop of interrupts). One of the hosts is running 8.2-PRERELEASE , and this is the one for which I'm providing stats now. For completeness, also find the graphs: http://my.gd/fw_graphs/ # vmstat -i interrupt total rate irq16: mpt0 320899 0 irq21: atapci1 35 0 irq22: ehci0 ehci1 1992267 2 cpu0: timer 1330310985 1979 irq258: igb0:que 0 829898 1 irq259: igb0:que 1 3255 0 irq260: igb0:que 2 2059 0 irq261: igb0:que 3 1060 0 irq262: igb0:link 2 0 irq263: igb1:que 0 2676083520 3981 irq264: igb1:que 1 2676853656 3982 irq265: igb1:que 2 2682493388 3990 irq266: igb1:que 3 2688637571 3999 irq267: igb1:link 2 0 irq273: igb3:que 0 2654678899 3949 irq274: igb3:que 1 2648682488 3940 irq275: igb3:que 2 2650599952 3943 irq276: igb3:que 3 2657367887 3953 irq277: igb3:link 2 0 cpu1: timer 1330301807 1978 cpu2: timer 1330301315 1978 cpu3: timer 1330301347 1978 Total 26659762294 39659 # pfctl -si Status: Enabled for 7 days 18:43:34 Debug: Urgent Interface Stats for igb3 IPv4 IPv6 Bytes In 1585211309166 0 Bytes Out 2044715081803 0 Packets In Passed 6238056055 0 Blocked 15350206 0 Packets Out Passed 6300823415 0 Blocked 1223577 0 State Table Total Rate current entries 37627 searches 25108284353 37351.6/s inserts 2157108574 3209.0/s removals 2157070947 3208.9/s Counters match 2175657232 3236.6/s bad-offset 0 0.0/s fragment 104 0.0/s short 5 0.0/s normalize 557 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 6 0.0/s proto-cksum 52649 0.1/s state-mismatch 340029 0.5/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 90 0.0/s igb0@pci0:7:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82575GB Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xdabc0000, size 131072, enabled bar [14] = type Memory, range 32, base 0xdac00000, size 2097152, enabled bar [18] = type I/O Port, range 32, base 0xdcc0, size 32, enabled bar [1c] = type Memory, range 32, base 0xdabb8000, size 16384, enabled cap 01[40] = powerspec 2 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit cap 11[60] = MSI-X supports 10 messages in map 0x1c enabled cap 10[a0] = PCI-Express 2 endpoint max data 256(256) link x4(x4) ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 001b21ffff12f438 synproxy 0 0.0/s (there are 4 of these, it's a quad port card)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D5D53C2.3010707>