Date: Wed, 28 Aug 2019 03:30:46 +0700 From: Eugene Grosbein <eugen@grosbein.net> To: Victor Gamov <vit@otcnet.ru>, "Andrey V. Elsukov" <bu7cher@yandex.ru>, freebsd-net@freebsd.org Subject: Re: finding optimal ipfw strategy Message-ID: <eb249ea4-1f14-4826-d235-ed81e1c5e4d0@grosbein.net> In-Reply-To: <f2aa4e0e-2339-d3e6-5a41-567b0c55b9e3@otcnet.ru> References: <f38b21a5-8f9f-4f60-4b27-c810f78cdc88@otcnet.ru> <4ff39c8f-341c-5d72-1b26-6558c57bff8d@grosbein.net> <a559d2bd-5218-f344-2e88-c00893272222@otcnet.ru> <ddaa55bc-1fa5-151b-258e-e3e9844802ef@yandex.ru> <c275f853-62a7-6bb7-d309-bf8a27d3dbae@grosbein.net> <f2aa4e0e-2339-d3e6-5a41-567b0c55b9e3@otcnet.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
28.08.2019 2:20, Victor Gamov wrote: >> Victor, do you have some non-default tuning in your /boot/loader.conf or /etc/sysctl.conf? >> If yes, could you show them? > Nothing special. > > loader.conf > ===== > machdep.hyperthreading_allowed="0" > net.inet.ip.fw.default_to_accept=1 > ===== > > sysctl.conf > ===== > net.link.ether.ipfw=1 > net.link.bridge.ipfw=1 > net.link.bridge.ipfw_arp=1 > net.link.bridge.pfil_member=1 > > net.inet.ip.fw.verbose_limit=100 > net.inet.ip.fw.verbose=1 > ===== You should avoid passing same packet multiple times through the ruleset. Less checks, better performance. Do you really use ipfw filtering based on layer2 parameters like MAC addresses? If not, you should disable net.link.ether.ipfw. If yes, you should use "layer2" keyword explicily in rules filtering by ethernet headers and place these rules above others and use "allow ip from any to any layer2" after L2 filtering is done, so L2 packets do not go through other rules extra time. Do you really need to filter each bridged L3 packet twice? Once as "out xmit $bridge" and once as "out xmit $brige_member"? If not, you should disable net.link.bridge.ipfw and keep net.link.bridge.pfil_member=1 only. Perhaps, you are ruining the performance with such settings making same work 3 times without real need. Do you really need filtering ARP? Disable net.link.bridge.ipfw_arp if not. > `sysctl net.isr` > ===== > sysctl net.isr > net.isr.numthreads: 1 > net.isr.maxprot: 16 > net.isr.defaultqlimit: 256 > net.isr.maxqlimit: 10240 > net.isr.bindthreads: 0 > net.isr.maxthreads: 1 > net.isr.dispatch: direct > ===== > > I don't know about internals but I think high interrupt load is bad and it because NIC does not support per CPU-queue for example. All decent igb(4) NICs support at least 8 hardware input queues unless disabled by driver/kernel. However, net.isr settings are not about such queues. High interrupt number is definitely better than dropping input frames by NIC chip due to overflow of its internal buffers just because CPU was not notified it's time to get traffic out of these buffers. The driver tries not to overload CPU with interrupts and that's fine but default 8000 limit is not adequate to modern CPU and was not adequate for many years. Raise limit to 32000. >> If not, you should try something like this. For loader.conf: > > Sorry, it's a production system and I can reboot it at the middle of October only. > >> #substitute total number of CPU cores in the system here >> net.isr.maxthreads=4 >> # EOF > > Is it ok for multicast? It's UDP traffic which must be ordered. I read 'maxthreads=1' used to keep TCP traffic ordered. It's a job for uplink to feed your bridge with ordered UDP flows. If you use igb(4) driver, FreeBSD kernel will keep flows ordered automatically. There is no place in the code where they could be reordered if you do not use lagg(4) without LACP. >> And if you haven't already seen it, you may find useful my blog post >> (in Russian) https://dadv.livejournal.com/139170.html >> It's a bit old but still can give you some light. > Yes, I read it already :-) Also some Calomel articles. I'll try to tune system at next reboot. > The main question for myself now is "how is this architecture correct" and "how many traffic is possible to process". You have read numbers from my posts. ipfw+dummynet+PPPoE+routing+LACP+vlan tagging takes much more CPU power than just ipfw+bridging and my system still processed much more traffic. Make sure you don't pass same packets multiple times through ipfw rules. ipfw has its counters for rules and you should use them to summarize octets carefully and compare with numbers shown by netstat or systat (they both have same in-kernel source) to verify whether packets go through ipfw extra times or not.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?eb249ea4-1f14-4826-d235-ed81e1c5e4d0>