From owner-freebsd-net@freebsd.org Tue Aug 27 18:46:35 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 61318DC3B1 for ; Tue, 27 Aug 2019 18:46:35 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46HyWt1tKxz4C0S for ; Tue, 27 Aug 2019 18:46:33 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id x7RIkGHO055171 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 27 Aug 2019 18:46:22 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: bu7cher@yandex.ru Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x7RIkAtM004145 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 28 Aug 2019 01:46:10 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: finding optimal ipfw strategy To: "Andrey V. Elsukov" , Victor Gamov , freebsd-net@freebsd.org References: <4ff39c8f-341c-5d72-1b26-6558c57bff8d@grosbein.net> From: Eugene Grosbein Message-ID: Date: Wed, 28 Aug 2019 01:46:03 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 46HyWt1tKxz4C0S X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-4.50 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.975,0]; IP_SCORE(-1.43)[ip: (-3.35), ipnet: 2a01:4f8::/29(-1.97), asn: 24940(-1.80), country: DE(-0.01)]; FREEMAIL_TO(0.00)[yandex.ru]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Aug 2019 18:46:35 -0000 28.08.2019 1:03, Andrey V. Elsukov wrote: > As you can see, when ipfw produces high load, interrupt column is more > than system. Interrupt numbers higher than others generally mean that traffic is processed without netisr queueing mostly. That is expected for plain routing. I'm not sure if this would be same in case of bridging. Victor, do you have some non-default tuning in your /boot/loader.conf or /etc/sysctl.conf? If yes, could you show them? If not, you should try something like this. For loader.conf: hw.igb.rxd=4096 hw.igb.txd=4096 net.isr.bindthreads=1 net.isr.defaultqlimit=4096 #substitute total number of CPU cores in the system here net.isr.maxthreads=4 # EOF For /etc/sysctl.conf: dev.igb.0.rx_processing_limit=-1 dev.igb.1.rx_processing_limit=-1 net.inet.ip.intr_queue_maxlen=40960 And if you haven't already seen it, you may find useful my blog post (in Russian) https://dadv.livejournal.com/139170.html It's a bit old but still can give you some light.