From owner-freebsd-net@freebsd.org Tue Aug 27 21:54:22 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19CA9C8D57 for ; Tue, 27 Aug 2019 21:54:22 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46J2hY35byz4P7n for ; Tue, 27 Aug 2019 21:54:21 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id x7RLsHJx057131 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 27 Aug 2019 21:54:18 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: vit@otcnet.ru Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x7RLsD6C013293 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 28 Aug 2019 04:54:13 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: finding optimal ipfw strategy To: Victor Gamov , "Andrey V. Elsukov" , freebsd-net@freebsd.org References: <4ff39c8f-341c-5d72-1b26-6558c57bff8d@grosbein.net> <50b0748d-bf8a-4de9-58bf-800ddd4f9c27@grosbein.net> <60d075ee-0d39-6606-ea5a-35e27818162a@otcnet.ru> <032b67e1-5359-4f00-a449-26c43b70a467@grosbein.net> <328ff261-f247-976f-ccba-e4b4e9098d8c@otcnet.ru> From: Eugene Grosbein Message-ID: Date: Wed, 28 Aug 2019 04:54:06 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <328ff261-f247-976f-ccba-e4b4e9098d8c@otcnet.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 46J2hY35byz4P7n X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-4.59 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.976,0]; IP_SCORE(-1.52)[ip: (-3.81), ipnet: 2a01:4f8::/29(-1.97), asn: 24940(-1.80), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Aug 2019 21:54:22 -0000 28.08.2019 3:20, Victor Gamov wrote: >>>> hw.igb.max_interrupt_rate=32000 >>> >>> It's about 5000-7000 per rxq >> >> 7000 is quite close considering it is average for quite long period (a second or seconds). >> It can hit 8000 for some ticks easily in your case. > > Eugene, can you explain me what max_interrupt_rate means? Calomel explain it as "# maximum number of interrupts per second generated by single igb". So if I increase it then more irq per sec will be generated and more context switch produced. Why I need to increase it? Already did in another post. >> You should raise the limit as soon as possible. I'd advise to double it upto 16000 first >> but if you cannot afford extra reboot better use 32000 as it's just fine for 1Gbps link. > # sysctl hw.igb.max_interrupt_rate > sysctl: unknown oid 'hw.igb.max_interrupt_rate' > # > Is it possible to set it at boot time only? Yes, unless you patched igb(4) sources with a patch as described in by blog: http://www.grosbein.net/freebsd/patches/igb_sysctl-11.0.diff.gz It adds per-interface sysctls to change this on-the-fly: dev.igb.0.max_interrupt_rate=32000 dev.igb.1.max_interrupt_rate=32000 Be warned that you still need to re-initialize the NIC to write changed value to the chip and this will temporary stop traffic. Easiest way is to bring the interface down/up: ifconfig igb0 down; ifconfig igb0 up hw.igb.max_interrupt_rate is used as default value for sysctls so if you raise the limit in loader.conf, it is used by chip on first "up" of interface at boot time.