From owner-freebsd-net@freebsd.org Thu Jul 16 09:03:54 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B945135B846 for ; Thu, 16 Jul 2020 09:03:54 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B6pG11TLkz4GWb for ; Thu, 16 Jul 2020 09:03:52 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 06G93Xt6014421 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 16 Jul 2020 09:03:37 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: patfbsd@davenulle.org Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 06G93YWT079516 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 16 Jul 2020 16:03:34 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: poor performance with Intel X520 card To: Patrick Lamaiziere References: <20200710084530.777ce321@mr185033.univ-rennes1.fr> <7b994967-17cd-8f4f-cb4d-8fcff349f7e9@grosbein.net> <20200716095732.728c0917@mr185033.univ-rennes1.fr> Cc: freebsd-net@freebsd.org From: Eugene Grosbein Message-ID: <2746b70d-7072-a10b-5ec0-06c5c4917fc7@grosbein.net> Date: Thu, 16 Jul 2020 16:03:18 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20200716095732.728c0917@mr185033.univ-rennes1.fr> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * 2.6 LOCAL_FROM From my domains * -0.0 NICE_REPLY_A Looks like a legit reply (A) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 4B6pG11TLkz4GWb X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-0.93 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.35)[-0.354]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.21)[-0.212]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[empty SPF record]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.26)[-0.260]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2020 09:03:54 -0000 16.07.2020 14:57, Patrick Lamaiziere wrote: >> I'm sure pf is the bottle-neck. Try testing such card without any >> packet filter enabled and you'll see great difference definitely. > > That's not a good news as I don't see how to simplify the ruleset :( > But thanks anyway :) First, you need to determine if all your CPU cores loaded evenly or not. Use top -SHPI to check it out. It's possible that they are not because defaults could be not optimal for your hardware and you have bottle-neck just because of wrong settings for hardware receiving queues of the NIC: FreeBSD/SMP: 2 package(s) x 6 core(s) ix0: Using MSI-X interrupts with 9 vectors This means 8 RX/TX queues (plus one link queue) for your 12 cores, not good. 8 is driver's default for systems with 8 cores or more unless you override it with hw.ix.num_queues=12 in /boot/loader.conf, do it. Also you should monitor interrupt counters for receiving queues for the card with "systat -vm 3" (in real time) and with vmstat -ai | grep ix0 (cumulative). If you have interrupts evenly distributed over receiving queues but still uneven CPU cores load, that would be next problem to deal with. Maybe you'll need to (re-)bind queues to cores with cpuset(1). Or maybe your received queues are loaded not evenly due to nature of Ethernet frames you work with, f.e. non-IP ethernet traffic (PPPoE frames) or many non-TCP/UDP IP packets (GRE, ESP etc.) cpuset will be your friend for that case, too.