From nobody Wed Aug 24 23:37:43 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MCjFW2xCsz4b7vx for ; Wed, 24 Aug 2022 23:37:51 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MCjFV3cygz3kbw; Wed, 24 Aug 2022 23:37:50 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 27ONbhmZ088876 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 24 Aug 2022 19:37:43 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:98e0:266c:af8f:4500] ([IPv6:2607:f3e0:0:4:98e0:266c:af8f:4500]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 27ONbgcn053238 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 24 Aug 2022 19:37:42 -0400 (EDT) (envelope-from mike@sentex.net) Content-Type: multipart/alternative; boundary="------------ldD92l6s0LM009C2pxGK06oA" Message-ID: <2fa9c9d7-1eb9-e7ad-1c19-a6202ac7082b@sentex.net> Date: Wed, 24 Aug 2022 19:37:43 -0400 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.1.2 Subject: Re: igc problems with heavy traffic Content-Language: en-US To: mike.jakubik@swiftsmsgateway.com Cc: "pieper, jeffrey e" , jim king , "stable@freebsd.org" , "kbowling@freebsd.org" References: <59b9cec0-d8c2-ce72-b5e9-99d1a1e807f8@sentex.net> <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> <3d874f65-8ce2-8f06-f19a-14cd550166e3@sentex.net> <879b9239-2b9a-f0ae-4173-4a226c84cd85@sentex.net> <182d22a6c6d.1119560c11283607.2998737705092721009@swiftsmsgateway.com> From: mike tancsa In-Reply-To: <182d22a6c6d.1119560c11283607.2998737705092721009@swiftsmsgateway.com> X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Rspamd-Queue-Id: 4MCjFV3cygz3kbw X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[stable@freebsd.org]; R_DKIM_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; MID_RHS_MATCH_FROM(0.00)[]; FREEFALL_USER(0.00)[mike]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[sentex.net]; TO_DN_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------ldD92l6s0LM009C2pxGK06oA Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 8/24/2022 7:22 PM, Mike Jakubik wrote: > What kind of HW are you running on? Im assuming some sort of fairly > modern x86 CPU with at least 4 cores.. Is it multiple CPUs with Numa > nodes perhaps? In any case, if you are testing with iperf3, try using > cpuset on iperf3 to bind it to specific cores. I had a performance > issue on a modern Epyc server with a Mellanox 25Gb card. It turns out > the issue was with the scheduler and how it was bouncing the processes > around diff cores/CPU caches. See "Poor performance with stable/13 and > Mellanox ConnectX-6 (mlx5)" on the freebsd-net mailing list for details. > > P.S. I also use a number of igc (Intel i225 @ 2.5Gb) cards at home and > have had no issues with them. > > Hi,     Performance is excellent. Its just the random link drops thats at issue.  With default settings, running iperf3 on back to back NICs via xover takes a good 20-45min before the link drop. If anything, I am surprised at how much traffic these small devices can forward.  IPSEC especially is super fast on RELENG_13. The link drops seem to be always on the sender.  With fc disabled, reducing the link speed to 1G seems to make the issue go away, or at least its not happening in overnight testing. Its a Celeron N5105. https :// www.aliexpress.com/item/1005003990581434.html Also, if you hook a couple back to back via xover cable, are you able to manually set the speed to 1G and pass traffic ? It doesnt work for me.     ---Mike --------------ldD92l6s0LM009C2pxGK06oA Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 8/24/2022 7:22 PM, Mike Jakubik wrote:
What kind of HW are you running on? Im assuming some sort of fairly modern x86 CPU with at least 4 cores.. Is it multiple CPUs with Numa nodes perhaps? In any case, if you are testing with iperf3, try using cpuset on iperf3 to bind it to specific cores. I had a performance issue on a modern Epyc server with a Mellanox 25Gb card. It turns out the issue was with the scheduler and how it was bouncing the processes around diff cores/CPU caches. See "Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)" on the freebsd-net mailing list for details.

P.S. I also use a number of igc (Intel i225 @ 2.5Gb) cards at home and have had no issues with them.


Hi,

    Performance is excellent. Its just the random link drops thats at issue.  With default settings, running iperf3 on back to back NICs via xover takes a good 20-45min before the link drop.  If anything, I am surprised at how much traffic these small devices can forward.  IPSEC especially is super fast on RELENG_13. The link drops seem to be always on the sender.  With fc disabled, reducing the link speed to 1G seems to make the issue go away, or at least its not happening in overnight testing. Its a Celeron N5105. https :// www.aliexpress.com/item/1005003990581434.html

Also, if you hook a couple back to back via xover cable, are you able to manually set the speed to 1G and pass traffic ? It doesnt work for me.

    ---Mike

--------------ldD92l6s0LM009C2pxGK06oA--