From nobody Fri Aug 5 12:48:15 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Lzlkp0LhCz4Xsb9 for ; Fri, 5 Aug 2022 12:48:18 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lzlkn2t5mz3Cp2; Fri, 5 Aug 2022 12:48:17 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 275CmFBC057591 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 5 Aug 2022 08:48:15 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:f0d5:2e0:4f18:84d6] ([IPv6:2607:f3e0:0:4:f0d5:2e0:4f18:84d6]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 275CmF90089670 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 5 Aug 2022 08:48:15 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <3d874f65-8ce2-8f06-f19a-14cd550166e3@sentex.net> Date: Fri, 5 Aug 2022 08:48:15 -0400 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.1.0 Subject: Re: igc problems with heavy traffic Content-Language: en-US To: Jim King , stable@freebsd.org, kbowling@FreeBSD.org References: <59b9cec0-d8c2-ce72-b5e9-99d1a1e807f8@sentex.net> <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> From: mike tancsa In-Reply-To: <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Rspamd-Queue-Id: 4Lzlkn2t5mz3Cp2 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; R_DKIM_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[mike]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[sentex.net]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 8/5/2022 2:19 AM, Jim King wrote: > On 8/4/2022 12:29 PM, mike tancsa wrote: >> On 8/4/2022 1:12 PM, mike tancsa wrote: >>> On 7/31/2022 11:41 PM, Jim King wrote: >>>> I'm trying to use a Qnap dual port 2.5Gb NIC with FreeBSD 13.1. >>>> FreeBSD recognizes the NIC and I'm able to connect to my LAN at >>>> 1Gb.  However, trying to push heavy traffic through the NIC results >>>> in a temporary loss of connectivity:  traffic stops and the message >>>> "igc0: link state changed to DOWN" appears in the system log. About >>>> 10 seconds later the message "igc0: link state changed to UP" is >>>> logged, followed by some messages from dhclient indicating that it >>>> has acquired a lease.  At that point network connectivity works >>>> normally, until I try to push a lot of traffic again.  I've been >>>> able to trigger this every time using iperf3 from another PC on my >>>> LAN, and by using Ookla's speedtest client (my Internet service is >>>> good for about 900 Mbps). >>>> >>> How long does it take to wedge the nic ? I have a pair of Topton >>> routers (4 IGC NICs) back to back and a simple iperf3 test seems to >>> work ok between the 2 units. >>> >> Actually, I was able to reproduce this too running iperf3 -P4 for >> about 60 seconds. Watching via sysutils/ifstat, it does indeed drop >> traffic when the link drops >> >> > When I'm receiving traffic the wedge usually happens pretty quick. > iperf while connected at 1Gb (I get similar results at 2.5Gb): > If I fiddle with the Flow Control settings it seems to make a difference.  I can do it without bouncing the link.  I wonder what the Linux driver has as default ? ie. sysctl -w dev.igc.1.fc=1 sysctl -w dev.igc.0.fc=1 Does it help in your case ?          * The possible values of the "fc" parameter are:          *      0:  Flow control is completely disabled          *      1:  Rx flow control is enabled (we can receive pause          *          frames but not send pause frames).          *      2:  Tx flow control is enabled (we can send pause frames          *          frames but we do not receive pause frames).          *      3:  Both Rx and Tx flow control (symmetric) is enabled.          *  other:  No other values should be possible at this point. I am also able to do netsend 10.1.255.209 500 1400 190000 50 Sending packet of payload size 1400 every 0.000005263s for 50 seconds calling time every 38 cycles start:             1659703401.000000000 finish:            1659703451.000103336 send calls:        9500304 send errors:       0 approx send rate:  190006 pps time/packet:       5263 ns approx error rate: 0 waited:            820237403 approx waits/sec:  16404748 approx wait rate:  86     ---Mike