From nobody Fri Aug 5 06:19:54 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Lzb6t6rPhz4XyZ0 for ; Fri, 5 Aug 2022 06:20:06 +0000 (UTC) (envelope-from jim@jimking.net) Received: from flounder.jimking.net (flounder.jimking.net [IPv6:2607:fc50:1:8b00:216:3eff:fe33:f0f9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "flounder.jimking.net", Issuer "jimking.net CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lzb6t02W1z3cph for ; Fri, 5 Aug 2022 06:20:05 +0000 (UTC) (envelope-from jim@jimking.net) Received: from [192.168.22.253] (c-98-199-170-72.hsd1.tx.comcast.net [98.199.170.72]) (authenticated bits=0) by flounder.jimking.net (8.15.2/8.15.2) with ESMTPSA id 2756Jx38020822 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 5 Aug 2022 01:19:59 -0500 (CDT) (envelope-from jim@jimking.net) X-Authentication-Warning: flounder.jimking.net: Host c-98-199-170-72.hsd1.tx.comcast.net [98.199.170.72] claimed to be [192.168.22.253] Message-ID: <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> Date: Fri, 5 Aug 2022 01:19:54 -0500 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.12.0 Subject: Re: igc problems with heavy traffic Content-Language: en-US To: stable@freebsd.org References: <59b9cec0-d8c2-ce72-b5e9-99d1a1e807f8@sentex.net> From: Jim King In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.99.4 at flounder.jimking.net X-Virus-Status: Clean X-Rspamd-Queue-Id: 4Lzb6t02W1z3cph X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of jim@jimking.net designates 2607:fc50:1:8b00:216:3eff:fe33:f0f9 as permitted sender) smtp.mailfrom=jim@jimking.net X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+a:flounder.jimking.net]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; DMARC_NA(0.00)[jimking.net]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FREEFALL_USER(0.00)[jim]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; HAS_XAW(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:36236, ipnet:2607:fc50::/36, country:US]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 8/4/2022 12:29 PM, mike tancsa wrote: > On 8/4/2022 1:12 PM, mike tancsa wrote: >> On 7/31/2022 11:41 PM, Jim King wrote: >>> I'm trying to use a Qnap dual port 2.5Gb NIC with FreeBSD 13.1. >>> FreeBSD recognizes the NIC and I'm able to connect to my LAN at >>> 1Gb.  However, trying to push heavy traffic through the NIC results >>> in a temporary loss of connectivity:  traffic stops and the message >>> "igc0: link state changed to DOWN" appears in the system log.  About >>> 10 seconds later the message "igc0: link state changed to UP" is >>> logged, followed by some messages from dhclient indicating that it >>> has acquired a lease.  At that point network connectivity works >>> normally, until I try to push a lot of traffic again.  I've been >>> able to trigger this every time using iperf3 from another PC on my >>> LAN, and by using Ookla's speedtest client (my Internet service is >>> good for about 900 Mbps). >>> >> How long does it take to wedge the nic ? I have a pair of Topton >> routers (4 IGC NICs) back to back and a simple iperf3 test seems to >> work ok between the 2 units. >> > Actually, I was able to reproduce this too running iperf3 -P4 for > about 60 seconds. Watching via sysutils/ifstat, it does indeed drop > traffic when the link drops > > > ifstat -b > >        igc0                igc1               pflog0 >  Kbps in  Kbps out   Kbps in  Kbps out   Kbps in  Kbps out >     0.88      1.93      0.00      0.00      0.00      0.00 >     1.32      1.45      0.00      0.00      0.00      0.00 >     0.88      1.45      0.00      0.00      0.00      0.00 >     1.32      1.45      0.00      0.00      0.00      0.00 >     0.88      1.45      0.00      0.00      0.00      0.00 >     0.88      1.45      0.00      0.00      0.00      0.00 >     4.68     12.23   2497.66  395238.6      0.00      2.34 >     0.94      6.08  15036.53  2.31e+06      0.00      0.00 >     0.94      7.26  15076.75  2.31e+06      0.00      0.00 >     0.94      6.08  15014.83  2.31e+06      0.00      0.00 >     0.94      7.26  15054.63  2.31e+06      0.00      0.00 >     0.94      7.26  15022.16  2.31e+06      0.00      0.00 >     0.94      7.25  14928.42  2.31e+06      0.00      0.00 >     1.87      7.25  15000.98  2.31e+06      0.00      0.00 >     1.40      7.25  14960.84  2.31e+06      0.00      0.00 >     1.40      7.25  14919.44  2.31e+06      0.00      0.00 >     1.87      7.26  14789.44  2.31e+06      0.00      0.00 >     2.70      8.56   1168.12  178704.6      0.00      0.00 >     1.77      6.85      0.00      0.00      0.00      0.00 >     1.40      9.97      0.00      0.00      0.00      0.00 >     1.87      7.26  10732.92  1.83e+06      0.00      0.00 >     0.94      7.25  14166.38  2.32e+06      0.00      0.00 >     0.94      6.09  14269.06  2.31e+06      0.00      0.00 >     1.40      6.09  14402.47  2.31e+06      0.00      0.00 > > > This is on 13.1-STABLE from Aug 3. > > igc0@pci0:2:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 > device=0x15f3 subvendor=0x8086 subdevice=0x0000 >     vendor     = 'Intel Corporation' >     device     = 'Ethernet Controller I225-V' >     class      = network >     subclass   = ethernet >     bar   [10] = type Memory, range 32, base 0x80600000, size 1048576, > enabled >     bar   [1c] = type Memory, range 32, base 0x80700000, size 16384, > enabled >     cap 01[40] = powerspec 3  supports D0 D3  current D0 >     cap 05[50] = MSI supports 1 message, 64 bit, vector masks >     cap 11[70] = MSI-X supports 5 messages, enabled >                  Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] >     cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS >                  max read 512 >                  link x1(x1) speed 5.0(5.0) ASPM L1(L1) >     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected >     ecap 0003[140] = Serial 1 00e269ffff5a4be9 >     ecap 0018[1c0] = LTR 1 >     ecap 001f[1f0] = Precision Time Measurement 1 >     ecap 001e[1e0] = L1 PM Substates 1 > > Jasper Lake chipset N5105 CPU 2.00GHz > > >     ---Mike > When I'm receiving traffic the wedge usually happens pretty quick. iperf while connected at 1Gb (I get similar results at 2.5Gb): jim@t730:~ $ iperf3 -c snorlax --reverse Connecting to host snorlax, port 5201 Reverse mode, remote host snorlax is sending [  5] local 192.168.22.178 port 36033 connected to 192.168.22.253 port 5201 [ ID] Interval           Transfer     Bitrate [  5]   0.00-1.00   sec   107 MBytes   897 Mbits/sec [  5]   1.00-2.00   sec   112 MBytes   937 Mbits/sec [  5]   2.00-3.01   sec  85.9 MBytes   711 Mbits/sec [  5]   3.01-4.04   sec  0.00 Bytes  0.00 bits/sec [  5]   4.04-5.02   sec  0.00 Bytes  0.00 bits/sec [  5]   5.02-6.06   sec  0.00 Bytes  0.00 bits/sec [  5]   6.06-7.03   sec  0.00 Bytes  0.00 bits/sec [  5]   7.03-8.04   sec  0.00 Bytes  0.00 bits/sec [  5]   8.04-9.05   sec  0.00 Bytes  0.00 bits/sec [  5]   9.05-10.00  sec  0.00 Bytes  0.00 bits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate [  5]   0.00-10.00  sec   340 MBytes   285 Mbits/sec                  sender [  5]   0.00-10.00  sec   305 MBytes   255 Mbits/sec                  receiver I'm running 13.1-RELEASE. igc0@pci0:3:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f2 subvendor=0x1baa subdevice=0xc002     vendor     = 'Intel Corporation'     device     = 'Ethernet Controller I225-LM'     class      = network     subclass   = ethernet     bar   [10] = type Memory, range 32, base 0xfe800000, size 1048576, enabled     bar   [1c] = type Memory, range 32, base 0xfe900000, size 16384, enabled     cap 01[40] = powerspec 3  supports D0 D3  current D0     cap 05[50] = MSI supports 1 message, 64 bit, vector masks     cap 11[70] = MSI-X supports 5 messages, enabled                  Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]     cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS                  max read 512                  link x1(x1) speed 5.0(5.0) ASPM disabled(L1)     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected     ecap 0003[140] = Serial 1 245ebeffff6b6898     ecap 0018[1c0] = LTR 1     ecap 001f[1f0] = Precision Time Measurement 1     ecap 001e[1e0] = L1 PM Substates 1 Jim