From nobody Mon Feb 28 19:36:13 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3889719DBEEC for ; Mon, 28 Feb 2022 19:36:21 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K6rGX1J2bz4Z33; Mon, 28 Feb 2022 19:36:20 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 21SJaD8V036966 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Feb 2022 14:36:13 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4::29] ([IPv6:2607:f3e0:0:4:0:0:0:29]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 21SJaDoc039075 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Mon, 28 Feb 2022 14:36:13 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <49736514-6500-4c7d-889d-15697d934f04@sentex.net> Date: Mon, 28 Feb 2022 14:36:13 -0500 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.6.1 Content-Language: en-US To: FreeBSD-STABLE Mailing List From: mike tancsa Subject: cxgbe firmware error on latest RELENG_13 Cc: Navdeep Parhar Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 X-Rspamd-Queue-Id: 4K6rGX1J2bz4Z33 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[mike]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sentex.net]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.997]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-stable]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received] X-ThisMailContainsUnwantedMimeParts: N Hi Navdeep,     I updated a pair of releng13 boxes today as I saw there are new drivers/firmware. However, the firmware seems to fail on both in different ways t5nex0: mem 0x91300000-0x9137ffff,0x90000000-0x90ffffff,0x91984000-0x91985fff irq 17 at device 0.4 on pci2 t5nex0: failed to load firmware module: 2. cfg 0, fw 0 t5nex0: firmware on card (1.25.0.40) is older than the version bundled with this driver, but the driver cannot take corrective action because it is unable to load the firmware module. t5nex0: KLD with default config is not available. t5nex0: failed to upload config file to card: 2. t5nex0: failed (2) to configure card with "default" profile, will fall back to a basic configuration and retry. cxl0: on t5nex0 cxl0: Ethernet address: 00:07:43:29:98:90 cxl0: 12 txq, 8 rxq (NIC) cxl1: on t5nex0 cxl1: Ethernet address: 00:07:43:29:98:98 cxl1: 12 txq, 8 rxq (NIC) cxl2: on t5nex0 cxl2: Ethernet address: 00:07:43:29:98:a0 cxl2: 12 txq, 8 rxq (NIC) cxl3: on t5nex0 cxl3: Ethernet address: 00:07:43:29:98:a8 cxl3: 12 txq, 8 rxq (NIC) t5nex0: PCIe gen3 x8, 4 ports, 34 MSI-X interrupts, 84 eq, 33 iq And on another adapter, different box t5nex0: mem 0xb2300000-0xb237ffff,0xb1000000-0xb1ffffff,0xb28a0000-0xb28a1fff irq 16 at device 0.4 on pci1 t5nex0: command 0x4 in mbox 4 timed out (0x80000000). t5nex0: mbox 4 cmdsent 04a0000000000001 0f41000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 t5nex0: mbox 4 current 04a0000000000001 0f41000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 t5nex0: firmware reports adapter error: Crash (0x80000000) t5nex0: device log follows.       Seq#           Tstamp     Level  Facility  Message          0           507376      INFO      CORE  log initialized @ 0x200a0000 size 32768 (128 entries) fwrev 0x011a0200 pcie_fw 0x00000000          1           510262      INFO      CORE  bootstrap firmware took 605 msecs to run          2           511784      INFO      CORE  Serial Configuration version: 0x11101000 VPD version: 0x2          3           511785    NOTICE      CORE  pcie: Serial Configuration SPARE3 has Cookie Generation enabled.          4           511786      INFO      CORE  pcie: npf 7 (pfbitmap 0x7f) nvf 64 (pf 0..7 0x1010101000000000) vfstride 4 t5nex0: encountered fatal error, adapter stopped. t5nex0: firmware reports adapter error: Crash (0x80000000) t5nex0: device log follows.       Seq#           Tstamp     Level  Facility  Message          0           507376      INFO      CORE  log initialized @ 0x200a0000 size 32768 (128 entries) fwrev 0x011a0200 pcie_fw 0x00000000          1           510262      INFO      CORE  bootstrap firmware took 605 msecs to run          2           511784      INFO      CORE  Serial Configuration version: 0x11101000 VPD version: 0x2          3           511785    NOTICE      CORE  pcie: Serial Configuration SPARE3 has Cookie Generation enabled.          4           511786      INFO      CORE  pcie: npf 7 (pfbitmap 0x7f) nvf 64 (pf 0..7 0x1010101000000000) vfstride 4 t5nex0: failed to connect to the firmware: 6, 0.  PCIE_FW 0x80000000 t5nex0: error during attach, adapter is now in recovery mode. none0@pci0:1:0:5:       class=0x010000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5511 subvendor=0x1425 subdevice=0x0000     vendor     = 'Chelsio Communications Inc'     device     = 'T520-LL-CR Unified Wire Storage Controller'     class      = mass storage     subclass   = SCSI     bar   [10] = type Memory, range 64, base 0xb2280000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0xb2200000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0xb2890000, size 8192, enabled     cap 01[40] = powerspec 3  supports D0 D3  current D0     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO                  max read 512                  link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1)     cap 11[b0] = MSI-X supports 40 messages                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]     cap 03[d0] = VPD     ecap 0001[100] = AER 2 1 fatal 0 non-fatal 0 corrected     ecap 0003[170] = Serial 1 0000000000000000     ecap 000e[190] = ARI 1     ecap 0019[1a0] = PCIe Sec 1 lane errors 0     ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled                      0 VFs configured out of 0 supported                      First VF RID Offset 0x0008, VF RID Stride 0x0004                      VF Device ID 0x5811                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304     ecap 0017[200] = TPH Requester 1 none1@pci0:1:0:6:       class=0x0c0400 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5611 subvendor=0x1425 subdevice=0x0000     vendor     = 'Chelsio Communications Inc'     device     = 'T520-LL-CR Unified Wire Storage Controller'     class      = serial bus     subclass   = Fibre Channel     bar   [10] = type Memory, range 64, base 0xb2100000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0xb2080000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0xb2880000, size 8192, enabled     cap 01[40] = powerspec 3  supports D0 D3  current D0     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO                  max read 512                  link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1)     cap 11[b0] = MSI-X supports 40 messages                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]     cap 03[d0] = VPD     ecap 0001[100] = AER 2 1 fatal 0 non-fatal 0 corrected     ecap 0003[170] = Serial 1 0000000000000000     ecap 000e[190] = ARI 1     ecap 0019[1a0] = PCIe Sec 1 lane errors 0     ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled                      0 VFs configured out of 0 supported                      First VF RID Offset 0x0008, VF RID Stride 0x0004                      VF Device ID 0x5811                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304     ecap 0017[200] = TPH Requester 1 t5iov3@pci0:2:0:3:      class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5003 subvendor=0x1425 subdevice=0x0000     vendor     = 'Chelsio Communications Inc'     device     = 'T540-CR Unified Wire Ethernet Controller'     class      = network     subclass   = ethernet     bar   [10] = type Memory, range 64, base 0x91480000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x91400000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0x91986000, size 8192, enabled     cap 01[40] = powerspec 3  supports D0 D3  current D0     cap 05[50] = MSI supports 8 messages, 64 bit, vector masks     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO                  max read 512                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)     cap 11[b0] = MSI-X supports 8 messages                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]     cap 03[d0] = VPD     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected     ecap 0003[170] = Serial 1 0000000000000000     ecap 000e[190] = ARI 1     ecap 0019[1a0] = PCIe Sec 1 lane errors 0     ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled                      0 VFs configured out of 16 supported                      First VF RID Offset 0x0008, VF RID Stride 0x0004                      VF Device ID 0x5803                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304     ecap 0017[200] = TPH Requester 1 t5nex0@pci0:2:0:4:      class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5403 subvendor=0x1425 subdevice=0x0000     vendor     = 'Chelsio Communications Inc'     device     = 'T540-CR Unified Wire Ethernet Controller'     class      = network     subclass   = ethernet     bar   [10] = type Memory, range 64, base 0x91300000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x90000000, size 16777216, enabled     bar   [20] = type Memory, range 64, base 0x91984000, size 8192, enabled     cap 01[40] = powerspec 3  supports D0 D3  current D0     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR                  max read 4096                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)     cap 11[b0] = MSI-X supports 128 messages, enabled                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]     cap 03[d0] = VPD     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected     ecap 0003[170] = Serial 1 0000000000000000     ecap 000e[190] = ARI 1     ecap 0019[1a0] = PCIe Sec 1 lane errors 0     ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled                      0 VFs configured out of 0 supported                      First VF RID Offset 0x0008, VF RID Stride 0x0004                      VF Device ID 0x5803                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304     ecap 0017[200] = TPH Requester 1 none7@pci0:2:0:5:       class=0x010000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5503 subvendor=0x1425 subdevice=0x0000     vendor     = 'Chelsio Communications Inc'     device     = 'T540-CR Unified Wire Storage Controller'     class      = mass storage     subclass   = SCSI     bar   [10] = type Memory, range 64, base 0x91280000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x91200000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0x91982000, size 8192, enabled     cap 01[40] = powerspec 3  supports D0 D3  current D0     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO                  max read 512                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)     cap 11[b0] = MSI-X supports 40 messages                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]     cap 03[d0] = VPD     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected     ecap 0003[170] = Serial 1 0000000000000000     ecap 000e[190] = ARI 1     ecap 0019[1a0] = PCIe Sec 1 lane errors 0     ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled                      0 VFs configured out of 0 supported                      First VF RID Offset 0x0008, VF RID Stride 0x0004                      VF Device ID 0x5803                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304     ecap 0017[200] = TPH Requester 1 none8@pci0:2:0:6:       class=0x0c0400 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5603 subvendor=0x1425 subdevice=0x0000     vendor     = 'Chelsio Communications Inc'     device     = 'T540-CR Unified Wire Storage Controller'     class      = serial bus     subclass   = Fibre Channel     bar   [10] = type Memory, range 64, base 0x91100000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x91080000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0x91980000, size 8192, enabled     cap 01[40] = powerspec 3  supports D0 D3  current D0     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO                  max read 512                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)     cap 11[b0] = MSI-X supports 40 messages                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]     cap 03[d0] = VPD     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected     ecap 0003[170] = Serial 1 0000000000000000     ecap 000e[190] = ARI 1     ecap 0019[1a0] = PCIe Sec 1 lane errors 0     ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled                      0 VFs configured out of 0 supported                      First VF RID Offset 0x0008, VF RID Stride 0x0004                      VF Device ID 0x5803                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304     ecap 0017[200] = TPH Requester 1 # git status On branch stable/13 Your branch is up to date with 'origin/stable/13'. nothing to commit, working tree clean 13.1-PRERELEASE FreeBSD 13.1-PRERELEASE #1 stable/13-226bb05eb: Mon Feb 28 15:03:59 EST 2022 Loader.conf is t5fw_cfg_load="YES" if_cxgbe_load="YES" hw.cxgbe.pause_settings="0" hw.cxgbe.attack_filter="1" hw.cxgbe.drop_pkts_with_l3_errors="1" hw.cxgbe.fcoecaps_allowed="0"     ---Mike