From nobody Tue Apr 16 17:52:09 2024 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VJs7G5nMgz5HG9w for ; Tue, 16 Apr 2024 17:52:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VJs7G3yhZz4gYF for ; Tue, 16 Apr 2024 17:52:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1713289930; a=rsa-sha256; cv=none; b=XnBwQ6CZJqtQkATmUoRKy3L4R14ipuFK7VqKIqRAWyDWzajlDlQbuEdg5vF5XfZDbJ7Ath B6T2G/wt6kseNuRtpELQq0bKInHr+o6HiLtWr/VXz4NRRjRxm1jq+I25SfQ/uKBho2BZgV hQXvbldszf/QOMROJ+yMOgsfUyOHd/4TvCBE6Uwpuky+1o5xqwgVFYof0dPO0mFk2bCYJL goHw5o22CnkfYTAIB5TF5N3u6WoekqRz4OWYOSOyFUrgKPyzWfIRSjWQXjEsEsDyBCyJwA n+U9TkhFa3C1hkL9litIOmBgsta9ASv4o0XsIUPq5wDAxaErHZp4dmkhoVh+Ig== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1713289930; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o5y74Rxgjz3NW8OBp+EQMLzT9mxP3hEkmvbFvPOIGnw=; b=Xh5orJMGhsg+5whxdVsY4eU/agMNNEGTf2BuG/XDPSsIu7wJaoZ95mi7/gU5zOmwcldVnX 2/+YXBK+ra0a9rHs7hgefFuM/XX2Vty+IVDDrpvbaaWATvYa4p4LNnZMJ0TulcGEY5YQdl aasJ7UeV0/qTOTPQ1vQouziochXTP79ua6ni0CRKzX+AwOG+jmEr/vI/ff/4mv8nHUS3gM VN1hNrMRew11E/8exMsBmSw0q4enhGrJXR5O39cEE+Kkne7wgZsXuV1Pg0ql2eh9ws+8XM /CrnkUGMgKsr3Kaa1iPLhy5uyYIyG3qHht6KpOO8TdULOTGTwgFQutn7/eNjbg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VJs7G3bKLzsvB for ; Tue, 16 Apr 2024 17:52:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 43GHqAKv007563 for ; Tue, 16 Apr 2024 17:52:10 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 43GHqArr007560 for bugs@FreeBSD.org; Tue, 16 Apr 2024 17:52:10 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 269133] bnxt(4): BCM57416 - HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error Date: Tue, 16 Apr 2024 17:52:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ken@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D269133 Kenneth D. Merry changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ken@FreeBSD.org --- Comment #64 from Kenneth D. Merry --- I have a machine with a BCM57414 on the motherboard: bnxt0: mem 0xd1d10000-0xd1d1ffff,0xd1c00000-0xd1cfffff,0xd1d22000-0xd1d23fff irq 36 at device 0.0 numa-domain 0 on pci3 bnxt0: Using 256 TX descriptors and 256 RX descriptors bnxt0: Using 16 RX queues 16 TX queues bnxt0: Using MSI-X interrupts with 17 vectors bnxt0: Ethernet address: 9c:6b:00:46:a2:0c bnxt1: mem 0xd1d00000-0xd1d0ffff,0xd1b00000-0xd1bfffff,0xd1d20000-0xd1d21fff irq 37 at device 0.1 numa-domain 0 on pci3 bnxt1: Using 256 TX descriptors and 256 RX descriptors bnxt1: Using 16 RX queues 16 TX queues bnxt1: Using MSI-X interrupts with 17 vectors bnxt1: Ethernet address: 9c:6b:00:46:a2:0d Here is the firmware information: dev.bnxt.0.nvram.available_size: 4173824 dev.bnxt.0.nvram.reserved_size: 16384 dev.bnxt.0.nvram.size: 8388608 dev.bnxt.0.nvram.sector_size: 4096 dev.bnxt.0.nvram.device_id: 16407 dev.bnxt.0.nvram.mfg_id: 239 dev.bnxt.0.ver.hwrm_min_ver: 1.10.2 dev.bnxt.0.ver.package_ver: dev.bnxt.0.ver.chip_type: ASIC dev.bnxt.0.ver.chip_bond_id: 0 dev.bnxt.0.ver.chip_metal: 1 dev.bnxt.0.ver.chip_rev: 1 dev.bnxt.0.ver.chip_num: 5847 dev.bnxt.0.ver.phy_partnumber: MCP7F00-A002 dev.bnxt.0.ver.phy_vendor: Mellanox dev.bnxt.0.ver.roce_fw_name: BONO_FW dev.bnxt.0.ver.netctrl_fw_name: KONG_FW dev.bnxt.0.ver.mgmt_fw_name: AFW_226.0.145.0 dev.bnxt.0.ver.hwrm_fw_name: CHIMP_FW dev.bnxt.0.ver.phy: 13.1.11 dev.bnxt.0.ver.fw_ver: 226.0.145.0/pkg N/A dev.bnxt.0.ver.roce_fw: 226.0.145 dev.bnxt.0.ver.netctrl_fw: 226.0.145 dev.bnxt.0.ver.mgmt_fw: 226.0.145 dev.bnxt.0.ver.hwrm_fw: 226.0.145 dev.bnxt.0.ver.driver_hwrm_if: 1.10.2.34 dev.bnxt.0.ver.hwrm_if: 1.10.2 dev.bnxt.0.%domain: 0 dev.bnxt.0.%parent: pci3 dev.bnxt.0.%pnpinfo: vendor=3D0x14e4 device=3D0x16d7 subvendor=3D0x1849 subdevice=3D0x1402 class=3D0x020000 dev.bnxt.0.%location: slot=3D0 function=3D0 dbsf=3Dpci0:195:0:0 dev.bnxt.0.%driver: bnxt dev.bnxt.0.%desc: Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet I have no VLANs configured. I'm running stable/13 from mid-2023, but I've tried the driver from the latest FreeBSD/head and FreeBSD stable/13 with no success. I still get: bnxt0: HWRM_RING_ALLOC command returned RESOURCE_ALLOC_ERROR error. bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: Link is UP full duplex, FC - none - 25000 Mbps=20 bnxt0: link state changed to UP bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt1: Link is UP full duplex, FC - none - 25000 Mbps=20 bnxt1: link state changed to UP bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed Strangely enough, though, the driver works fine if I first PXE boot the mac= hine off of the chip. If I do that, it works normally. But if I boot off disk,= I get the RESOURCE_ALLOC_ERROR messages above. This suggests that there is some kind of initialization issue that the PXE = boot environment takes care of, but the driver does not. Also, bnxtnvm and niccli don't work with the driver in my kernel. But it isn't, apparently, because of the state of the driver, it is because of the ioctl definition in the driver. The ioctl call doesn't even get to bnxt_mgmt_ioctl. I've verified that with dtrace, but here is the ioctl call from ktrace/kdump: 7818 bnxtnvm CALL openat(AT_FDCWD,0x46199d,0x2) 7818 bnxtnvm NAMI "/dev/bnxt_mgmt" 7818 bnxtnvm RET openat 4 7818 bnxtnvm CALL ioctl(0x4,0x80000000,0x821199a70) 7818 bnxtnvm RET ioctl -1 errno 25 Inappropriate ioctl for device 7818 bnxtnvm CALL close(0x4) Using this Dtrace script: #pragma D option cleanrate=3D5000hz #pragma D option dynvarsize=3D8192000 fbt::bnxt_mgmt_ioctl:entry { printf("cmd =3D %#x\n", args[1]); } fbt::bnxt_mgmt_open:entry { printf("opened bnxt mgmt device\n"); } fbt::sys_ioctl:entry /args[1]->com =3D=3D 0x80000000/ { printf("got ioctl command 0x80000000\n"); } I verified that it isn't getting down to bnxt_mgmt_ioctl: # dtrace -s bnxt.d=20 dtrace: script 'bnxt.d' matched 3 probes CPU ID FUNCTION:NAME 31 2108 bnxt_mgmt_open:entry opened bnxt mgmt device 31 22882 sys_ioctl:entry got ioctl command 0x80000000 31 2108 bnxt_mgmt_open:entry opened bnxt mgmt device 31 22882 sys_ioctl:entry got ioctl command 0x80000000 ^C When I boot the machine via PXE, though, bnxtnvm listdev shows the device: # ./bnxtnvm listdev N/A #1 Device Interface Name : bnxt0 MACAddress : 9c:6b:00:46:a2:0c PCI Device Name : 0000:c3:00.0 And strangely the ioctl works, although from my reading of sys_ioctl(), it shouldn't but I think I've discovered why it does: 1904 bnxtnvm CALL openat(AT_FDCWD,0x46199d,0x2) 1904 bnxtnvm NAMI "/dev/bnxt_mgmt" 1904 bnxtnvm RET openat 3 1904 bnxtnvm CALL ioctl(0x3,0x80000000,0x8212303d0) 1904 bnxtnvm RET ioctl 0 1904 bnxtnvm CALL close(0x3) This code is from sys_ioctl(): /* * Interpret high order word to find amount of data to be * copied to/from the user's address space. */ size =3D IOCPARM_LEN(com); if ((size > IOCPARM_MAX) || ((com & (IOC_VOID | IOC_IN | IOC_OUT)) =3D=3D 0) || #if defined(COMPAT_FREEBSD5) || defined(COMPAT_FREEBSD4) || defined(COMPAT_= 43) ((com & IOC_OUT) && size =3D=3D 0) || #else ((com & (IOC_IN | IOC_OUT)) && size =3D=3D 0) || #endif ((com & IOC_VOID) && size > 0 && size !=3D sizeof(int))) return (ENOTTY); My regular kernel config file doesn't have COMPAT_FREEBSD4/5, but the PXE kernel config file does. Here are the bit definitions from sys/ioccom.h: #ifndef _SYS_IOCCOM_H_ #define _SYS_IOCCOM_H_ /* * Ioctl's have the command encoded in the lower word, and the size of * any in or out parameters in the upper word. The high 3 bits of the * upper word are used to encode the in/out status of the parameter. * * 31 29 28 16 15 8 7 0 * +---------------------------------------------------------------+ * | I/O | Parameter Length | Command Group | Command | * +---------------------------------------------------------------+ */ #define IOCPARM_SHIFT 13 /* number of bits for ioctl size */ #define IOCPARM_MASK ((1 << IOCPARM_SHIFT) - 1) /* parameter length mask= */ #define IOCPARM_LEN(x) (((x) >> 16) & IOCPARM_MASK) #define IOCBASECMD(x) ((x) & ~(IOCPARM_MASK << 16)) #define IOCGROUP(x) (((x) >> 8) & 0xff) #define IOCPARM_MAX (1 << IOCPARM_SHIFT) /* max size of ioctl */ #define IOC_VOID 0x20000000UL /* no parameters */ #define IOC_OUT 0x40000000UL /* copy out parameters */ #define IOC_IN 0x80000000UL /* copy in parameters */ #define IOC_INOUT (IOC_IN|IOC_OUT)/* copy parameters in and out */ #define IOC_DIRMASK (IOC_VOID|IOC_OUT|IOC_IN)/* mask for IN/OUT/VOID */ Because the BNXT_MGMT_OPCODE_GET_DEV_INFO ioctl the same as the IOC_IN bit definition, the ioctl breaks if the old compat stuff isn't built into the kernel. The ioctls for the bnxt(4) driver need to be changed to use the usual _IOW/_IOWR macros from sys/ioccom.h. I realize that will break the managem= ent tools. Perhaps they can have a version check and a fallback to the old ioc= tls if need be. --=20 You are receiving this mail because: You are the assignee for the bug.=