From nobody Wed Jul 12 11:48:01 2023
X-Original-To: bugs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R1GFt42Tdz4lTLS
for <bugs@mlmmj.nyi.freebsd.org>; Wed, 12 Jul 2023 11:48:02 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4R1GFs2vw3z44Bt
for <bugs@FreeBSD.org>; Wed, 12 Jul 2023 11:48:01 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
s=dkim; t=1689162481;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding;
bh=26Nmsi2z63mmwH0ZWdiKapH5iWlJt8ecnwW/tjjdJcs=;
b=IqIuoDBM6bjbOpmbNaKEmPfRl3bFB5rNXw0/11ZUhO3vM77VLC0UL50e8BOZWOd8Koh3Xc
1/C1B7vXhzrA/6SG36Qgvboj0EFxiHhQge4kfx+kE6Ar4J2EPREikyyhulRhr9Qrkk9c5R
3hehwLy0zhz9LdaMJdTrZv/SlrxMwWtfhMLHdHhIgol+TbDUJOq458No2GXHdY7WOz7MmA
823vihhruLPJ8/GQ2uwAVB2H6BoAmA3PTK9fACkZbwFZ6BF9j6H4Czz3NRMwzqxedczFw3
xcFNXniaExDePPnt1YY4aHD0PjxoVMeVphkK2DB/YyNC9WCL0hx0l9o8L00BwQ==
ARC-Authentication-Results: i=1;
mx1.freebsd.org;
none
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1689162481; a=rsa-sha256; cv=none;
b=cKr+3k+RhGQip/DTc6pk3So/9M7OU3GzgH6bs9OZLDok3Nrtx2VBJR0p3WtVO2EFKANYVJ
HAOj9z/fd9EOiUDXz82jfX+PshUj/uRqtknp/HWYpFs8eDgI5V5wHRRW+FTwkV/kJD1eU3
vLgIBV0zuws101JQEK26/n7eFWcsq/AUTd4la7jOFJJItjmPu+XI7TM2nf/dKLvUe8wJZo
VPprpPwHTlhX7P4dXSbOqj/1tlmm/Oh3mrKnyLaW1+9NT6cTcUdIh1DRRl5/glnfNUtueD
xlN9Mjke2v4Ydyxxqy41GYtTFsDljxDrOAEldxyTdi6zAS+GXfXVAwnN48Zi+Q==
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4R1GFs1vp5z171n
for <bugs@FreeBSD.org>; Wed, 12 Jul 2023 11:48:01 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 36CBm1Nt050699
for <bugs@FreeBSD.org>; Wed, 12 Jul 2023 11:48:01 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 36CBm1mt050698
for bugs@FreeBSD.org; Wed, 12 Jul 2023 11:48:01 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: bugs@FreeBSD.org
Subject: [Bug 272469] Broadcom mpi3mr driver: MSIX allocation fail on DELL
PowerEdge R7625 system
Date: Wed, 12 Jul 2023 11:48:01 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: new
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 13.2-RELEASE
X-Bugzilla-Keywords:
X-Bugzilla-Severity: Affects Many People
X-Bugzilla-Who: chandrakanth.patil@broadcom.com
X-Bugzilla-Status: New
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: bugs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform
op_sys bug_status bug_severity priority component assigned_to reporter
attachments.created
Message-ID: <bug-272469-227@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Bug reports <freebsd-bugs.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-bugs
List-Help: <mailto:freebsd-bugs+help@freebsd.org>
List-Post: <mailto:freebsd-bugs@freebsd.org>
List-Subscribe: <mailto:freebsd-bugs+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-bugs+unsubscribe@freebsd.org>
Sender: owner-freebsd-bugs@freebsd.org
MIME-Version: 1.0
X-ThisMailContainsUnwantedMimeParts: N
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D272469
Bug ID: 272469
Summary: Broadcom mpi3mr driver: MSIX allocation fail on DELL
PowerEdge R7625 system
Product: Base System
Version: 13.2-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Many People
Priority: ---
Component: kern
Assignee: bugs@FreeBSD.org
Reporter: chandrakanth.patil@broadcom.com
Created attachment 243352
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D243352&action=
=3Dedit
msix_table_dump
mpi3mr avenger driver:
system details:=20
1. Dell PowerEdge R7625 with 196 physical cores and 256 logical cores
mpi3mr driver will allocate the single msix for handshaking with the driver
during
the initial load phase using pci_alloc_msix() API. After allocating the sin=
gle
msix, the driver is sending the get IOC_FACTS commands to firmware through
which the driver will fetch all the controller properties. The issue is the
driver is not getting the interrupt for IOC_FACTS completion leads to timeo=
ut
which in turn leads to driver load failure. but the driver can see that the
command is completed by the firmware if it polls the reply queue.
After creating the single msix in the driver, the vmstat -i in the OS should
show the interrupt but it is not showing so the interrupt binding is failin=
g.
ideally in this case the pci_alloc_msix() API should throw some error during
allocation but it is not throwing any error.
Note:=20
1. This issue is happening only on this specific server where the numb=
er
of=20
CPUs are > 128 (total CPUs are 256).
2. But when we reduce the number of cores to 24 in the BIOS then the
driver=20
is working without any issues.=20
We have dumped the MSIX table before and after the allocation of a single m=
six
and after the command times out. Please find it in the attachment.
I wanted to understand if is there any OS limitation w.r.t MSIX allocation =
on
larger cores system.
Please find attached driver logs and MSIX table dump.
--=20
You are receiving this mail because:
You are the assignee for the bug.=