From nobody Thu Apr 20 19:59:23 2023
X-Original-To: virtualization@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q2T573lyFz46WTP
	for <virtualization@mlmmj.nyi.freebsd.org>; Thu, 20 Apr 2023 19:59:23 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Q2T5729nFz42cd
	for <virtualization@FreeBSD.org>; Thu, 20 Apr 2023 19:59:23 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1682020763;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding;
	bh=YJ92cWqyuC/1DHYkjF675h6TzoR3e9iAoP1YJTJdpXI=;
	b=pZJXaLwC1L6/YVjC7lv5BjoH0eehJKCrXybjK79Xbw5I6G9oNxVvQesP5hfeBAXKH8Jn1f
	PLWgM6Jxh8HG+FypLzeP6yHit6GDyfFYZB4DaHN+TbBQoqCOQ1frK9fbGD+GUsYSLv558t
	c1V+iYpO9/WXU4G5FNXcwyOswQ4D7RWC2e2cIJjpgQHSwIIjqZC1VMfOXNRCkxteLlis1v
	pt9dVAOMvNpf6hqHFlHaA4pu57jTdeDzoljsuPBHcQgS/GluDWiU2ZG9niAoxe8Fvg+Ktf
	LoP4mLvFW1m6cce64jxyEpszlOYSVAXpw1Z/+gFclKyLvP74RR7G3mCGPtzGgw==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682020763; a=rsa-sha256; cv=none;
	b=VkeSLp/52Y0Sqqhn2gbaHzm5guzgLGpCLYjgCIEQtN1l8EEVn17IxRzL82DUhe56l992Jj
	EOaKUib9WIue3Q4NjmZBiS8bL0r6GA6en/ZjdsmTcY2QuQDQCf8ga3P2sMu0EZJeLvdCX5
	u3yGaKsIIDerfUKz0trujhADWqxq0w17XFgWD9ww9cKdTjEPCZkCDivdTTlYrp2Fh4i+PI
	s6cr+k0/JyV7ab0N/xungz8ziqtGpZOG3RmVuTtERkYYF2oSKbLi6RypwSmrd6YUzvjGXh
	7iJmjkvDOE4B8eKS6t8gQe6vPGJvX588Mbg2yJmweW2j4edxoGiwlJgcJ/wuDw==
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Q2T571GgfzrRf
	for <virtualization@FreeBSD.org>; Thu, 20 Apr 2023 19:59:23 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 33KJxN7o097475
	for <virtualization@FreeBSD.org>; Thu, 20 Apr 2023 19:59:23 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 33KJxNM6097474
	for virtualization@FreeBSD.org; Thu, 20 Apr 2023 19:59:23 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: virtualization@FreeBSD.org
Subject: [Bug 270966] PCI passthru stops working after ~30 guest reboots
 (ivhd, ILLEGAL CMD, IO_PAGE_FAULT)
Date: Thu, 20 Apr 2023 19:59:23 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: new
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: bhyve
X-Bugzilla-Version: 13.2-RELEASE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: raul.munoz@custos.es
X-Bugzilla-Status: New
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: virtualization@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform
 op_sys bug_status bug_severity priority component assigned_to reporter
Message-ID: <bug-270966-27103@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Discussion <freebsd-virtualization.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization
List-Help: <mailto:virtualization+help@freebsd.org>
List-Post: <mailto:virtualization@freebsd.org>
List-Subscribe: <mailto:virtualization+subscribe@freebsd.org>
List-Unsubscribe: <mailto:virtualization+unsubscribe@freebsd.org>
Sender: owner-freebsd-virtualization@freebsd.org
X-BeenThere: freebsd-virtualization@freebsd.org
MIME-Version: 1.0
X-ThisMailContainsUnwantedMimeParts: N

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270966

            Bug ID: 270966
           Summary: PCI passthru stops working after ~30 guest reboots
                    (ivhd, ILLEGAL CMD, IO_PAGE_FAULT)
           Product: Base System
           Version: 13.2-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: bhyve
          Assignee: virtualization@FreeBSD.org
          Reporter: raul.munoz@custos.es

I've seen this problem (or related) since 13.1 using passthru/SRIOV and
somewhat complex configs. Setting a new server give me the chance to reduce=
 the
problem to this scenario. Reboot a bhyve guest using ppt nic, ~30 times.

fresh install on new server, using passthru with an integrated nic (bge)

With a script like:
[....]
#!/bin/sh
count=3D1
while :
do
  vm stop debian1
  sleep 5
  vm start debian1
  sleep 15
  echo "Reboots: $count"
  count=3D`expr $count + 1`
done
[....]

I see:
[....]
Sending ACPI shutdown to debian1=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20
Starting debian1=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  * found guest in /usr/local/vm-bhyve/debian1=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  * booting...=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
Reboots: 1=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
Sending ACPI shutdown to debian1=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20
Starting debian1=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  * found guest in /usr/local/vm-bhyve/debian1=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  * booting...=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
Reboots: 2=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
Sending ACPI shutdown to debian1=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20
Starting debian1=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  * found guest in /usr/local/vm-bhyve/debian1=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  * booting...=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20

... ... ...

Reboots: 27
Sending ACPI shutdown to debian1
Starting debian1
  * found guest in /usr/local/vm-bhyve/debian1
  ! guest appears to be running already
[....]

and messages show:
[....]
Apr 20 21:26:17 cache-D-2023 kernel: ivhd0: EVT INTR 0 Status:0xa EVT Head:=
0x0
Tail:0x10]
Apr 20 21:26:17 cache-D-2023 kernel:   [CMD Total 0x105] Tail:0x50, Head:0x=
30.
Apr 20 21:26:17 cache-D-2023 kernel: ivhd0:     [Event0: Head:0x0 Tail:0x10]
Apr 20 21:26:17 cache-D-2023 kernel:    [ILLEGAL CMD EVT]
Apr 20 21:26:17 cache-D-2023 kernel:    CMD opcode=3D 0x3 0xc009 0x14
0x7ffffffffffff003
Apr 20 21:26:18 cache-D-2023 kernel: ivhd0: Error: completion failed tail:0=
xe0,
head:0x30.
Apr 20 21:26:18 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:18 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:20 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x190, head:0x30.
Apr 20 21:26:20 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:20 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:29 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x240, head:0x30.
Apr 20 21:26:29 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:29 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:31 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x2f0, head:0x30.
Apr 20 21:26:31 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:31 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:32 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x3a0, head:0x30.
Apr 20 21:26:32 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:32 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:40 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x450, head:0x30.
Apr 20 21:26:40 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:40 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:41 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x500, head:0x30.
Apr 20 21:26:41 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:41 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:42 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x5b0, head:0x30.
Apr 20 21:26:42 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:42 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:51 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x660, head:0x30.
Apr 20 21:26:51 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:51 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:52 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x710, head:0x30.
Apr 20 21:26:52 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:52 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:26:53 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x7c0, head:0x30.
Apr 20 21:26:53 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:26:53 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:01 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x870, head:0x30.
Apr 20 21:27:01 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:01 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:02 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x920, head:0x30.
Apr 20 21:27:02 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:02 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:03 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x9d0, head:0x30.
Apr 20 21:27:03 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:03 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:12 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xa80, head:0x30.
Apr 20 21:27:12 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:12 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:13 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xb30, head:0x30.
Apr 20 21:27:13 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:13 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:14 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xbe0, head:0x30.
Apr 20 21:27:14 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:14 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:22 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xc90, head:0x30.
Apr 20 21:27:22 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:22 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:23 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xd40, head:0x30.
Apr 20 21:27:23 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:23 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:24 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xdf0, head:0x30.
Apr 20 21:27:24 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:24 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:33 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xea0, head:0x30.
Apr 20 21:27:33 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:33 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:34 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xf50, head:0x30.
Apr 20 21:27:34 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:34 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:35 cache-D-2023 kernel: ivhd0: Error: completion failed tail:0=
x0,
head:0x30.
Apr 20 21:27:35 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:35 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:38 cache-D-2023 kernel: ivhd0: EVT INTR 1 Status:0xa EVT Head:=
0x10
Tail:0x20]
Apr 20 21:27:38 cache-D-2023 kernel:   [CMD Total 0x200] Tail:0x0, Head:0x3=
0.
Apr 20 21:27:38 cache-D-2023 kernel: ivhd0:     [Event0: Head:0x10 Tail:0x2=
0]
Apr 20 21:27:38 cache-D-2023 kernel:    [IO_PAGE_FAULT EVT: devId:0xc301
DomId:0x17 Addr:0x101290000 0x0]
Apr 20 21:27:43 cache-D-2023 kernel: ivhd0: Error: completion failed tail:0=
xb0,
head:0x30.
Apr 20 21:27:43 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:43 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:44 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x160, head:0x30.
Apr 20 21:27:44 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:44 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:45 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x210, head:0x30.
Apr 20 21:27:45 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:45 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:55 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x2c0, head:0x30.
Apr 20 21:27:55 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:55 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:56 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x370, head:0x30.
Apr 20 21:27:56 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:56 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:27:57 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x420, head:0x30.
Apr 20 21:27:57 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:27:57 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:05 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x4d0, head:0x30.
Apr 20 21:28:05 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:05 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:06 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x580, head:0x30.
Apr 20 21:28:06 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:06 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:07 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x630, head:0x30.
Apr 20 21:28:07 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:07 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:16 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x6e0, head:0x30.
Apr 20 21:28:16 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:16 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:17 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x790, head:0x30.
Apr 20 21:28:17 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:17 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:18 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x840, head:0x30.
Apr 20 21:28:18 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:18 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:26 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x8f0, head:0x30.
Apr 20 21:28:26 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:26 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:27 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0x9a0, head:0x30.
Apr 20 21:28:27 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:27 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:28 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xa50, head:0x30.
Apr 20 21:28:28 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:28 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:37 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xb00, head:0x30.
Apr 20 21:28:37 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:37 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:38 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xbb0, head:0x30.
Apr 20 21:28:38 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:38 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:39 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xc60, head:0x30.
Apr 20 21:28:39 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:39 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:42 cache-D-2023 kernel: ivhd0: EVT INTR 2 Status:0xa EVT Head:=
0x20
Tail:0x30]
Apr 20 21:28:42 cache-D-2023 kernel:   [CMD Total 0x2c6] Tail:0xc60, Head:0=
x30.
Apr 20 21:28:42 cache-D-2023 kernel: ivhd0:     [Event0: Head:0x20 Tail:0x3=
0]
Apr 20 21:28:42 cache-D-2023 kernel:    [IO_PAGE_FAULT EVT: devId:0xc301
DomId:0x1a Addr:0x1074a0000 0x0]
Apr 20 21:28:47 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xd10, head:0x30.
Apr 20 21:28:47 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:47 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:49 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xdc0, head:0x30.
Apr 20 21:28:49 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:49 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:49 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xe70, head:0x30.
Apr 20 21:28:49 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:49 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:28:59 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xf20, head:0x30.
Apr 20 21:28:59 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:28:59 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:29:00 cache-D-2023 kernel: ivhd0: Error: completion failed
tail:0xfd0, head:0x30.
Apr 20 21:29:00 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:29:00 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
Apr 20 21:29:01 cache-D-2023 kernel: ivhd0: Error: completion failed tail:0=
x80,
head:0x30.
Apr 20 21:29:01 cache-D-2023 kernel: ivhd0: Dump last 1 command(s):
Apr 20 21:29:01 cache-D-2023 kernel:   [CMD0, off:0x20] opcode=3D 0x1 0x50a=
ebb1
0x0 0xa5a5
[....]

% uname -a
FreeBSD cache-D-2023 13.2-RELEASE FreeBSD 13.2-RELEASE
releng/13.2-n254617-525ecfdad597 GENERIC amd64

% dmesg | grep CPU:
CPU: AMD EPYC 9124 16-Core Processor                 (3000.16-MHz K8-class =
CPU)

% dmesg | grep ivhd
ivhd0: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
ivhd0: Flag:b0<IotlbSup,Coherent>
ivhd0: Features(type:0x11) MsiNumPPR =3D 0 PNBanks=3D 2 PNCounters=3D 0
ivhd0: Extended
features[31:0]:a2295afe<PPRSup,<b2>,NXSup,GTSup,<b5>,IASup,GASup,PCSup> HAT=
S =3D
0x2 GATS =3D 0x1 GLXSup =3D 0x1 SmiFSup =3D 0x1 SmiFRC =3D 0x2 GAMSup =3D 0=
x1
DualPortLogSup =3D 0x2 DualEventLogSup =3D 0x2
ivhd0: Extended
features[62:32]:25bf732f<USSup,PprOvrflwEarlySup,PPRAutoRspSup,BlKStopMrkSu=
p,PerfOptSup,MsiCapMmioSup,GIOSup,HASup,EPHSup,AttrFWSup,HDSup>
Max PASID: 0x2f DevTblSegSup =3D 0x2 MarcSup =3D 0x0
ivhd0: supported paging level:7, will use only: 4
ivhd0: device [0xc003 - 0xfffe] config:0
ivhd0: PCI cap 0x190b640f@0x40 feature:19<IOTLB,EFR,CapExt>
ivhd1: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
ivhd1: Flag:b0<IotlbSup,Coherent>
ivhd1: Features(type:0x11) MsiNumPPR =3D 0 PNBanks=3D 2 PNCounters=3D 0
ivhd1: Extended
features[31:0]:a2295afe<PPRSup,<b2>,NXSup,GTSup,<b5>,IASup,GASup,PCSup> HAT=
S =3D
0x2 GATS =3D 0x1 GLXSup =3D 0x1 SmiFSup =3D 0x1 SmiFRC =3D 0x2 GAMSup =3D 0=
x1
DualPortLogSup =3D 0x2 DualEventLogSup =3D 0x2
ivhd1: Extended
features[62:32]:25bf732f<USSup,PprOvrflwEarlySup,PPRAutoRspSup,BlKStopMrkSu=
p,PerfOptSup,MsiCapMmioSup,GIOSup,HASup,EPHSup,AttrFWSup,HDSup>
Max PASID: 0x2f DevTblSegSup =3D 0x2 MarcSup =3D 0x0
ivhd1: supported paging level:7, will use only: 4
ivhd1: device [0x8003 - 0xbffe] config:0
ivhd1: PCI cap 0x190b640f@0x40 feature:19<IOTLB,EFR,CapExt>
ivhd2: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
ivhd2: Flag:b0<IotlbSup,Coherent>
ivhd2: Features(type:0x11) MsiNumPPR =3D 0 PNBanks=3D 2 PNCounters=3D 0
ivhd2: Extended
features[31:0]:a2295afe<PPRSup,<b2>,NXSup,GTSup,<b5>,IASup,GASup,PCSup> HAT=
S =3D
0x2 GATS =3D 0x1 GLXSup =3D 0x1 SmiFSup =3D 0x1 SmiFRC =3D 0x2 GAMSup =3D 0=
x1
DualPortLogSup =3D 0x2 DualEventLogSup =3D 0x2
ivhd2: Extended
features[62:32]:25bf732f<USSup,PprOvrflwEarlySup,PPRAutoRspSup,BlKStopMrkSu=
p,PerfOptSup,MsiCapMmioSup,GIOSup,HASup,EPHSup,AttrFWSup,HDSup>
Max PASID: 0x2f DevTblSegSup =3D 0x2 MarcSup =3D 0x0
ivhd2: supported paging level:7, will use only: 4
ivhd2: device [0x3 - 0x3ffe] config:0
ivhd2: device [0xff00 - 0xffff] config:0
ivhd2: PCI cap 0x190b640f@0x40 feature:19<IOTLB,EFR,CapExt>
ivhd3: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
ivhd3: Flag:b0<IotlbSup,Coherent>
ivhd3: Features(type:0x11) MsiNumPPR =3D 0 PNBanks=3D 2 PNCounters=3D 0
ivhd3: Extended
features[31:0]:a2295afe<PPRSup,<b2>,NXSup,GTSup,<b5>,IASup,GASup,PCSup> HAT=
S =3D
0x2 GATS =3D 0x1 GLXSup =3D 0x1 SmiFSup =3D 0x1 SmiFRC =3D 0x2 GAMSup =3D 0=
x1
DualPortLogSup =3D 0x2 DualEventLogSup =3D 0x2
ivhd3: Extended
features[62:32]:25bf732f<USSup,PprOvrflwEarlySup,PPRAutoRspSup,BlKStopMrkSu=
p,PerfOptSup,MsiCapMmioSup,GIOSup,HASup,EPHSup,AttrFWSup,HDSup>
Max PASID: 0x2f DevTblSegSup =3D 0x2 MarcSup =3D 0x0
ivhd3: supported paging level:7, will use only: 4
ivhd3: device [0x4003 - 0x7ffe] config:0
ivhd3: PCI cap 0x190b640f@0x40 feature:19<IOTLB,EFR,CapExt>

I'll try on 14.
Thanks in advance.

--=20
You are receiving this mail because:
You are the assignee for the bug.=