From owner-freebsd-virtualization@freebsd.org Tue Oct 22 18:41:47 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8D56E1620F0 for ; Tue, 22 Oct 2019 18:41:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46yMmW3BTcz48mn for ; Tue, 22 Oct 2019 18:41:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 6BA5A1620EF; Tue, 22 Oct 2019 18:41:47 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6B60C1620EE for ; Tue, 22 Oct 2019 18:41:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46yMmW2FpKz48ml for ; Tue, 22 Oct 2019 18:41:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 321A4188FD for ; Tue, 22 Oct 2019 18:41:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x9MIflrp096054 for ; Tue, 22 Oct 2019 18:41:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x9MIflgO096053 for virtualization@FreeBSD.org; Tue, 22 Oct 2019 18:41:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 234838] ena drop-outs on 12.0-RELEASE Date: Tue, 22 Oct 2019 18:41:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: langseth@iteris.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Feedback Timeout X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Oct 2019 18:41:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234838 Ryan Langseth changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |langseth@iteris.com --- Comment #6 from Ryan Langseth --- It seems like there is still an issue with this. We are running FreeBSD 12.0-RELEASE-p10 on a c5.2xl instance. And have had the system reset the network device twice in the last 24 hours. The current traffic to it is a zfs recv over ssh running at ~40MiB/s. The f= irst time it dropped out it was just the ena device. The second time I also got = nvme 'Missing Interrupts' as well. The system has 6 gp2 volumes for the zpool. `grep kern.crit /var/log/messages` Oct 21 12:26:52 apache-00 kernel: Trying to mount root from ufs:/dev/gpt/rootfs [rw]... Oct 21 12:26:52 apache-00 kernel: ena0: device is going UP Oct 21 12:26:52 apache-00 kernel: ena0: device is going DOWN Oct 21 12:26:52 apache-00 kernel: ena0: device is going UP Oct 21 12:26:52 apache-00 kernel: intsmb0: port 0xb100-0xb10f at device 1.3 on pci0 Oct 21 12:26:52 apache-00 kernel: intsmb0: intr IRQ 9 enabled revision 255 Oct 21 12:26:52 apache-00 kernel: smbus0: on intsmb0 Oct 21 12:26:53 apache-00 kernel: Security policy loaded: MAC/n= tpd (mac_ntpd) Oct 22 06:33:55 apache-00 kernel: ena0: The number of lost tx completion is above the threshold (129 > 128). Reset the device Oct 22 06:33:55 apache-00 kernel: ena0: Trigger reset is on Oct 22 06:33:55 apache-00 kernel: ena0: device is going DOWN Oct 22 06:34:02 apache-00 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x1f2 Oct 22 06:34:03 apache-00 kernel: ena0: ena0: device is going UP Oct 22 06:34:03 apache-00 kernel: link is UP Oct 22 13:18:10 apache-00 kernel: ena0: The number of lost tx completion is above the threshold (129 > 128). Reset the device Oct 22 13:18:10 apache-00 kernel: ena0: Trigger reset is on Oct 22 13:18:10 apache-00 kernel: ena0: device is going DOWN Oct 22 13:18:16 apache-00 kernel: ena0: free uncompleted tx mbuf qid 4 idx 0x3a6 Oct 22 13:18:16 apache-00 kernel:=20 Oct 22 13:18:16 apache-00 kernel: ena0: device is going UP Oct 22 13:18:16 apache-00 kernel: ena0: link is UP Oct 22 13:18:47 apache-00 kernel: nvme0: Missing interrupt Oct 22 13:18:51 apache-00 kernel: nvme5: Missing interrupt Oct 22 13:18:51 apache-00 kernel: nvme2: Missing interrupt Oct 22 13:19:17 apache-00 kernel: nvme0: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme2: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme5: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme0: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme5: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme6: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme2: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme0: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme5: Missing interrupt Oct 22 13:20:47 apache-00 kernel: nvme0: Missing interrupt Oct 22 13:22:17 apache-00 kernel: nvme4: Missing interrupt Oct 22 13:22:17 apache-00 kernel: nvme2: Missing interrupt Oct 22 13:22:17 apache-00 kernel: nvme5: Missing interrupt Oct 22 13:22:17 apache-00 kernel: nvme0: Missing interrupt Oct 22 13:22:17 apache-00 kernel: nvme5: nvme2: nvme4: Missing interrupt Oct 22 13:22:17 apache-00 kernel: Missing interrupt Oct 22 13:22:17 apache-00 syslogd: last message repeated 1 times Oct 22 13:22:17 apache-00 kernel: nvme0: Missing interrupt Oct 22 13:22:17 apache-00 kernel: nvme2: Missing interrupt Oct 22 13:22:17 apache-00 kernel: nvme0:=20 Oct 22 13:22:17 apache-00 kernel:=20 Oct 22 13:22:17 apache-00 kernel: Missing interrupt Oct 22 13:22:21 apache-00 kernel: nvme2: nvme6: Missing interru= pt Oct 22 13:22:21 apache-00 kernel: nvme5: Missing interrupt Oct 22 13:22:21 apache-00 kernel: Missing interrupt Oct 22 13:22:21 apache-00 kernel: nvme4: Missing interrupt Oct 22 13:22:51 apache-00 kernel: nvme6: nvme4: Missing interru= pt Oct 22 13:22:51 apache-00 kernel: Missing interrupt Oct 22 13:22:51 apache-00 kernel: nvme5: Missing interrupt Oct 22 13:22:51 apache-00 kernel: nvme2: Missing interrupt Oct 22 13:23:16 apache-00 kernel: nvme0: Missing interrupt Oct 22 13:23:21 apache-00 kernel: nvme2: Missing interrupt Oct 22 13:23:21 apache-00 kernel: nvme6: Missing interrupt Oct 22 13:23:26 apache-00 kernel: nvme4: Missing interrupt I will add that this instance was originally a FreeBSD 11.x system that was freebsd-update'd to 12. As a 11 system it was panicing on the transfer every 3-4 hours. I am bringing up a fresh 12.x system to do additional testing. --=20 You are receiving this mail because: You are the assignee for the bug.=