From owner-freebsd-virtualization@freebsd.org Thu Apr 2 22:01:54 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CABA52678A4 for ; Thu, 2 Apr 2020 22:01:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48tcV91dnXz48Tp for ; Thu, 2 Apr 2020 22:01:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 44B3C26789A; Thu, 2 Apr 2020 22:01:47 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3E721267899 for ; Thu, 2 Apr 2020 22:01:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48tcV20z50z48Rh for ; Thu, 2 Apr 2020 22:01:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 743A820BE5 for ; Thu, 2 Apr 2020 22:01:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 032M1dMv075113 for ; Thu, 2 Apr 2020 22:01:39 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 032M1dv7075112 for virtualization@FreeBSD.org; Thu, 2 Apr 2020 22:01:39 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Thu, 02 Apr 2020 22:01:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2020 22:01:55 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #46 from mail@rubenvos.com --- (In reply to Colin Percival from comment #45) Hi Colin, Well, another crash this morning. A coworker also had to reboot the instance again. There are a lot of messages regarding ENA in the logs. This is /var/log/messages from the event: Apr 1 13:33:45 zfs01 ntpd[73646]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): expired less than 96 days ago Apr 2 03:04:51 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:04:58 zfs01 syslogd: last message repeated 1 times Apr 2 03:05:12 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Apr 2 03:05:12 zfs01 kernel: ena0: Trigger reset is on Apr 2 03:05:12 zfs01 kernel: ena0: device is going DOWN Apr 2 03:05:13 zfs01 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x25d Apr 2 03:05:14 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 03:05:14 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 03:05:14 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 03:05:14 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 03:05:14 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 03:05:14 zfs01 kernel: ena0: device is going UP Apr 2 03:05:14 zfs01 kernel: ena0: link is UP Apr 2 03:05:26 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:05:40 zfs01 syslogd: last message repeated 1 times Apr 2 03:09:20 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Apr 2 03:09:20 zfs01 kernel: ena0: Trigger reset is on Apr 2 03:09:20 zfs01 kernel: ena0: device is going DOWN Apr 2 03:09:20 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:09:20 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:09:20 zfs01 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x25bena0: free uncompleted tx mbuf qid 1 idx 0x2ad Apr 2 03:09:21 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 03:09:21 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 03:09:21 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 03:09:21 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 03:09:21 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 03:09:21 zfs01 kernel: ena0: device is going UP Apr 2 03:09:21 zfs01 kernel: ena0: link is UP Apr 2 03:11:29 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:11:40 zfs01 kernel: nvme1: cpl does not map to outstanding cmd Apr 2 03:11:40 zfs01 kernel: cdw0:00000000 sqhd:0013 sqid:0001 cid:001c p:0 sc:00 sct:0 m:0 dnr:0 Apr 2 03:11:40 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:11:40 zfs01 kernel: nvme1: Resetting controller due to a timeout. Apr 2 03:11:40 zfs01 kernel: nvme1: resetting controller Apr 2 03:11:40 zfs01 kernel: nvme1: temperature threshold not supported Apr 2 03:11:40 zfs01 kernel: nvme1: aborting outstanding i/o Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14013800= 00 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14013808= 00 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14013815= 76 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14013824= 00 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014058= 64 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014082= 64 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014127= 20 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014132= 40 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014145= 04 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:19386367= 92 len:8 Apr 2 03:11:46 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:11:56 zfs01 kernel: nvme1: Resetting controller due to a timeout. Apr 2 03:11:56 zfs01 kernel: nvme1: resetting controller Apr 2 03:11:56 zfs01 kernel: nvme1: temperature threshold not supported Apr 2 03:11:56 zfs01 kernel: nvme1: aborting outstanding i/o Apr 2 03:11:56 zfs01 syslogd: last message repeated 10 times Apr 2 03:12:02 zfs01 kernel: nvme1: WRITE sqid:1 cid:5 nsid:1 lba:3847102 len:64 Apr 2 03:12:02 zfs01 kernel: nvme1: INVALID OPCODE (00/01) sqid:1 cid:28 cdw0:0 Apr 2 03:12:02 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:14:18 zfs01 kernel: ena0: Keep alive watchdog timeout. Apr 2 03:14:18 zfs01 kernel: ena0: Trigger reset is on Apr 2 03:14:18 zfs01 kernel: ena0: device is going DOWN Apr 2 03:15:36 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:15:36 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:15:39 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 03:15:39 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 03:15:39 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 03:15:39 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 03:15:39 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 03:15:39 zfs01 kernel: ena0: ena0: device is going UP Apr 2 03:15:39 zfs01 kernel: link is UP Apr 2 03:17:22 zfs01 dhclient[69443]: send_packet: Network is down Apr 2 03:17:40 zfs01 syslogd: last message repeated 4 times Apr 2 03:17:44 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:17:44 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:17:50 zfs01 dhclient[69443]: send_packet: Network is down Apr 2 03:18:21 zfs01 syslogd: last message repeated 1 times Apr 2 03:19:37 zfs01 syslogd: last message repeated 2 times Apr 2 03:19:39 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:19:39 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:23:20 zfs01 kernel: ena0: Keep alive watchdog timeout. Apr 2 03:23:20 zfs01 kernel: ena0: Trigger reset is on Apr 2 03:23:20 zfs01 kernel: ena0: device is going DOWN Apr 2 03:23:20 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:23:20 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:23:23 zfs01 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x2ac Apr 2 03:23:24 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 03:23:24 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 03:23:24 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 03:23:24 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 03:23:24 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 03:23:24 zfs01 kernel: ena0: device is going UP Apr 2 03:23:24 zfs01 kernel: ena0: link is UP Apr 2 03:25:09 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:25:09 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:27:00 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:27:00 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:28:49 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:28:49 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:37:52 zfs01 nrpe[41394]: Could not read request from client 172.28.8.16, bailing out... Apr 2 04:43:45 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Apr 2 04:43:45 zfs01 kernel: ena0: Trigger reset is on Apr 2 04:43:45 zfs01 kernel: ena0: device is going DOWN Apr 2 04:44:22 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:44:22 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:44:25 zfs01 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x28f Apr 2 04:44:26 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 04:44:26 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 04:44:26 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 04:44:26 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 04:44:26 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 04:44:26 zfs01 kernel: stray irq260 Apr 2 04:44:26 zfs01 kernel: ena0: ena0: device is going UP Apr 2 04:44:26 zfs01 kernel: link is UP Apr 2 04:46:33 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:46:33 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:48:36 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:48:36 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:50:25 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:50:25 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:52:37 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:52:37 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:54:40 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:54:40 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:09:52 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Apr 2 05:09:52 zfs01 kernel: ena0: Trigger reset is on Apr 2 05:09:52 zfs01 kernel: ena0: device is going DOWN Apr 2 05:11:03 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:11:03 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:11:04 zfs01 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x22b Apr 2 05:11:05 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 05:11:05 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 05:11:05 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 05:11:05 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 05:11:05 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 05:11:05 zfs01 kernel: stray irq260 Apr 2 05:11:05 zfs01 kernel: ena0: device is going UP Apr 2 05:11:05 zfs01 kernel: ena0: link is UP Apr 2 05:13:00 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:13:00 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:14:57 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:14:57 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:16:46 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:16:46 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:18:39 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:18:39 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:20:41 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:20:41 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:22:33 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:22:33 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:28:52 zfs01 kernel: ena0: Keep alive watchdog timeout. Apr 2 05:28:52 zfs01 kernel: ena0: Trigger reset is on Apr 2 05:28:52 zfs01 kernel: ena0: device is going DOWN Apr 2 05:28:52 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:28:52 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:29:24 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 05:29:24 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 05:29:24 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 05:29:24 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 05:29:24 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 05:29:24 zfs01 kernel: stray irq260 Apr 2 05:29:24 zfs01 kernel: ena0: ena0: device is going UP Apr 2 05:29:24 zfs01 kernel: link is UP Apr 2 05:57:05 zfs01 syslogd: kernel boot file is /boot/kernel/kernel Apr 2 05:57:05 zfs01 kernel: pflog0: promiscuous mode disabled Apr 2 05:57:05 zfs01 kernel:=20 Apr 2 05:57:05 zfs01 syslogd: last message repeated 1 times Apr 2 05:57:05 zfs01 kernel: Fatal trap 12: page fault while in kernel mode Apr 2 05:57:05 zfs01 kernel: cpuid =3D 0; apic id =3D 00 Apr 2 05:57:05 zfs01 kernel: fault virtual address =3D 0x78 Apr 2 05:57:05 zfs01 kernel: fault code =3D supervisor read= data, page not present Apr 2 05:57:05 zfs01 kernel: instruction pointer =3D 0x20:0xffffffff80b17c97 Apr 2 05:57:05 zfs01 kernel: stack pointer =3D 0x28:0xfffffe000045d1e0 Apr 2 05:57:05 zfs01 kernel: frame pointer =3D 0x28:0xfffffe000045d210 Apr 2 05:57:05 zfs01 kernel: code segment =3D base 0x0, limit 0xfffff, type 0x1b Apr 2 05:57:05 zfs01 kernel: =3D DPL 0, pres 1, long 1, = def32 0, gran 1 Apr 2 05:57:05 zfs01 kernel: processor eflags =3D interrupt enabled, resu= me, IOPL =3D 0 Apr 2 05:57:05 zfs01 kernel: current process =3D 12 (irq257: nvme0:io0) Apr 2 05:57:05 zfs01 kernel: trap number =3D 12 Apr 2 05:57:05 zfs01 kernel: panic: page fault Apr 2 05:57:05 zfs01 kernel: cpuid =3D 0 Apr 2 05:57:05 zfs01 kernel: time =3D 1585806979 Apr 2 05:57:05 zfs01 kernel: KDB: stack backtrace: Apr 2 05:57:05 zfs01 kernel: #0 0xffffffff80c1d2b7 at kdb_backtrace+0x67 Apr 2 05:57:05 zfs01 kernel: #1 0xffffffff80bd05ed at vpanic+0x19d Apr 2 05:57:05 zfs01 kernel: #2 0xffffffff80bd0443 at panic+0x43 Apr 2 05:57:05 zfs01 kernel: #3 0xffffffff810a7dcc at trap_fatal+0x39c Apr 2 05:57:05 zfs01 kernel: #4 0xffffffff810a7e19 at trap_pfault+0x49 Apr 2 05:57:05 zfs01 kernel: #5 0xffffffff810a740f at trap+0x29f Apr 2 05:57:05 zfs01 kernel: #6 0xffffffff81081a2c at calltrap+0x8 Apr 2 05:57:05 zfs01 kernel: #7 0xffffffff8079c5ac at nvme_qpair_complete_tracker+0x1bc Apr 2 05:57:05 zfs01 kernel: #8 0xffffffff8079c2c4 at nvme_qpair_process_completions+0xd4 Apr 2 05:57:05 zfs01 kernel: #9 0xffffffff80b93dd4 at ithread_loop+0x1d4 Apr 2 05:57:05 zfs01 kernel: #10 0xffffffff80b90c43 at fork_exit+0x83 Apr 2 05:57:05 zfs01 kernel: #11 0xffffffff81082a6e at fork_trampoline+0xe Apr 2 05:57:05 zfs01 kernel: Uptime: 20d16h39m50s Apr 2 05:57:05 zfs01 kernel: Rebooting... Apr 2 05:57:05 zfs01 kernel: ---<>--- This occured only minutes again after running the periodic framework: root@zfs01:~ # ls -lahtuT /etc/periodic/daily/ total 128 -rwxr-xr-x 1 root wheel 451B Apr 2 03:02:09 2020 480.leapfile-ntpd -rwxr-xr-x 1 root wheel 2.0K Apr 2 03:01:58 2020 460.status-mail-rejec= ts -rwxr-xr-x 1 root wheel 1.0K Apr 2 03:01:00 2020 450.status-security -rwxr-xr-x 1 root wheel 1.4K Apr 2 03:01:00 2020 440.status-mailq -rwxr-xr-x 1 root wheel 705B Apr 2 03:01:00 2020 430.status-uptime -rwxr-xr-x 1 root wheel 611B Apr 2 03:01:00 2020 420.status-network -rwxr-xr-x 1 root wheel 684B Apr 2 03:01:00 2020 410.status-mfi -rwxr-xr-x 1 root wheel 590B Apr 2 03:01:00 2020 409.status-gconcat -rwxr-xr-x 1 root wheel 590B Apr 2 03:01:00 2020 408.status-gstripe -rwxr-xr-x 1 root wheel 591B Apr 2 03:01:00 2020 407.status-graid3 -rwxr-xr-x 1 root wheel 596B Apr 2 03:01:00 2020 406.status-gmirror -rwxr-xr-x 1 root wheel 807B Apr 2 03:01:00 2020 404.status-zfs -rwxr-xr-x 1 root wheel 583B Apr 2 03:01:00 2020 401.status-graid -rwxr-xr-x 1 root wheel 773B Apr 2 03:01:00 2020 400.status-disks -rwxr-xr-x 1 root wheel 724B Apr 2 03:01:00 2020 330.news -r-xr-xr-x 1 root wheel 1.4K Apr 2 03:01:00 2020 310.accounting -rwxr-xr-x 1 root wheel 693B Apr 2 03:01:00 2020 300.calendar -rwxr-xr-x 1 root wheel 1.0K Apr 2 03:01:00 2020 210.backup-aliases -rwxr-xr-x 1 root wheel 1.7K Apr 2 03:01:00 2020 200.backup-passwd -rwxr-xr-x 1 root wheel 603B Apr 2 03:01:00 2020 150.clean-hoststat -rwxr-xr-x 1 root wheel 1.0K Apr 2 03:01:00 2020 140.clean-rwho -rwxr-xr-x 1 root wheel 709B Apr 2 03:01:00 2020 130.clean-msgs -rwxr-xr-x 1 root wheel 1.1K Apr 2 03:01:00 2020 120.clean-preserve -rwxr-xr-x 1 root wheel 1.5K Apr 2 03:01:00 2020 110.clean-tmps -rwxr-xr-x 1 root wheel 1.3K Apr 2 03:01:00 2020 100.clean-disks -rwxr-xr-x 1 root wheel 811B Apr 1 03:54:15 2020 999.local -rwxr-xr-x 1 root wheel 2.8K Apr 1 03:54:15 2020 800.scrub-zfs -rwxr-xr-x 1 root wheel 845B Apr 1 03:54:15 2020 510.status-world-kern= el -rwxr-xr-x 1 root wheel 737B Apr 1 03:54:15 2020 500.queuerun -rwxr-xr-x 1 root wheel 498B Apr 1 03:54:15 2020 480.status-ntpd drwxr-xr-x 2 root wheel 1.0K Dec 7 06:23:36 2018 . drwxr-xr-x 6 root wheel 512B Dec 7 06:23:36 2018 .. root@zfs01:~ # ls -lahtuT /etc/periodic/security/ total 68 -r--r--r-- 1 root wheel 2.8K Apr 2 03:01:49 2020 security.functions -rwxr-xr-x 1 root wheel 2.3K Apr 2 03:01:49 2020 900.tcpwrap -rwxr-xr-x 1 root wheel 2.3K Apr 2 03:01:49 2020 800.loginfail -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 700.kernelmsg -rwxr-xr-x 1 root wheel 2.0K Apr 2 03:01:49 2020 610.ipf6denied -rwxr-xr-x 1 root wheel 2.2K Apr 2 03:01:49 2020 550.ipfwlimit -rwxr-xr-x 1 root wheel 2.1K Apr 2 03:01:49 2020 520.pfdenied -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 510.ipfdenied -rwxr-xr-x 1 root wheel 2.0K Apr 2 03:01:49 2020 500.ipfwdenied -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 410.logincheck -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 400.passwdless -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 300.chkuid0 -rwxr-xr-x 1 root wheel 2.3K Apr 2 03:01:49 2020 200.chkmounts -rwxr-xr-x 1 root wheel 2.2K Apr 2 03:01:25 2020 110.neggrpperm -rwxr-xr-x 1 root wheel 2.2K Apr 2 03:01:00 2020 100.chksetuid drwxr-xr-x 2 root wheel 512B Dec 7 06:23:36 2018 . drwxr-xr-x 6 root wheel 512B Dec 7 06:23:36 2018 .. root@zfs01:~ #=20 I don't know how long we will be able to keep this installation running und= er these circumstances; the platform it provides NFS for will be "LIVE" in a couple of weeks and this issue needs to be resolved prior to that. Kind regards, Ruben --=20 You are receiving this mail because: You are the assignee for the bug.=