From owner-freebsd-virtualization@freebsd.org Wed Feb 19 13:20:37 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 507DD239D59 for ; Wed, 19 Feb 2020 13:20:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48MyyY1Dlyz4fhq for ; Wed, 19 Feb 2020 13:20:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 278CD239D58; Wed, 19 Feb 2020 13:20:37 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 274D2239D57 for ; Wed, 19 Feb 2020 13:20:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48MyyY03QTz4fhY for ; Wed, 19 Feb 2020 13:20:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E7307947F for ; Wed, 19 Feb 2020 13:20:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 01JDKaUW086408 for ; Wed, 19 Feb 2020 13:20:36 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 01JDKaqg086407 for virtualization@FreeBSD.org; Wed, 19 Feb 2020 13:20:36 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Wed, 19 Feb 2020 13:20:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Feb 2020 13:20:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #23 from mail@rubenvos.com --- (In reply to Colin Percival from comment #21) Hmm. The difference of exactly 1 hour doesn't seem to have a relationship w= ith different timezones... Today we had another occurance on one of the machines: Feb 19 03:24:53 volume3 kernel: nvme1: cpl does not map to outstanding cmd Feb 19 03:24:53 volume3 kernel: cdw0:00000000 sqhd:000c sqid:0002 cid:0017 = p:0 sc:00 sct:0 m:0 dnr:0 Feb 19 03:24:53 volume3 kernel: nvme1: Missing interrupt Feb 19 03:24:53 volume3 kernel: nvme1: Resetting controller due to a timeou= t. Feb 19 03:24:53 volume3 kernel: nvme1: resetting controller Feb 19 03:24:54 volume3 kernel: nvme1: temperature threshold not supported Feb 19 03:24:54 volume3 kernel: nvme1: aborting outstanding i/o Comparing 03:24:53 with the access times of the daily scripts though: ls -lahtuT /etc/periodic/daily/ total 128 -rwxr-xr-x 1 root wheel 1.0K Feb 19 12:31:47 2020 450.status-security -rwxr-xr-x 1 root wheel 811B Feb 19 05:32:14 2020 999.local -rwxr-xr-x 1 root wheel 2.8K Feb 19 05:32:14 2020 800.scrub-zfs -rwxr-xr-x 1 root wheel 845B Feb 19 05:32:14 2020 510.status-world-kern= el -rwxr-xr-x 1 root wheel 737B Feb 19 05:32:14 2020 500.queuerun -rwxr-xr-x 1 root wheel 498B Feb 19 05:32:14 2020 480.status-ntpd -rwxr-xr-x 1 root wheel 451B Feb 19 05:32:14 2020 480.leapfile-ntpd -rwxr-xr-x 1 root wheel 2.0K Feb 19 05:32:14 2020 460.status-mail-rejec= ts -rwxr-xr-x 1 root wheel 1.4K Feb 19 03:01:00 2020 440.status-mailq -rwxr-xr-x 1 root wheel 705B Feb 19 03:01:00 2020 430.status-uptime -rwxr-xr-x 1 root wheel 611B Feb 19 03:01:00 2020 420.status-network -rwxr-xr-x 1 root wheel 684B Feb 19 03:01:00 2020 410.status-mfi -rwxr-xr-x 1 root wheel 590B Feb 19 03:01:00 2020 409.status-gconcat -rwxr-xr-x 1 root wheel 590B Feb 19 03:01:00 2020 408.status-gstripe -rwxr-xr-x 1 root wheel 591B Feb 19 03:01:00 2020 407.status-graid3 -rwxr-xr-x 1 root wheel 596B Feb 19 03:01:00 2020 406.status-gmirror -rwxr-xr-x 1 root wheel 807B Feb 19 03:01:00 2020 404.status-zfs -rwxr-xr-x 1 root wheel 583B Feb 19 03:01:00 2020 401.status-graid -rwxr-xr-x 1 root wheel 773B Feb 19 03:01:00 2020 400.status-disks -rwxr-xr-x 1 root wheel 724B Feb 19 03:01:00 2020 330.news -r-xr-xr-x 1 root wheel 1.4K Feb 19 03:01:00 2020 310.accounting -rwxr-xr-x 1 root wheel 693B Feb 19 03:01:00 2020 300.calendar -rwxr-xr-x 1 root wheel 1.0K Feb 19 03:01:00 2020 210.backup-aliases -rwxr-xr-x 1 root wheel 1.7K Feb 19 03:01:00 2020 200.backup-passwd -rwxr-xr-x 1 root wheel 603B Feb 19 03:01:00 2020 150.clean-hoststat -rwxr-xr-x 1 root wheel 1.0K Feb 19 03:01:00 2020 140.clean-rwho -rwxr-xr-x 1 root wheel 709B Feb 19 03:01:00 2020 130.clean-msgs -rwxr-xr-x 1 root wheel 1.1K Feb 19 03:01:00 2020 120.clean-preserve -rwxr-xr-x 1 root wheel 1.5K Feb 19 03:01:00 2020 110.clean-tmps -rwxr-xr-x 1 root wheel 1.3K Feb 19 03:01:00 2020 100.clean-disks drwxr-xr-x 2 root wheel 1.0K Nov 1 07:06:41 2019 . drwxr-xr-x 6 root wheel 512B Nov 1 07:06:41 2019 .. but if the periodic framework executes the jobs serially I see no link with= =20 440.status-mailq (that does not sound like high io) :S. I think there definately is a link between this bug and high disk-/network-= io so the periodic framework probably classifies as a nice trigger (especially= the security bits with the find commands).... We will continue to cross-reference the access times of the daily scripts w= ith the "Missing interrupt" occurences and post updates. Kind regards, Ruben --=20 You are receiving this mail because: You are the assignee for the bug.=