From owner-freebsd-virtualization@freebsd.org Mon Feb 17 10:47:29 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6C092254D2D for ; Mon, 17 Feb 2020 10:47:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48Lgfn1RMqz3MgM for ; Mon, 17 Feb 2020 10:47:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 31392254D2A; Mon, 17 Feb 2020 10:47:29 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 30D79254D29 for ; Mon, 17 Feb 2020 10:47:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48Lgfn0HDfz3MgB for ; Mon, 17 Feb 2020 10:47:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 006995B67 for ; Mon, 17 Feb 2020 10:47:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 01HAlS9s074613 for ; Mon, 17 Feb 2020 10:47:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 01HAlSnl074612 for virtualization@FreeBSD.org; Mon, 17 Feb 2020 10:47:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Mon, 17 Feb 2020 10:47:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Feb 2020 10:47:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #18 from mail@rubenvos.com --- Hi Colin, > 1. How repeatable is this? Does it happen to every instance you launch (= after a variable number of days)? Unfortunately we are not in the habit of often redeplying our zfs nodes (si= nce they provide storage for whole platforms) :( It still happens to the affect= ed nodes though: bzgrep "nvme1: Missing interrupt" /var/log/messages.0.bz2 ; grep "nvme1: Missing interrupt" /var/log/messages Nov 16 03:04:18 zfs01 kernel: nvme1: Missing interrupt Nov 16 03:05:19 zfs01 kernel: nvme1: Missing interrupt Nov 25 03:04:36 zfs01 kernel: nvme1: Missing interrupt Nov 25 03:05:07 zfs01 kernel: nvme1: Missing interrupt Nov 25 03:06:07 zfs01 kernel: nvme1: Missing interrupt Dec 13 03:04:34 zfs01 kernel: nvme1: Missing interrupt Dec 13 03:05:35 zfs01 kernel: nvme1: Missing interrupt Dec 13 03:06:26 zfs01 kernel: nvme1: Missing interrupt Dec 13 03:06:57 zfs01 kernel: nvme1: Missing interrupt Dec 13 03:07:58 zfs01 kernel: nvme1: Missing interrupt Jan 25 03:06:02 zfs01 kernel: nvme1: Missing interrupt Jan 25 03:07:02 zfs01 kernel: nvme1: Missing interrupt Feb 11 03:05:32 zfs01 kernel: nvme1: Missing interrupt Feb 11 03:07:01 zfs01 kernel: nvme1: Missing interrupt Feb 17 03:06:29 zfs01 kernel: nvme1: Missing interrupt =3D=3D=3D bzgrep "nvme1: Missing interrupt" /var/log/messages.0.bz2 ; grep "nvme1: Missing interrupt" /var/log/messages Jan 25 04:29:03 volume3 kernel: nvme1: Missing interrupt Feb 4 04:04:45 volume3 kernel: nvme1: Missing interrupt Feb 11 04:04:48 volume3 kernel: nvme1: Missing interrupt Kind of interesting that zfs01 and volume03 have totally different customer= s, usage patterns but have a collission of 2 dates :|=20 > 2. Have you tried different instance types? Yes. This issue is not manifesting itself on an r4.xlarge instance. Same am= i on r5.large: problems... > 3. What sort of disk is this? We use cloudformation/ansible to deploy these servers, so they are all kind= of identically configured (apart from sizing). Both instances suffering from = this issue are 500GB+ EBS GP2 or IO disks with GPT and a zpool configured onto t= hem.=20 Please let me know if you would like to receive more information. Kind regards, Ruben --=20 You are receiving this mail because: You are the assignee for the bug.=