From owner-freebsd-virtualization@freebsd.org Sun Nov 1 14:54:45 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3C14744FD05 for ; Sun, 1 Nov 2020 14:54:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CPJx10pbzz4FH4 for ; Sun, 1 Nov 2020 14:54:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 1BBC044F9B1; Sun, 1 Nov 2020 14:54:45 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1B80444FB34 for ; Sun, 1 Nov 2020 14:54:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CPJx06mgCz4F7k for ; Sun, 1 Nov 2020 14:54:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CBDE61A1C2 for ; Sun, 1 Nov 2020 14:54:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0A1EsiWU081851 for ; Sun, 1 Nov 2020 14:54:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0A1Esixu081850 for virtualization@FreeBSD.org; Sun, 1 Nov 2020 14:54:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 250770] AWS EC2 system freezes up possibly associated with NFS (EFS) Date: Sun, 01 Nov 2020 14:54:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: raj@gusw.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Nov 2020 14:54:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D250770 --- Comment #2 from Gunther Schadow --- Before anyone spends time on this, I am still trying some options. For now I have made 3 modification in my NFS mount options: 1. timeo=3D150 down from 600 - that alone didn't help 2. oneopenown - that alone didn't help 3. minorversion=3D1 added, the manual isn't clear if it is a requirement = for oneopenown or if it implicitly sets minorversion=3D1, but since I added tha= t, I have not frozen up for almost 6 hours. That's new. I also enabled crude NFS logging with=20 sysctl vfs.nfs.debuglevel=3D4 And this is what I see in the /var/log/messages: Nov 1 08:59:23 freebsd su[926]: ec2-user to root on /dev/pts/1 Nov 1 10:28:20 freebsd su[1271]: ec2-user to root on /dev/pts/2 Nov 1 10:28:38 freebsd kernel: clnt call=3D0 Nov 1 10:28:43 freebsd syslogd: last message repeated 8 times Nov 1 10:28:43 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 10:28:43 freebsd kernel: clnt call=3D0 Nov 1 10:29:14 freebsd syslogd: last message repeated 1125 times Nov 1 10:31:20 freebsd syslogd: last message repeated 63 times Nov 1 10:36:05 freebsd syslogd: last message repeated 368 times Nov 1 10:36:05 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 10:36:05 freebsd kernel: clnt call=3D0 ... Nov 1 13:25:37 freebsd kernel: clnt call=3D0 Nov 1 13:25:37 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:25:37 freebsd kernel: clnt call=3D0 Nov 1 13:25:37 freebsd syslogd: last message repeated 6 times Nov 1 13:25:37 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:25:37 freebsd kernel: clnt call=3D0 Nov 1 13:25:37 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:25:37 freebsd kernel: clnt call=3D0 Nov 1 13:25:37 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:25:37 freebsd kernel: clnt call=3D0 Nov 1 13:25:37 freebsd syslogd: last message repeated 6 times Nov 1 13:25:37 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:25:37 freebsd kernel: clnt call=3D0 Nov 1 13:25:37 freebsd syslogd: last message repeated 6 times Nov 1 13:25:37 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:25:37 freebsd kernel: clnt call=3D0 Nov 1 13:25:37 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:25:37 freebsd kernel: clnt call=3D0 Nov 1 13:25:56 freebsd syslogd: last message repeated 53 times Nov 1 13:25:56 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:25:56 freebsd kernel: clnt call=3D0 Nov 1 13:26:27 freebsd syslogd: last message repeated 80 times Nov 1 13:26:55 freebsd syslogd: last message repeated 204 times Nov 1 13:26:55 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:26:55 freebsd kernel: clnt call=3D0 Nov 1 13:27:36 freebsd syslogd: last message repeated 58 times Nov 1 13:29:26 freebsd syslogd: last message repeated 87 times Nov 1 13:30:47 freebsd syslogd: last message repeated 78 times Nov 1 13:30:47 freebsd kernel: readrpc: aft doiods=3D5 Nov 1 13:30:47 freebsd kernel: clnt call=3D0 Nov 1 13:31:21 freebsd syslogd: last message repeated 26 times Nov 1 13:33:00 freebsd syslogd: last message repeated 58 times Nov 1 13:43:21 freebsd syslogd: last message repeated 119 times Nov 1 13:53:06 freebsd syslogd: last message repeated 277 times Nov 1 14:03:00 freebsd syslogd: last message repeated 263 times Nov 1 14:13:22 freebsd syslogd: last message repeated 216 times Nov 1 14:23:06 freebsd syslogd: last message repeated 5110 times Nov 1 14:33:00 freebsd syslogd: last message repeated 491 times Nov 1 14:43:22 freebsd syslogd: last message repeated 120 times nothing really meaningful to me. But I hope that if it does lock up, I will= see something peculiar.=20 On the other hand, now on c5.large with: 1. minorversion=3D1 2. sysctl vfs.nfs.debuglevel=3D4 I have not locked up again. If I can make it a full day, I will take back t= hese 3 changes one by one, first the debug logging, then the minorversion=3D1, t= o see when the problem recurs. --=20 You are receiving this mail because: You are the assignee for the bug.=