From owner-freebsd-virtualization@freebsd.org Sun Nov 1 10:04:41 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0601F448A60 for ; Sun, 1 Nov 2020 10:04:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CPBVJ6Pztz3yPH for ; Sun, 1 Nov 2020 10:04:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id DA57B448A5E; Sun, 1 Nov 2020 10:04:40 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D909144896D for ; Sun, 1 Nov 2020 10:04:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CPBVJ5Lgrz3ygj for ; Sun, 1 Nov 2020 10:04:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 96BF416C2B for ; Sun, 1 Nov 2020 10:04:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0A1A4e17017707 for ; Sun, 1 Nov 2020 10:04:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0A1A4eFd017706 for virtualization@FreeBSD.org; Sun, 1 Nov 2020 10:04:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 250770] AWS EC2 system freezes up possibly associated with NFS (EFS) Date: Sun, 01 Nov 2020 10:04:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: raj@gusw.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Nov 2020 10:04:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D250770 --- Comment #1 from Gunther Schadow --- Note, this is not like the ENA kernel panic nor the other AWS EC2 freeze on= t3 bug (apparently fixed in 12.1-RELENG) nor does the presence or absence of t= he "intsmb0: Could not allocate I/O space" error during boot (open since 2018) make any difference. So this is not a duplicate of those bugs. sysctl -a |... kern.features.nfsd: 1 kern.features.nfscl: 1 vfs.nfs.downdelayinitial: 12 vfs.nfs.downdelayinterval: 30 vfs.nfs.defect: 0 vfs.nfs.iodmax: 20 vfs.nfs.iodmin: 0 vfs.nfs.iodmaxidle: 120 vfs.nfs.use_buf_pager: 1 vfs.nfs.fileid_maxwarnings: 10 vfs.nfs.diskless_rootpath: vfs.nfs.diskless_valid: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.nfs_directio_allow_mmap: 1 vfs.nfs.nfs_keep_dirty_on_error: 0 vfs.nfs.nfs_directio_enable: 0 vfs.nfs.clean_pages_on_close: 1 vfs.nfs.commit_on_close: 0 vfs.nfs.prime_access_cache: 0 vfs.nfs.access_cache_timeout: 60 vfs.nfs.dssameconn: 0 vfs.nfs.ignore_eexist: 0 vfs.nfs.pnfsiothreads: -1 vfs.nfs.userhashsize: 100 vfs.nfs.debuglevel: 0 vfs.nfs.callback_addr: vfs.nfs.realign_count: 0 vfs.nfs.realign_test: 0 vfs.nfs.pnfsmirror: 1 vfs.nfs.enable_uidtostring: 0 vfs.nfs.dsretries: 2 vfs.nfs.skip_wcc_data_onerr: 1 vfs.nfs.nfs3_jukebox_delay: 10 vfs.nfs.reconnects: 0 vfs.nfs.bufpackets: 4 searching for ways to do some debug logging, should I enable this? debug.fail_point.status_nfscl_force_fileid_warning: off debug.fail_point.nfscl_force_fileid_warning: off How might I log NFS warnings? Current setup with out of the box unchanged /etc/syslog.conf # $FreeBSD: releng/12.2/usr.sbin/syslogd/syslog.conf 338146 2018-08-21 17:01:47Z brd $ # # Spaces ARE valid field separators in this file. However, # other *nix-like systems still insist on using tabs as field # separators. If you are sharing this file between systems, you # may want to use only tabs as field separators here. # Consult the syslog.conf(5) manpage. *.err;kern.warning;auth.notice;mail.crit /dev/console *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err=20=20 /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log mail.info /var/log/maillog cron.* /var/log/cron !-devd *.=3Ddebug /var/log/debug.log *.emerg * # uncomment this to log all writes to /dev/console to /var/log/console.log # touch /var/log/console.log and chmod it to mode 600 before it will work #console.info /var/log/console.log # uncomment this to enable logging of all log messages to /var/log/all.log # touch /var/log/all.log and chmod it to mode 600 before it will work #*.* /var/log/all.log # uncomment this to enable logging to a remote loghost named loghost #*.* @loghost # uncomment these if you're running inn # news.crit /var/log/news/news.crit # news.err /var/log/news/news.err # news.notice /var/log/news/news.notice # Uncomment this if you wish to see messages produced by devd # !devd # *.>=3Dnotice /var/log/devd.log !* include /etc/syslog.d include /usr/local/etc/syslog.d in /var/log/messages you see the reboot without any issues reported: Nov 1 04:00:13 freebsd ec2[777]: ############################################################# ... Nov 1 05:49:13 freebsd su[1602]: ec2-user to root on /dev/pts/5 Nov 1 08:56:09 freebsd syslogd: exiting on signal 15 Nov 1 08:59:01 freebsd syslogd: kernel boot file is /boot/kernel/kernel Nov 1 08:59:01 freebsd kernel: ---<>--- not sure why syslogd is exiting on signal 15, that might have to do wit the lock-up, but more likely has to do with my force-stop action on the AWS console. Here is the earlier lock-up sequence: Oct 31 23:58:12 freebsd su[888]: ec2-user to root on /dev/pts/2 Oct 31 23:58:23 freebsd su[891]: ec2-user to root on /dev/pts/1 Oct 31 23:58:43 freebsd fsck[861]: /dev/gpt/varfs: 550 files, 1650 used, 12= 5234 free (106 frags, 15641 blocks, 0.1% fragmentation) Nov 1 03:48:09 freebsd syslogd: exiting on signal 15 Nov 1 03:50:35 freebsd syslogd: kernel boot file is /boot/kernel/kernel Nov 1 03:50:35 freebsd kernel: ---<>--- about 4 hours like I estimated, and the "syslogd: exiting on signal 15" is = just too close to the next boot that I don't think it is a herald of the lock-up, but rather it happens when I do the force-stop on the AWS EC2 management dashboard. And this would prove that we are not losing log messages due to = the lock-up. If the system had anything to say, it would have said it! --=20 You are receiving this mail because: You are the assignee for the bug.=