From owner-freebsd-current@freebsd.org Sat Nov 14 00:20:02 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 55247462160 for ; Sat, 14 Nov 2020 00:20:02 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from dec.sakura.ne.jp (dec.sakura.ne.jp [210.188.226.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CXwvh6FQZz3Lx7 for ; Sat, 14 Nov 2020 00:20:00 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from kalamity.joker.local (115-38-187-204.shizuoka1.commufa.jp [115.38.187.204]) (authenticated bits=0) by dec.sakura.ne.jp (8.15.2/8.15.2/[SAKURA-WEB]/20080708) with ESMTPA id 0AE0JpW9055793 for ; Sat, 14 Nov 2020 09:19:51 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Sat, 14 Nov 2020 09:19:51 +0900 From: Tomoaki AOKI To: freebsd-current@freebsd.org Subject: Re: Shutdown errors and timeout Message-Id: <20201114091951.4888878c686d07ad73e55da8@dec.sakura.ne.jp> In-Reply-To: <20201113.200459.520180046556100070.yasu@utahime.org> References: <65b1ff51-a946-61d0-79d9-104c1e053554@gmail.com> <20201113.200459.520180046556100070.yasu@utahime.org> Reply-To: junchoon@dec.sakura.ne.jp Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4CXwvh6FQZz3Lx7 X-Spamd-Bar: - X-Spamd-Result: default: False [-1.60 / 15.00]; HAS_REPLYTO(0.00)[junchoon@dec.sakura.ne.jp]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[210.188.226.8:from]; ASN(0.00)[asn:9370, ipnet:210.188.224.0/19, country:JP]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[115.38.187.204:received]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[sakura.ne.jp]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[210.188.226.8:from:127.0.2.255]; RCVD_TLS_LAST(0.00)[]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Nov 2020 00:20:02 -0000 On Fri, 13 Nov 2020 20:04:59 +0900 (JST) Yasuhiro KIMURA wrote: > From: Johan Hendriks > Subject: Shutdown errors and timeout > Date: Fri, 13 Nov 2020 11:35:53 +0100 > > > Hello all, i have two FreeBSD 13 machines, one is a bare metal and one > > is virtualbox machine which i both update about once a week. > > > > The vritual machine seems to fail stopping something and gives a > > timeout after 90 sec. > > > > The console ends with > > > > Writing entropy file: . > > Writing early boot entropy file: . > > > > 90 second watchdog timeout expired. Shutdown terminated. > > Fri Nov13 11:20:40 CEST 2020 > > Nov 13 11:20:40 test-head init[1]: /etc/rc.shutdown terminated > > abnormally, going to single user mode > > ... > > > > On the bare metal machine i see the following. > > Writing entropy file: . > > Writing early boot entropy file: . > > cannot unmount '/var/run': umount failed > > cannot unmount '/var/log': umount failed > > cannot unmount '/var': umount failed > > cannot unmount '/usr/home': umount failed > > cannot unmount '/usr': umount failed > > cannot unmount '/': umount failed > > > (snip) > > > > The pools have not been upgraded after the latest openzfs import, > > maybe that is related? > > > > FreeBSD test-freebsd-head 13.0-CURRENT FreeBSD 13.0-CURRENT #2 > > r367585: > > > > First thing i noticed is about a week ago. > > I'm facing same problem with 13.0-CURRENT amd64 r367487 and > virtualbox. In my case I use autofs to mount remote file system of > 12.2-RELEASE amd64 server with NFSv4. When there is still filesystem > mounted by autofs, then watchdog timeout happens while shutdown. The > watchdog timeout can be worked around by executing `automount -fu` > before shutting down. But 'cannot unmount ...' error messages are > still displayed. > > I added 'rc_debug="YES"' to /etc/rc.conf and checked which rc script > causes this message. Then it is displayed when following `zfs_stop` > function of /etc/rc.d/zfs is executed. > > ---------------------------------------------------------------------- > zfs_stop_main() > { > zfs unshare -a > zfs unmount -a > } > ---------------------------------------------------------------------- > > At this point syslog process still running and it opens some files > under /var/log. So it make sence that `zfs unmount -a` results in the > message. > > Probably order of executing each rc script in shutdown time should be > changed so `/etc/rc.d/zfs faststop` is executed after all processes > other than `init' are exited. > > --- > Yasuhiro KIMURA > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" This happens on stable/12, too. As a workaround, reverting r367291 on head (r367546 on stable/12) would stop the issue until this is really fixed. If you have shared dataset or jail(s) mounting dataset, the workaround would be discouraged. Read commit message for detail. I couldn't determine the problematic commit on head as there was multiple commits between previous build, but could determine on stable/12 as there was no other zfs-related commit between previous build. -- Tomoaki AOKI