From owner-freebsd-current@freebsd.org Mon Nov 16 18:16:57 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 993F146D693 for ; Mon, 16 Nov 2020 18:16:57 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-ed1-x543.google.com (mail-ed1-x543.google.com [IPv6:2a00:1450:4864:20::543]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CZcjN3qlCz4R4t for ; Mon, 16 Nov 2020 18:16:55 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by mail-ed1-x543.google.com with SMTP id cq7so19695043edb.4 for ; Mon, 16 Nov 2020 10:16:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=RvPeaNV0vxJnhdmKyKu1iD0AvosfYL4LAQWeHpK18V4=; b=Ers7DewXPQLWL3rqMvX1k3YVdE6r8Oiaefh7jVcqyyMw4S013yALH8paO/XAEQHw+e GDGlh+Z/ncJUun8INd7/if+8kmbaPXOBo+kjZRtpbNc1fNZarirIDi5Qv20rOWlZ8/cI //hCrH3ddacENQQuDmS5Ax+rFeJ3jii4LZ7/p/S0BNnfO/+LD+9Jxfs/c/tEh3HEWXEY Jjk2a8CmGU21Hs9vEViRgWyQkx67+TUvpwAGU8m3fJFSTpUWd6zXiYsecEdDKlH8b9PQ jlsfgPthPWKhtGeOD/3vnmsqjixscNSEpA4qRj2UQQNb7e1E06HbiO5JJL8RFDccG9eQ 7jTw== X-Gm-Message-State: AOAM532K5n909jXDi+zp58bAkA5LPJytVDn7yDUhuIhM+epviyseJoSs wA32tn8mZrFNDq0UVVbrfcCBe5sSjsA= X-Google-Smtp-Source: ABdhPJwqUXF/xwZ9L7TzU5KMrAUHmM6MQiKiAJfcImpKLeRxhEY7c3nfY/jt4LwI45rEgRV93VX2wg== X-Received: by 2002:a05:6402:1a58:: with SMTP id bf24mr16433351edb.191.1605550614516; Mon, 16 Nov 2020 10:16:54 -0800 (PST) Received: from MacBook-Pro-van-Johan.local (85-147-130-226.cable.dynamic.v4.ziggo.nl. [85.147.130.226]) by smtp.gmail.com with ESMTPSA id m2sm10642484edp.48.2020.11.16.10.16.52 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Nov 2020 10:16:53 -0800 (PST) Subject: Re: Shutdown errors and timeout To: freebsd-current@freebsd.org References: <65b1ff51-a946-61d0-79d9-104c1e053554@gmail.com> <20201113.200459.520180046556100070.yasu@utahime.org> <20201114091951.4888878c686d07ad73e55da8@dec.sakura.ne.jp> <7316979e-1a87-791a-075c-7f3d7a75f43f@FreeBSD.org> From: Johan Hendriks Message-ID: Date: Mon, 16 Nov 2020 19:16:51 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.4.3 MIME-Version: 1.0 In-Reply-To: <7316979e-1a87-791a-075c-7f3d7a75f43f@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4CZcjN3qlCz4R4t X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::543:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::543:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::543:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2020 18:16:57 -0000 On 14/11/2020 13:03, Mateusz Piotrowski wrote: > Hi, > > On 11/14/20 1:19 AM, Tomoaki AOKI wrote: >> On Fri, 13 Nov 2020 20:04:59 +0900 (JST) >> Yasuhiro KIMURA wrote: >> >>> From: Johan Hendriks >>> >>>> Hello all, i have two FreeBSD 13 machines, one is a bare metal and one >>>> is virtualbox machine which i both update about once a week. >>>> >>>> The vritual machine seems to fail stopping something and gives a >>>> timeout after 90 sec. >>>> >>>> The console ends with >>>> >>>> Writing entropy file: . >>>> Writing early boot entropy file: . >>>> >>>> 90 second watchdog timeout expired. Shutdown terminated. >>>> Fri Nov13 11:20:40 CEST 2020 >>>> Nov 13 11:20:40 test-head init[1]: /etc/rc.shutdown terminated >>>> abnormally, going to single user mode >>>> ... >>>> >>>> On the bare metal machine i see the following. >>>> Writing entropy file: . >>>> Writing early boot entropy file: . >>>> cannot unmount '/var/run': umount failed >>>> cannot unmount '/var/log': umount failed >>>> cannot unmount '/var': umount failed >>>> cannot unmount '/usr/home': umount failed >>>> cannot unmount '/usr': umount failed >>>> cannot unmount '/': umount failed >>>> >>> (snip) >>>> The pools have not been upgraded after the latest openzfs import, >>>> maybe that is related? >>>> >>>> FreeBSD test-freebsd-head 13.0-CURRENT FreeBSD 13.0-CURRENT #2 >>>> r367585: >>>> >>>> First thing i noticed is about a week ago. >>> I'm facing same problem with 13.0-CURRENT amd64 r367487 and >>> virtualbox. In my case I use autofs to mount remote file system of >>> 12.2-RELEASE amd64 server with NFSv4. When there is still filesystem >>> mounted by autofs, then watchdog timeout happens while shutdown. The >>> watchdog timeout can be worked around by executing `automount -fu` >>> before shutting down. But 'cannot unmount ...' error messages are >>> still displayed. >>> >>> I added 'rc_debug="YES"' to /etc/rc.conf and checked which rc script >>> causes this message. Then it is displayed when following `zfs_stop` >>> function of /etc/rc.d/zfs is executed. >>> >>> ---------------------------------------------------------------------- >>> zfs_stop_main() >>> { >>>     zfs unshare -a >>>     zfs unmount -a >>> } >>> ---------------------------------------------------------------------- >>> >>> At this point syslog process still running and it opens some files >>> under /var/log. So it make sence that `zfs unmount -a` results in the >>> message. >>> >>> Probably order of executing each rc script in shutdown time should be >>> changed so `/etc/rc.d/zfs faststop` is executed after all processes >>> other than `init' are exited. >> This happens on stable/12, too. >> As a workaround, reverting r367291 on head (r367546 on stable/12) >> would stop the issue until this is really fixed. >> >> If you have shared dataset or jail(s) mounting dataset, the workaround >> would be discouraged. Read commit message for detail. > > I've committed r367291 and r367546. > > I am not sure if I can think of a proper fix for the described issues, > so I guess the best idea would be to revert those changes for now > until we figure out how to do it properly. > > Sorry for the regression. > > Best, > > Mateusz I can tell that reverting the mentioned commit i do not have the symptoms when i reboot my servers. Thank you all for your time, and no sorry needed ;-) regards, Johan