From owner-freebsd-current@freebsd.org Sat Nov 14 12:03:55 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CE3C92E874D for ; Sat, 14 Nov 2020 12:03:55 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-ej1-f65.google.com (mail-ej1-f65.google.com [209.85.218.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CYDWt6WtPz4rXm for ; Sat, 14 Nov 2020 12:03:54 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-ej1-f65.google.com with SMTP id y17so12022072ejh.11 for ; Sat, 14 Nov 2020 04:03:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=i9ZzI2e4dmqym7PeXDX3C08L3nsnq8u7ovfEP/iyNiU=; b=eNq3G9AUIz1bfNSRYUMNR0c0rI3xpSU4McaR0m8PrsJ3fItWz41/Cg2oComqR3j4Xk wEa665TF9tMVQGNoT3lXmf7VKWVbl87kRbeCVCQbEZFtnKRxsj/YdnG2aWbNgs8yJ85r GtaT1s2wDxiptnVFJ3KSS0E+7+6JogyjsqrIYwlZH+SGnWCW5Rhtf7Qluj7mn3dn8DQG HLCy/SlilOAYaR6xu5SKH9lAa2JNExClxhEecUnX90M+3EmuMgYiom8oG0GCWDAsksYV bSW0WA2JBTFf1PHACorR6+s9BRtR0guvInTTXL3pQh3YHxhK4YZf0I/FSbQb1LQNkK/b tAdg== X-Gm-Message-State: AOAM532L6uHOt1gUfe2Xx56hPhalHA+/7zmICIbWgIEb9RDtC59nVqHE Fz74No9KRP7ioJoI6i1aLpJMJ/evzc0= X-Google-Smtp-Source: ABdhPJzM4McvPFIFtZ0O8/NU4Y/d88Ae3UV8urVa0P/45tQIbK6Zx+xhX+IGmrMdGU1p/uPXeLuBgw== X-Received: by 2002:a17:906:8485:: with SMTP id m5mr6597936ejx.205.1605355431163; Sat, 14 Nov 2020 04:03:51 -0800 (PST) Received: from ?IPv6:2a02:8109:98c0:1bc0:5e5f:67ff:fef4:ffd8? ([2a02:8109:98c0:1bc0:5e5f:67ff:fef4:ffd8]) by smtp.gmail.com with ESMTPSA id o21sm6281289ejx.68.2020.11.14.04.03.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 14 Nov 2020 04:03:50 -0800 (PST) Subject: Re: Shutdown errors and timeout To: junchoon@dec.sakura.ne.jp, freebsd-current@freebsd.org References: <65b1ff51-a946-61d0-79d9-104c1e053554@gmail.com> <20201113.200459.520180046556100070.yasu@utahime.org> <20201114091951.4888878c686d07ad73e55da8@dec.sakura.ne.jp> From: Mateusz Piotrowski <0mp@FreeBSD.org> Message-ID: <7316979e-1a87-791a-075c-7f3d7a75f43f@FreeBSD.org> Date: Sat, 14 Nov 2020 13:03:56 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 In-Reply-To: <20201114091951.4888878c686d07ad73e55da8@dec.sakura.ne.jp> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4CYDWt6WtPz4rXm X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[209.85.218.65:from]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[0mp@FreeBSD.org,mpp302@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.218.65:from]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[0mp@FreeBSD.org,mpp302@gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; SPAMHAUS_ZRD(0.00)[209.85.218.65:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.218.65:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Nov 2020 12:03:55 -0000 Hi, On 11/14/20 1:19 AM, Tomoaki AOKI wrote: > On Fri, 13 Nov 2020 20:04:59 +0900 (JST) > Yasuhiro KIMURA wrote: > >> From: Johan Hendriks >> >>> Hello all, i have two FreeBSD 13 machines, one is a bare metal and one >>> is virtualbox machine which i both update about once a week. >>> >>> The vritual machine seems to fail stopping something and gives a >>> timeout after 90 sec. >>> >>> The console ends with >>> >>> Writing entropy file: . >>> Writing early boot entropy file: . >>> >>> 90 second watchdog timeout expired. Shutdown terminated. >>> Fri Nov13 11:20:40 CEST 2020 >>> Nov 13 11:20:40 test-head init[1]: /etc/rc.shutdown terminated >>> abnormally, going to single user mode >>> ... >>> >>> On the bare metal machine i see the following. >>> Writing entropy file: . >>> Writing early boot entropy file: . >>> cannot unmount '/var/run': umount failed >>> cannot unmount '/var/log': umount failed >>> cannot unmount '/var': umount failed >>> cannot unmount '/usr/home': umount failed >>> cannot unmount '/usr': umount failed >>> cannot unmount '/': umount failed >>> >> (snip) >>> The pools have not been upgraded after the latest openzfs import, >>> maybe that is related? >>> >>> FreeBSD test-freebsd-head 13.0-CURRENT FreeBSD 13.0-CURRENT #2 >>> r367585: >>> >>> First thing i noticed is about a week ago. >> I'm facing same problem with 13.0-CURRENT amd64 r367487 and >> virtualbox. In my case I use autofs to mount remote file system of >> 12.2-RELEASE amd64 server with NFSv4. When there is still filesystem >> mounted by autofs, then watchdog timeout happens while shutdown. The >> watchdog timeout can be worked around by executing `automount -fu` >> before shutting down. But 'cannot unmount ...' error messages are >> still displayed. >> >> I added 'rc_debug="YES"' to /etc/rc.conf and checked which rc script >> causes this message. Then it is displayed when following `zfs_stop` >> function of /etc/rc.d/zfs is executed. >> >> ---------------------------------------------------------------------- >> zfs_stop_main() >> { >> zfs unshare -a >> zfs unmount -a >> } >> ---------------------------------------------------------------------- >> >> At this point syslog process still running and it opens some files >> under /var/log. So it make sence that `zfs unmount -a` results in the >> message. >> >> Probably order of executing each rc script in shutdown time should be >> changed so `/etc/rc.d/zfs faststop` is executed after all processes >> other than `init' are exited. > This happens on stable/12, too. > As a workaround, reverting r367291 on head (r367546 on stable/12) > would stop the issue until this is really fixed. > > If you have shared dataset or jail(s) mounting dataset, the workaround > would be discouraged. Read commit message for detail. I've committed r367291 and r367546. I am not sure if I can think of a proper fix for the described issues, so I guess the best idea would be to revert those changes for now until we figure out how to do it properly. Sorry for the regression. Best, Mateusz