From owner-freebsd-stable@FreeBSD.ORG Mon Mar 9 16:19:39 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21A91F10 for ; Mon, 9 Mar 2015 16:19:39 +0000 (UTC) Received: from mail-la0-x229.google.com (mail-la0-x229.google.com [IPv6:2a00:1450:4010:c03::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 83EB4F2A for ; Mon, 9 Mar 2015 16:19:38 +0000 (UTC) Received: by lamq1 with SMTP id q1so8202379lam.12 for ; Mon, 09 Mar 2015 09:19:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=AeWs1qyfRRFAfQG7ekWb1snENt3WPIV1cVqkfsquqVo=; b=irI3RLCW8Z73anRIil+qqbZ3t+hPk2veyLj1mZ62S5FR7kpynzZ805QZszoeY3QrIx nso3iSuNcDa2QHYcVqjlLE7lI3ilzsLgMGLRYMjQNPiQE+iFlclQvBxIPfgpAadOs3lT rZ2hgzgzsNNSh4gc0dv1gCKV4xXCfHAmWzf0SpEKri749fKIKRcqz0X5SbdXqzk3iyju IuobCeIVL4SFkpfFssjHEQP12JOPTF+5w5Bx/8VYRIqk5yFWWqAhsVxuVuwP69CKY870 swnu1M3dyYb2PX4wo3Guu53ZQoJ11plY+ZcQtl6206L9/84fR4prgXE8xsxVyFQiza8x eQ0g== MIME-Version: 1.0 X-Received: by 10.112.98.201 with SMTP id ek9mr26760588lbb.68.1425917976487; Mon, 09 Mar 2015 09:19:36 -0700 (PDT) Received: by 10.25.90.131 with HTTP; Mon, 9 Mar 2015 09:19:36 -0700 (PDT) In-Reply-To: References: <2F9DC176-912C-40C0-BAB7-DB66BD572ABA@vnode.se> <54D8E341.101@pix.net> <1423501720.16794.18.camel@freebsd.org> Date: Mon, 9 Mar 2015 09:19:36 -0700 Message-ID: Subject: Re: freebsd-update and hang during reboot From: Nick Rogers Cc: FreeBSD STABLE Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2015 16:19:39 -0000 On Tue, Feb 10, 2015 at 1:37 PM, Nick Rogers wrote: > > > On Mon, Feb 9, 2015 at 9:08 AM, Ian Lepore wrote: > >> On Mon, 2015-02-09 at 11:41 -0500, Kurt Lidl wrote: >> > Joel wrote: >> > > Hi, >> > > >> > > Just about every machine I have seems to hang after running >> freebsd-update and doing a reboot. The last message on the screen is "Al= l >> buffers synced=E2=80=9D and it just freezes. >> > > >> > > This happens when doing a freebsd-update and going from 10.0 to 10.1= , >> but also when doing a fresh 10.1 install and using freebsd-update to get >> the latest -pX security patches. As soon as I reboot the machine, it han= gs. >> > > >> > > I=E2=80=99ve tried it on several different HP ProLiant models, on In= tel NUCs >> and on VMware virtual machines. Same phenomenon everywhere. It=E2=80=99s= really >> easy to trigger: just install 10.1, use default settings everywhere, >> freebsd-update fetch/install, shutdown -r now and BOOM. It hangs. I thin= k >> I=E2=80=99ve seen it on >> > > >> > > >> > > >> > > >> > > 30 servers or so now. >> > > >> > > Everything works like it should after the initial hang tough - no >> matter how many times I reboot it completes the reboot cycle just fine. >> > > >> > > I=E2=80=99ve seen several people (mostly on IRC) mention this proble= m, but no >> solution. >> > > >> > > Is anyone working on fixing this? >> > >> > I ran into this problem in spades when upgrading a set of servers from >> > FreeBSD 9.0 to 9.1. I happened consistently. Normal reboots worked, >> > but when going from 9.0 to 9.1, it *ALWAYS* hung, and it always hung >> > at the same place, after printing the "All buffers synced" message. >> > >> > I ultimately determined that if I did the following, rather than >> > just a "reboot" or "shutdown -r now 'FreeBSD 9.1-RELEASE upgrade'", >> > it would consistently AVOID the hang: >> > >> > sync ; sync ; sync ; shutdown -o -n -r now "FreeBSD 9.1 install" >> > >> > Your mileage may vary, but you don't have a lot to lose by trying it. >> > >> > -Kurt >> > >> >> That is just bad advice. sync(1) does not g'tee that all data has been >> written, no matter how many times you type it. shutdown -n tells the >> system to abandon unwritten data. All in all, this is a recipe for >> silent filesystem corruption. Using it after an update is just asking >> to have a mix of old and new files on the system after the reboot. >> >> A more robust workaround would be to "mount -r" on all filesystems >> before invoking the shutdown (even a shutdown -n should be safe after >> everything has been remounted readonly). If the mount -r hangs on one >> of the filesystems, then you've probably got a clue as to where a normal >> shutdown is hanging. >> > > FWIW mount -r on the root filesystem hangs for me. If I disable > softupdates-journaling on the root filesystem before the upgrade process, > the system no longer hangs on the last reboot after userland upgrade. > However, the root filesystem still comes up dirty with an incorrect free > block count during fsck. > Is anyone working on fixing this problem? It seems like this should have some kind of "full court press" as it is obviously affecting plenty of people, some of which have spoken up in the following PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D195458 I realize its a tough problem to track down, and if I had the appropriate skills I would help. But so far all I've been able to do, like others, is replicate and complain about the problem. Its still affecting upgrading to 10.1-RELEASE-p6 from the official 10.1-RELEASE distribution, and from 10.1-RELEASE-p5. I just had another production server hang during reboot after updating to p6, and I don't see this changing for the inevitable p7 unless this problem gets more attention. Can someone with the right skill-set please help figure this out? Thank you. >> -- Ian >> >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org= " >> > >