From owner-freebsd-bugs@FreeBSD.ORG Thu Nov 27 22:44:46 2014 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4DF0BC4B for ; Thu, 27 Nov 2014 22:44:46 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34CAF937 for ; Thu, 27 Nov 2014 22:44:46 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id sARMikqw025845 for ; Thu, 27 Nov 2014 22:44:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 195458] New: Hang on shutdown/root unmount after FreeBSD 10.1R upgrade Date: Thu, 27 Nov 2014 22:44:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: walter@lifeforms.nl X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Nov 2014 22:44:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458 Bug ID: 195458 Summary: Hang on shutdown/root unmount after FreeBSD 10.1R upgrade Product: Base System Version: 10.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Many People Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: walter@lifeforms.nl Created attachment 149945 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=149945&action=edit Screenshot of the hang On three out of six FreeBSD installs upgraded from 10.0R to 10.1R, the system freezes as the 10.1 system shutdowns for the first time. After "All buffers synced." the system remains at 100% CPU and makes no progress for a long time. After a forced reset, the file system is dirty. On another upgrade, the first reboot in 10.1 seemed to go fine. However, a subsequent shutdown flashed the following error: All buffers synced. softdep_waitidle: Failed to flush worklist for 0xfffff800027b4330 unmount of / failed (BUSY) leaving the file system also dirty. This seems related to the first problem (see below). -- Throughout the 10.1-RC cycle, various others have described the hang after upgrading to 10.1 on the freebsd-stable mailinglist: https://lists.freebsd.org/pipermail/freebsd-stable/2014-October/080595.html I spoke to another user on IRC who confirmed the hang with 10.1-RELEASE on two physical servers. In #195183, another hang at reboot after updating to 10.1 is reported (although I am not sure about the effect of ipfw; in any case enabling/disabling ipfw has no effect for me.) -- I reproduced the problem on a clean 10.0-RELEASE install in VMware after a freebsd-update to 10.1-RELEASE. I snapshotted this VM after freebsd-update but before rebooting it, so I can do experiments on it if needed. It appears to me that the problem happens during unmounting of the UFS root filesystem. If after upgrading I drop to single user mode ("shutdown now"), and attempt the command "/sbin/mount -o ro /", this should normally succeed. However, on a failed 10.1 machine, the CPU goes to 100% and the command never finishes. Just like during a shutdown, the kernel is alive (e.g. the host pings) but it's not possible to recover from the situation. I have not seen this problem on subsequent reboots of 10.1 systems, nor on clean 10.1 installs (non-upgraded), but very consistently after a 10.0 to 10.1 upgrade. It's a pretty big showstopper for me at this point, so please let me know if I can provide more info from the test box or help in other ways. -- Reproduce: - download 10.0-RELEASE amd64 ISO - create a VM in VMware - install 10.0-RELEASE, UFS2, all defaults # problem happens on SU and SU+J - freebsd-update upgrade -r 10.1-RELEASE - freebsd-update install - shutdown -r now - freebsd-update install - freebsd-update install # make a VM snapshot before continuing - shutdown -r now # CPU goes to 100% after "All buffers synced." -- You are receiving this mail because: You are the assignee for the bug.