From owner-freebsd-current@freebsd.org Sat Feb 24 14:38:49 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F601F23C13 for ; Sat, 24 Feb 2018 14:38:49 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C15C864F2 for ; Sat, 24 Feb 2018 14:38:48 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 6E9732123D for ; Sat, 24 Feb 2018 09:38:48 -0500 (EST) Received: from web6 ([10.202.2.216]) by compute7.internal (MEProxy); Sat, 24 Feb 2018 09:38:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=OfQwyDVZBIB84MhRDSosLHOekalZS ni5KoWszAVRVDg=; b=FeRqOfIpDlTFAgJQMlM8/FS2o0VyNHU02/GSUusCUXx7g HwG0TGhOhgy1a3frtR+vSxulqE1Ip/JhPvYhGfs0bkDsTvFHH+wMN5BXIexwJhkW IC8GCi/Ipj5GxOYZ+9dfbAZzYuGF3ZiaTnEwSRtDxu9A3LfGx9PFKJgeiJ4xYyqA 58Hm3bfZbPm5q/W07zQ4pd04DLiOke2u9/kmsFmar00SJlSqGtc06Jx9Zt5d4oda gbKDbwLCUucerhcBSttO9EYv0xDeTTC4ryyumyw0kWiV+Yt1biSbLiiqR7iHGr1q tlQd7sfHat+HZTMJ7tUCe2TJ+VE+RhC6fyUYzp3Iw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=OfQwyD VZBIB84MhRDSosLHOekalZSni5KoWszAVRVDg=; b=ajsLgf9b4YIgxE1Lt5Righ 7squBISt5Gd34mJ4st9Nh+v+yWRTnL5wrDjwjxHLrAoCvFox1qdapleTR1CcnCxM b/eze3SZOpyVLjjT7qkO5xDr6YoC0n5kc0a6+s8qjSMiU6iJg2z+aUmuD7L4ri0T 1Sbgsse7vGWALErAnTIvUzIIFM2h+RWcfiRYECdORhTx6ECKRCs+TmxEu6ni7JPg SlwB2CMJ27NTAv/GE9yRaXwtUmL1ZyQI97f6QWiG4lgBkkf5TLEK7nXJ+HUewGdH anl0m6zqQVsU1tCG4BsPEJ7ZOLmhXzdN0Ik4dAM1EshPpp3L7TzDNu8lU/TJ3kyw == X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 413214230; Sat, 24 Feb 2018 09:38:48 -0500 (EST) Message-Id: <1519483128.3534220.1281952752.51A3BEEF@webmail.messagingengine.com> From: Dave Cottlehuber To: freebsd-current@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-efbb3405 In-Reply-To: <20180222134131.GA45189@v007.zyxst.net> References: <1519297066.646719.1279504848.0851DCB5@webmail.messagingengine.com> <20180222134131.GA45189@v007.zyxst.net> Subject: Re: make installworld broke / how to recover Date: Sat, 24 Feb 2018 15:38:48 +0100 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 14:38:49 -0000 On Thu, 22 Feb 2018, at 14:41, tech-lists wrote: > On Thu, Feb 22, 2018 at 03:10:17AM -0800, Jack L. wrote: > >maybe try a clean buildworld, update /usr/src to the latest version, rm -rf > >/usr/obj, then make buildworld && make installworld && make kernel and see > >if that fixes the issue? > > Is there a documented way of say booting to memstick, starting fixit, > mounting the zfs from the broken system then building/installing a > new world? Is there enough "environment" and utilities to do this on the > fixit disk? I can't help from the source/installworld side, but I have a couple of things to share that may help, apologies if this is already familiar to you. The email's long enough to be a blog entry... 1. fixing a dead/borked system I love using mfsbsd[1] by Martin Matuska for this. It's a bootable FreeBSD image with DHCP support so you can ssh into it from a separate system. You can then import the zpool and basically do all the cleanup from there. zpool import -R /mnt -N -f zroot then mount the datasets you need manually. 2. hacking a semblance of sanity back into the borked current Try this, after doing a zfs snapshot of your filesystem. - download the latest CURRENT snapshot and untar kernel + base in place over top of the above mounted rootfs. Copy back anything "important" from your snapshot like /etc/ and /boot/loader.conf and the system will probably boot now. 3. use boot environments I use these all the time, so awesome. basically when you install 11.x the zfs dataset layout already supports this by default. You can tweak an existing system to fit this if needed, it's not too hard. Details elided, help available if you need it later on. install sysutils/beadm-devel and have a go. Briefly, your / is kept in zroot/ROOT/ (called a Boot Environment) and variable data lives outside that namespace, but overlaid using zfs mountpoints. You can make multiple copies of these as you go along, before upgrades, during patches, or major config changes. Real Data like /usr/home, /var/log or (in my case) /var/db/* are in separate datasets and will always be present and up to date, irrespective of which boot environment you are currently running under. For example, I have a 11.1R boot environment, and about 3-4 12.0-CURRENT ones over the last few weeks, in case something unexpected breaks for me. At boot time, the loader (thanks Allan Jude IIRC for implementing this) allows you to choose between those copies so you can simply switch back to the working one and decide what you want to do with the borked one. see [2] for some really cool tricks from Michael Dexter about mounting that into bhyve and then "live testing" before upgrading. 4. beinstall.sh script from Will Andrews If you're building from source, then you can combine #3 with [3] and have your next incarnation automatically installed not into your existing working / dataset but into its own dataset, cloned off your current /, and upgraded, all within the boot environment. This is so awesome it's changed my world (pun intended), I install a new CURRENT every 3-4 days now because rolling back is absolutely painless. I borrowed xmj@ [4] tips on using ccache and metamode, which makes incremental builds pretty quick. I can later on mount the /usr/src and /usr/obj over NFS to other lesser powered systems like my laptop, and beinstall.sh there too. LIKE A BOSS. Here's my current hacky scripts to build world & current, and stash everything into a boot environment. It works for me, but I'm hardly a build-from-source guru so maybe that could be improved [5] [1]: http://mfsbsd.vx.sk/ [2]: http://www.callfortesting.org/bhyve-boot-environments/ [3]: https://svnweb.freebsd.org/base/head/tools/build/beinstall.sh?view=markup [4]: https://xmj.github.io/articles/sysadmin/builds_ccache_memcached.html [5]: https://gist.github.com/dch/99a0e3c42e971a08dd922aab20670d7d