From owner-freebsd-current@freebsd.org Tue Dec 4 23:04:16 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDD941324BA9 for ; Tue, 4 Dec 2018 23:04:15 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49C57778F1 for ; Tue, 4 Dec 2018 23:04:15 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1543964639; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=h9S+VTH09BQTPyW3102DZOokT6fQpGiIWe6FQi51uauYYQ2V7IGeF9GkbDZ2wct7jIqlCY6sHye1z aTp8nRRqxtgeDExR75i0EH/EEIZWG5PQUrB7WPkzOFwKpQOxCfMhCXHVNhAAdFS4JbEuMLzXgybNBq Rrsg3+Mq95tN7iCiQW4Bw0E3S4YjFFs5cEHHzJaLbqVYL5JFf9vNR5qkGWaZpFAbq056ww90/Xnhlv 4YeNeX+r9Zsf/QcDgHxRrwPczLHLiaqsXzWuskR9MFmoFf7nIlXowuGmlLQ2XzJenxouuQ2BTOtzdd TJLqaRDrUFAnG8H7QIPjkHbXFUKOj1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=UekFezDpbrIvUhFqilQIl02VxyNj60tXTGYTHrYIH/A=; b=YTjGA0o/BJ0e3ylICf0esCqiLEgdFOl7Z8K69g/k6HL7+yyhIb28GdOq6aObvWtR/COAJ2FExZa+t ZqFRHiZtrOXDtGB4rV0UVd4nq58ektXvNjY7RfikvsKv4+mhyIfXEMj430mWtT7EYngm8HxSuqX5Tz myO+aoyN6WgPiV371M3nmYW/jHAmwr9quuy3LIooW7xFT0EaGQ6OsDjNBIM2ZRUzq7b1LDlDCBcjB2 r7LkrNPci4F+k2HVF+OrJmF34+MMvdKikrZqOkPeyxD09fWO6u7zngh+k/yXoWuIs1F8BUfWoRii4+ HF9ulLMwogiebCUjCAQUNze+WXy03Rg== ARC-Authentication-Results: i=1; outbound2.eu.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=UekFezDpbrIvUhFqilQIl02VxyNj60tXTGYTHrYIH/A=; b=dzvNyoiAE6nUKqyfvKeTEk8BgrsH4E9H/0F8kXShzXkGtsoSeTrMtAhz1C8if3qA4u5obn+yRzt15 O3Uu33CIbJdcVmunLx56c3R+dAe0SGpdIh539h8vGK4fBbTEo2yjt+S9So/56jCzuTpazR8y9dRK8g zOVFwvozoTguBQjslPiLpb/SuhFfA70fYniWdF341hwSxSUMbYPN0TWrigex4z8sD3YOi5LQHgnlgK CWJZSVDSCObGCw+CqxcC+8+2woz9Kt4XtYoFCWeb+d+QoPGdWGTLOmxj1TFwHNFhUsSEfxl0IdE4h/ 9uktP3rrR6mMVa3RucsXbYTY+avlEFw== X-MHO-RoutePath: aGlwcGll X-MHO-User: dcd36052-f818-11e8-a887-bd2f23b465e5 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound2.eu.mailhop.org (Halon) with ESMTPSA id dcd36052-f818-11e8-a887-bd2f23b465e5; Tue, 04 Dec 2018 23:03:55 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id wB4KJDeC056026; Tue, 4 Dec 2018 13:19:13 -0700 (MST) (envelope-from ian@freebsd.org) Message-ID: <1543954753.1860.243.camel@freebsd.org> Subject: Re: Boot loader stuck after first stage upgrading 11.2 to 12.0-RC2 From: Ian Lepore To: Toomas Soome , Mark Martinec Cc: freebsd-current , freebsd-stable@freebsd.org Date: Tue, 04 Dec 2018 13:19:13 -0700 In-Reply-To: References: <22f5b92a09ea4d62ac3feb74457067f7@ijs.si> <5EEBAFC0-4FA3-4219-A918-7376F4223656@me.com> <0F5FCC70-EADB-4F9E-A391-F1A73BE5608F@me.com> Content-Type: text/plain; charset="windows-1251" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 49C57778F1 X-Spamd-Result: default: False [-0.07 / 15.00]; TAGGED_RCPT(0.00)[freebsd]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.19)[-0.193,0]; ASN(0.00)[asn:16509, ipnet:52.58.0.0/15, country:US]; NEURAL_SPAM_SHORT(0.36)[0.355,0]; NEURAL_HAM_LONG(-0.23)[-0.228,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Dec 2018 23:04:16 -0000 On Tue, 2018-12-04 at 21:51 +0200, Toomas Soome via freebsd-stable wrote: > > > > > On 4 Dec 2018, at 19:59, Mark Martinec > i> wrote: > > > > > > > > > > > > > 2018-11-29 18:43, Toomas Soome wrote: > > > > > > > > > > I just did push biosdisk updates to stable/12, I wonder if > > > > > you could > > > > > test those bits… > > Myself wrote: > > > > > > > > > > > Thank you!  I haven't tried it yet, but I wonder whether this > > > > fix was > > > > already incorporated into 12.0-RC3, which would make my rescue > > > > easier. > > > > Otherwise I can build a stable/12 on another host and > > > > transplant > > > > the problematic file(s) to the affected host - if I knew which > > > > files > > > > to copy. > > 2018-12-02 18:59, Toomas wrote: > > > > > > The files are /boot/loader* binaries - to be exact, check which > > > one is > > > linked to /boot/loader. I can provide binaries if needed. > > > [...] > > > rgds, > > > toomas > > I got a maintenance window today so I tried with the new loader, > > and it did not help. > > > > More specifically: > > > > As it comes with 12-RC2, the /boot/loader was hard linked with > > loader_lua. > > Its size is 421888 bytes. So I concentrated on this loader. > > > > I build a fresh stable/12 on another host, and copied the newly > > built loader_lua (425984 bytes) to the /boot directory of the > > affected > > host, deleted the file 'loader', and hard-linked loader_lua to > > loader. > > > > The situation has not changed: the BTX loader lists all BIOS drives > > C..J (disk0..disk7), then a spinner starts and gets stuck forever. > > It never reaches the 'BIOS 635kB/3537856kB available memory' line. > > > > While trying to restore the old /boot from 11.2, I tried booting > > a live image from a 12.0-RC3 memory stick - and the loader got > > stuck again, same as when booting from a disk. > > > > So I had to boot from an 11.2 memstick to be able to regain > > control. > > > >  Mark > > > > > ok, if you could perform 2 tests: > > 1. from loader prompt enter 0x413 0xa000 - @w . cr > > 2. on first spinner, press space and type on boot: prompt: > /boot/loader_4th and see if that will do better > thanks, > toomas > I don't think that will be an option.  If it hasn't gotten to the point of saying how much BIOS available memory there is, it's only halfway through loader main() and has hung before getting to interact(). In fact, if that line hasn't printed, but some disk drives have been listed, it pretty much has to be hung in the "March through the device switch probing for things" loop. If all the disks are listed, then it got through that entry in the devsw, and is likely hanging in the dv_init calls for either the pxedisk or zfsdev devices. -- Ian > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 29 Nov 2018, at 17:01, Mark Martinec > > > > > bsd@ijs.si> wrote: > > > > > > After successfully upgraded three hosts from 11.2-p4 to > > > > > > 12.0-RC2 (amd64, > > > > > > zfs, bios), I tried my luck with one of our production > > > > > > hosts, and ended up > > > > > > with a stuck loader after rebooting with a new kernel > > > > > > (after the first > > > > > > stage of upgrade). > > > > > > These were the steps, and all went smoothly and normally > > > > > > until a reboot: > > > > > > freebsd-update upgrade -r 12.0-RC2 > > > > > > freebsd-update install > > > > > > shutdown -r now > > > > > > While booting, the 'BTX loader' comes up, lists the BIOS > > > > > > drives, > > > > > > then the spinner below the list comes up and begins > > > > > > turning, > > > > > > stuttering, and after a couple of seconds it grinds to a > > > > > > standstill > > > > > > and nothing happens afterwards. > > > > > > At this point the ZFS and the bootstrap loader is supposed > > > > > > to > > > > > > come up, but it doesn't. > > > > > > This host has too zfs pools, the system pool consists of > > > > > > two SSDs > > > > > > in a zfs mirror (also holding a freebsd-boot partition > > > > > > each), the > > > > > > other pool is a raidz2 with six JBOD disks on an LSI > > > > > > controller. > > > > > > The gptzfsboot in both freebsd-boot partitions is fresh > > > > > > from 11.2, > > > > > > both zpool versions are up-to-date with 11.2. The 'zpool > > > > > > status -v' > > > > > > is happy with both pools. > > > > > > After rebooting from an USB drive and reverting the /boot > > > > > > directory > > > > > > to a previous version, the machine comes up normally again > > > > > > with the 11.2-RELEASE-p4. > > > > > > I found a file init.core in the / directory, slightly > > > > > > predating the > > > > > > last reboot with a salvaged system - although it was > > > > > > probably not > > > > > > a cause of the problem, but a consequence of the rescue > > > > > > operation. > > > > > > It is unfortunate that this is a production host, so I > > > > > > can't play > > > > > > much with it. One or two more quick experiments I can > > > > > > probably > > > > > > afford, but not much more. Should I just first wait for the > > > > > > official 12.0 release? Should I try booting with a 12.0 on > > > > > > USB > > > > > > and try to import pools? Suggestions welcome. > > > > > > Now that the /boot has been manually restored to the 11.2 > > > > > > state, > > > > > > A SECOND QUESTION is about freebsd-update, which still > > > > > > thinks we are > > > > > > in the middle of an upgrade procedure. Trying now to just > > > > > > update > > > > > > the 11.2-RELEASE-p4 to 11.2-RELEASE-p5, the fetch > > > > > > complains: > > > > > > # uname -a > > > > > > FreeBSD xxx 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 > > > > > > # > > > > > > # freebsd-version > > > > > > 11.2-RELEASE-p4 > > > > > > # > > > > > > # freebsd-update fetch > > > > > > src component not installed, skipped > > > > > > You have a partially completed upgrade pending > > > > > > Run '/usr/sbin/freebsd-update install' first. > > > > > > Run '/usr/sbin/freebsd-update fetch -F' to proceed anyway. > > > > > > So what is the right way to get rid of all traces of the > > > > > > unsuccessful upgrade, and let freebsd-update believe we are > > > > > > cleanly > > > > > > at 11.2-p4 ?  Removing /var/db/freebsd-update did not help. > > > > > > Mark > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd. > org" >