From owner-freebsd-stable@freebsd.org Tue Dec 4 17:59:37 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF4D81314BEE; Tue, 4 Dec 2018 17:59:36 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DE64D863D8; Tue, 4 Dec 2018 17:59:35 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from amavis-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 438V4P4jwKzb3; Tue, 4 Dec 2018 18:59:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1543946369; x=1546538370; bh=XDd t8WFxv8n8JLAG0742BEpkW+aoWSd+70CsBVj9RrM=; b=JBvW1VrcDw7/EJ/UypB Xph8IJJ9rUTlyPsVPzEGiAzOsF9EuF7jhuKXAuZPjVTxxBq05YnNNKCqO4WyOn8O yD/y9KUJzGNHj7Os/L9Wa3d7eObXp8ePm3+xqjz+HkjjH+8rF31bx0iTKtVjXaHf IHMtjNOf6nrhd2a8CY6/hJOk= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026) with LMTP id rl5agQ3noaQJ; Tue, 4 Dec 2018 18:59:29 +0100 (CET) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP id 438V4K4xg3zb1; Tue, 4 Dec 2018 18:59:29 +0100 (CET) Received: from nabiralnik.ijs.si (nabiralnik.ijs.si [IPv6:2001:1470:ff80::80:16]) by mildred.ijs.si (Postfix) with ESMTP id 438V4K2FXjzr8; Tue, 4 Dec 2018 18:59:29 +0100 (CET) Received: from neli.ijs.si (2001:1470:ff80:88:21c:c0ff:feb1:8c91) by nabiralnik.ijs.si with HTTP (HTTP/1.1 POST); Tue, 04 Dec 2018 18:59:29 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Tue, 04 Dec 2018 18:59:29 +0100 From: Mark Martinec To: Toomas Soome Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Boot loader stuck after first stage upgrading 11.2 to 12.0-RC2 Organization: Jozef Stefan Institute In-Reply-To: <0F5FCC70-EADB-4F9E-A391-F1A73BE5608F@me.com> References: <22f5b92a09ea4d62ac3feb74457067f7@ijs.si> <5EEBAFC0-4FA3-4219-A918-7376F4223656@me.com> <0F5FCC70-EADB-4F9E-A391-F1A73BE5608F@me.com> Message-ID: X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.3.1 X-Rspamd-Queue-Id: DE64D863D8 X-Spamd-Result: default: False [-3.25 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[ijs.si]; RCVD_COUNT_FIVE(0.00)[6]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2001:1470:ff80::25]; NEURAL_HAM_LONG(-0.99)[-0.992,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_SOME(0.00)[]; HAS_ORG_HEADER(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ijs.si:+]; DMARC_POLICY_ALLOW(-0.50)[ijs.si,none]; RCVD_IN_DNSWL_NONE(0.00)[5.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.f.f.0.7.4.1.1.0.0.2.list.dnswl.org : 127.0.11.0]; MX_GOOD(-0.01)[mail4.ijs.si,mail.ijs.si]; NEURAL_HAM_SHORT(-0.25)[-0.248,0]; FREEMAIL_TO(0.00)[me.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.00)[country: SI(-0.00)]; ASN(0.00)[asn:2107, ipnet:2001:1470::/32, country:SI]; TAGGED_FROM(0.00)[freebsd]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Dec 2018 17:59:37 -0000 >> 2018-11-29 18:43, Toomas Soome wrote: >>> I just did push biosdisk updates to stable/12, I wonder if you could >>> test those bits… Myself wrote: >> Thank you! I haven't tried it yet, but I wonder whether this fix was >> already incorporated into 12.0-RC3, which would make my rescue easier. >> Otherwise I can build a stable/12 on another host and transplant >> the problematic file(s) to the affected host - if I knew which files >> to copy. 2018-12-02 18:59, Toomas wrote: > The files are /boot/loader* binaries - to be exact, check which one is > linked to /boot/loader. I can provide binaries if needed. > [...] > rgds, > toomas I got a maintenance window today so I tried with the new loader, and it did not help. More specifically: As it comes with 12-RC2, the /boot/loader was hard linked with loader_lua. Its size is 421888 bytes. So I concentrated on this loader. I build a fresh stable/12 on another host, and copied the newly built loader_lua (425984 bytes) to the /boot directory of the affected host, deleted the file 'loader', and hard-linked loader_lua to loader. The situation has not changed: the BTX loader lists all BIOS drives C..J (disk0..disk7), then a spinner starts and gets stuck forever. It never reaches the 'BIOS 635kB/3537856kB available memory' line. While trying to restore the old /boot from 11.2, I tried booting a live image from a 12.0-RC3 memory stick - and the loader got stuck again, same as when booting from a disk. So I had to boot from an 11.2 memstick to be able to regain control. Mark >>>> On 29 Nov 2018, at 17:01, Mark Martinec >>>> wrote: >>>> After successfully upgraded three hosts from 11.2-p4 to 12.0-RC2 >>>> (amd64, >>>> zfs, bios), I tried my luck with one of our production hosts, and >>>> ended up >>>> with a stuck loader after rebooting with a new kernel (after the >>>> first >>>> stage of upgrade). >>>> These were the steps, and all went smoothly and normally until a >>>> reboot: >>>> freebsd-update upgrade -r 12.0-RC2 >>>> freebsd-update install >>>> shutdown -r now >>>> While booting, the 'BTX loader' comes up, lists the BIOS drives, >>>> then the spinner below the list comes up and begins turning, >>>> stuttering, and after a couple of seconds it grinds to a standstill >>>> and nothing happens afterwards. >>>> At this point the ZFS and the bootstrap loader is supposed to >>>> come up, but it doesn't. >>>> This host has too zfs pools, the system pool consists of two SSDs >>>> in a zfs mirror (also holding a freebsd-boot partition each), the >>>> other pool is a raidz2 with six JBOD disks on an LSI controller. >>>> The gptzfsboot in both freebsd-boot partitions is fresh from 11.2, >>>> both zpool versions are up-to-date with 11.2. The 'zpool status -v' >>>> is happy with both pools. >>>> After rebooting from an USB drive and reverting the /boot directory >>>> to a previous version, the machine comes up normally again >>>> with the 11.2-RELEASE-p4. >>>> I found a file init.core in the / directory, slightly predating the >>>> last reboot with a salvaged system - although it was probably not >>>> a cause of the problem, but a consequence of the rescue operation. >>>> It is unfortunate that this is a production host, so I can't play >>>> much with it. One or two more quick experiments I can probably >>>> afford, but not much more. Should I just first wait for the >>>> official 12.0 release? Should I try booting with a 12.0 on USB >>>> and try to import pools? Suggestions welcome. >>>> Now that the /boot has been manually restored to the 11.2 state, >>>> A SECOND QUESTION is about freebsd-update, which still thinks we are >>>> in the middle of an upgrade procedure. Trying now to just update >>>> the 11.2-RELEASE-p4 to 11.2-RELEASE-p5, the fetch complains: >>>> # uname -a >>>> FreeBSD xxx 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 >>>> # >>>> # freebsd-version >>>> 11.2-RELEASE-p4 >>>> # >>>> # freebsd-update fetch >>>> src component not installed, skipped >>>> You have a partially completed upgrade pending >>>> Run '/usr/sbin/freebsd-update install' first. >>>> Run '/usr/sbin/freebsd-update fetch -F' to proceed anyway. >>>> So what is the right way to get rid of all traces of the >>>> unsuccessful upgrade, and let freebsd-update believe we are cleanly >>>> at 11.2-p4 ? Removing /var/db/freebsd-update did not help. >>>> Mark