From owner-freebsd-questions@freebsd.org Mon Oct 3 15:31:56 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0350AF36FC for ; Mon, 3 Oct 2016 15:31:56 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 469D880B for ; Mon, 3 Oct 2016 15:31:55 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id u93FVmmm068097; Tue, 4 Oct 2016 02:31:48 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Tue, 4 Oct 2016 02:31:48 +1100 (EST) From: Ian Smith To: Warren Block cc: David Christensen , Perry Hutchison , freebsd-questions@freebsd.org Subject: Re: FreeBSD-10.3-RELEASE-i386-memstick.img installer changes contents of USB flash drive In-Reply-To: Message-ID: <20161004010853.E6806@sola.nimnet.asn.au> References: <20160929014801.W6806@sola.nimnet.asn.au> <20161001235138.N6806@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Oct 2016 15:31:57 -0000 On Sat, 1 Oct 2016 11:39:35 -0600, Warren Block wrote: > On Sun, 2 Oct 2016, Ian Smith wrote: > > > > A secondary copy of the partition table is saved at the end of a > > > GPT-partitioned drive. This is a problem for GPT when a binary image is > > > written to a device that is not exactly the same size as the image. That > > > is pretty much always the case with installer images. > > > > Indeed. MBR/BSD images don't have this problem because all metadata is > > at the beginning of and within the image in the case of UFS superblocks. > > So an image can be copied to a device like /dev/da0, in the case of the > > hitherto dedicated drive /dev/da0a, or as easily to a slice like da0s1 > > where it becomes da0s1a, as long as there's enough room on the memstick. > > > > A GPT image seems more of a whole-disk image rather than (what equally > > can serve as) a single partition image. It must have a primary GPT (all > > 34 sectors?) but do you know if these images already contain the full 34 > > sector secondary GPT? i.e, I don't see why only the last sector of the > > copied image file was changed, with no update to any of the prior 33? > > Most of those 34 blocks (not a fixed size, by the way) contain partition > information. But there are global flags and things like the PMBR which could > change. Without knowing exactly what that byte does, it's a guess. Ok. I've been using the Wikipedia page as reference, as I tend to find diagrams more useful: https://en.wikipedia.org/wiki/GUID_Partition_Table I find it slightly weird that the secondary GPT (there) uses LBA-33 for the first four partitions; if it was my design (cough) I'd have reversed that also, and used LBA-2 for the first four, etc. But then, I've never been enamoured of this (problematic) end-of-physical-disk positioning. > Note that we don't know when that change is happening. It could be the > machine checking the secondary GPT on the memory stick that is helpfully and > silently "fixing" something when it recognizes the stick, before it is read. As David mentioned, doing so silently would be less than helpful (POLA!) > It could be malicious or buggy firmware in any of several places, including > the UEFI or BIOS or the memory stick micontroller itself. The point is that > we don't know that it is the FreeBSD installer making that change. Right, so I guess we need more people to test this on their machines, but I'm not in a position to do so lately .. maybe David's PR will help. And Perry's issue is also outstanding, and rather inexplicable so far. > I remember a few years back that someone helped me understand what was going > on, but forget the details (and, unfortunately, who it was). Many of us could relate to that :) "I thought I met a man .. who said he knew a man .. who knew what was going on .." David Crosby - IIRC! > > Personally I'm wanting to get to a memstick with 2 or 3 bootable slices > > for installers (amd64 and i386) - as least the amd64 one from dvd1 with > > packages - plus a mountable r/w data partition. Using only native tools > > currently implies using boot0, thus MBR/BSD schema. Converting a new > > GPT-scheme memstick image to ONE slice on my new 16GB memstick is what I > > need to figure out, possibly by means of using a tar pipeline similar to > > dvd1_to_memstick.sh. I hope to get time to look at 11.0 make release > > memstick scripts, but I'd appreciate any clues / ideas meanwhile .. > > I'd think you could just dd(1) the contents of the single installer disk > partition into a big-enough MBR partition. Repeat for each image, then use > boot0cfg. Sure, for old-style raw BSD images, but that won't work for images with: 1) PMBR, 2) primary GPT, 3) efifat partition, 4) gptboot partition then 5) the UFS partition - which is all we want to copy to a bootable BSD slice - then 6) 1M(??) swap. We need to use /boot/boot bootblocks for each slice; still, it may be easy to locate and dd just that partition out of the mdconfig'd image - or mount it and copy; some tests needed. > It really is a shame we still require separate installers for amd64 and i386, > a single installer can install either. It's got a lot simpler on 11 src/release, using mkimg(1) rather than scripting gpart steps as before, but both memstick and iso still differ between i385 and amd64. Perhaps the EFI issues are the main difference? root@x200:/usr/head/src/release # diff -u i386/make-memstick.sh amd64/make-memstick.sh --- i386/make-memstick.sh 2016-05-22 06:27:55.712694000 +1000 +++ amd64/make-memstick.sh 2016-05-22 06:27:55.713721000 +1000 @@ -7,7 +7,7 @@ # # Usage: make-memstick.sh # -# $FreeBSD: head/release/i386/make-memstick.sh 293188 2016-01-05 03:20:45Z gjb $ +# $FreeBSD: head/release/amd64/make-memstick.sh 293188 2016-01-05 03:20:45Z gjb $ [.. clipped and reformatted for clarity on the list ..] -mkimg -s gpt -b ${1}/boot/pmbr \ -p freebsd-boot:=${1}/boot/gptboot -p freebsd-ufs:=${2}.part \ -p freebsd-swap::1M -o ${2} +mkimg -s gpt -b ${1}/boot/pmbr -p efi:=${1}/boot/boot1.efifat \ -p freebsd-boot:=${1}/boot/gptboot -p freebsd-ufs:=${2}.part \ -p freebsd-swap::1M -o ${2} cheers, Ian