From owner-freebsd-xen@freebsd.org Tue Oct 25 09:33:06 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E287AC1F381 for ; Tue, 25 Oct 2016 09:33:06 +0000 (UTC) (envelope-from prvs=099ae7f8d=roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.eu.citrix.com [185.25.65.24]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5D4EF9E0 for ; Tue, 25 Oct 2016 09:33:05 +0000 (UTC) (envelope-from prvs=099ae7f8d=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.31,545,1473120000"; d="scan'208";a="33669072" Date: Tue, 25 Oct 2016 11:32:57 +0200 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: CC: Subject: Re: FreeBSD Xen DomU boot from drbd fails with error 19 Message-ID: <20161025093257.eu2b25hs7svv3g46@mac> References: <20161024170219.j5btql5s6f6xeylb@MacBook-Air-de-Roger.local> <9c49bd4e-abfe-3d9a-70f5-12895586f389@bigfoot.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <9c49bd4e-abfe-3d9a-70f5-12895586f389@bigfoot.com> User-Agent: NeoMutt/20161003 (1.7.0) X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-DLP: AMS1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2016 09:33:07 -0000 On Mon, Oct 24, 2016 at 08:50:01PM -0400, niek@bigfoot.com wrote: > On 24-10-2016 13:02, Roger Pau Monné wrote: > > On Sat, Oct 22, 2016 at 04:14:33PM -0400, niek@bigfoot.com wrote: > >> Running a FreeBSD 11.0-RELEASE HVM guest, using ufs, on Xen 4.4 (debian > >> jessie) on LVM with drbd 8.4. The FreeBSD guest boots fine from LVM > >> using 'disk' /dev// in domU config. > >> > >> I have a number of Linux guests running on drbd (8.4). Booting FreeBSD > >> from the drbd device associated with drbd (with 'disk' /dev/drbdx) > >> fails. I know I can't use drbd: with HVM guest. > >> > >> Trying to mount root from ufs:/dev/xbd0p2 [rw]... > >> mountroot: waiting for device /dev/xbd0p2... > >> Mounting from ufs:/dev/xbd0p2 failed with error 19. > > > > Ops, hit send to soon. Can you also paste the output of `xenstore-ls -fp` > > executed from Dom0 when the guest is in this state? > > > > Thanks, Roger. > > > > Thanks for replying. > > Sorry for the long post. > There is one Linux PV guest on this host, not on drbd, but there are > configured drbd devices on this host that belong to linux PV guests on > another host also running debian jessie. They boot from drbd without > problems. > > To collect more data I created a fresh FreeBSD HVM guest on the same > host. Interestingly, this guest does boot from drbd. You'll find it as > domain 16 (vps13) in the attached output of xenstore-ls. > > The HVM guest that fails to mount root is domain 17 (vps10). > The error message is: > Trying to mount root from ufs:/dev/xbd0p2 [rw]... > GEOM: xbd0: corrupt or invalid GPT detected. > GEOM: xbd0: GPT rejected -- may not be recoverable. > > I should mention that there seems to be a conflict with the second GEOM > GPT table in FreeBSD and the drbd metadata both writing to the end of > the partition (we are using internal metadata). After creating metadata > (drbdadm create-md) the kernel complains about the secondary GPT table, > but that does not prevent it from mounting the fs. > > Inside the domu this can be resolved by issuing 'gpart recover', but > this in turn seems to destroy the drbd metadata, which is apparent by > the need to re-create md and re-sync at the next boot. > > Perhaps we should use external metadata in this case. I have no idea how drbd works, but it certainly shouldn't store internal metadata in the device presented to the guest, or else havoc is going to happen for sure. I would suggest that you try this "external metadata" mode. > As an aside, for your information, we tried to run the FreeBSD 11.0 > guest in PVH mode as well, having followed the instructions on > https://wiki.xen.org/wiki/FreeBSD_PVH. > > The guest failed to start with xl reporting en error: > xc: error: elf_xen_addr_calc_check: ERROR: ELF start or entries are out > of bounds.: Invalid kernel > Output of readelf -l /xenkernels/freebsd/kernel-11.0-RELEASE: > Elf file type is EXEC (Executable file) > Entry point 0xffffffff802ff000 > There are 6 program headers, starting at offset 64 That's expected, Xen 4.4 has some PVH support, but that's not enough, IIRC you should use Xen 4.6 or newer. Roger.