From owner-freebsd-virtualization@freebsd.org Mon Apr 25 08:47:32 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97FDFB11CB4 for ; Mon, 25 Apr 2016 08:47:32 +0000 (UTC) (envelope-from prvs=916a2d648=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 636A71F89 for ; Mon, 25 Apr 2016 08:47:32 +0000 (UTC) (envelope-from prvs=916a2d648=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.24,532,1454976000"; d="scan'208";a="356159660" Date: Mon, 25 Apr 2016 10:47:23 +0200 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Marcin Cieslak CC: Subject: Re: Booting r298488 as Xen Dom0 may break ZFS pool? Message-ID: <20160425084714.64ejioyqgquwu4gw@mac> References: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.0-neo (2016-04-07) X-DLP: MIA2 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 08:47:32 -0000 On Sat, Apr 23, 2016 at 10:40:20PM +0000, Marcin Cieslak wrote: > On a freshly installed (via upgrade from 10.3 from source) > -CURRENT on this machine: > > FreeBSD 11.0-CURRENT #0 r298488: Sat Apr 23 11:10:01 UTC 2016 > root@o.saper.info:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) > WARNING: WITNESS option enabled, expect reduced performance. > VT(vga): resolution 640x480 > CPU: Intel(R) Xeon(R) CPU E31245 @ 3.30GHz (3300.09-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x206a7 Family=0x6 Model=0x2a Stepping=7 > Features=0xbfebfbff > Features2=0x1fbae3ff > AMD Features=0x28100800 > AMD Features2=0x1 > XSAVE Features=0x1 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > TSC: P-state invariant, performance statistics > real memory = 17179869184 (16384 MB) > avail memory = 16475140096 (15711 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs > FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads > random: unblocking device. > ioapic0 irqs 0-23 on motherboard > > ... i am trying to boot the system as Dom0 under Xen (4.5.2_2 > installed via pkg install). > > The kernel boots in one of four cases, mostly > though I don't get a block cursor after Xen messages > and the machine sits and waits. > > > After trying Xen suddenly the system no longer > boots again: > > Booting from local disk... > PXE-M0F: Existing Intel Boot Agent. > > ZFS: i/o error - all block copies unavailable > ZFS: i/o error - all block copies unavailable > ZFS: i/o error - all block copies unavailable > > Can't find /boot/zfsloader > > FreeBSD/x86 boot > Default: zroot:/boot/kernel/kernel > boot: > ZFS: i/o error - all block copies unavailable I'm currently running a r298464 world+kernel with ZFS on root and have seen no issues so far. > (...) > > The zpool imports without problems when > booting from the rescue mfsbsd (10.3): > > pool: zroot > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > ada0p3 ONLINE 0 0 0 > ada1p3 ONLINE 0 0 0 > > errors: No known data errors > > gpart layout: > > => 34 5860533101 ada0 GPT (2.7T) > 34 1024 1 freebsd-boot (512K) > 1058 4194304 2 freebsd-swap (2.0G) > 4195362 5856337773 3 freebsd-zfs (2.7T) > > => 34 5860533101 ada1 GPT (2.7T) > 34 1024 1 freebsd-boot (512K) > 1058 4194304 2 freebsd-swap (2.0G) > 4195362 5856337773 3 freebsd-zfs (2.7T) > > I have managed to make zpool boot again by doing voodoo > similar to this one: > > [root@rescue ~]# zpool import -R /mnt zroot > [root@rescue ~]# mount -t devfs devfs /mnt/dev > [root@rescue ~]# chroot /mnt /bin/tcsh > > (... Running make install in /usr/src/sys/boot ...) > > root@rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada0 > partcode written to ada0p1 > root@rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada1 > partcode written to ada1p1 > root@rescue:/ # exit > [root@rescue ~]# umount /mnt/dev > [root@rescue ~]# zpool export zroot > [root@rescue ~]# reboot > > Why zpool metadata get corrupted? TBH, I have no idea. Can you also paste the log of the system (Xen + FreeBSD) when it fails to boot? If that's not possible, can you at least paste the output of `xl dmesg` when booted correctly under Xen? What operations did you perform when the system booted correctly using FreeBSD/Xen? Does the disk get corrupted even if the system fails to boot? AFAICT, it seems like it's only the bootcode that gets corrupted, is that right? Roger.