From owner-freebsd-virtualization@freebsd.org Sat Apr 23 22:40:24 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81F71B1A3E8 for ; Sat, 23 Apr 2016 22:40:24 +0000 (UTC) (envelope-from saper@saper.info) Received: from m.saper.info (m.saper.info [IPv6:2a01:4f8:a0:7383::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "m.saper.info", Issuer "Marcin Cieslak 2016" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 158051491 for ; Sat, 23 Apr 2016 22:40:23 +0000 (UTC) (envelope-from saper@saper.info) Received: from m.saper.info (saper@m.saper.info [IPv6:2a01:4f8:a0:7383::]) by m.saper.info (8.15.2/8.15.2) with ESMTPS id u3NMeLJi053183 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 23 Apr 2016 22:40:21 GMT (envelope-from saper@saper.info) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=saper.info; s=Sep2014; t=1461451221; bh=fECJv5kbSx5vePjCV0kB0tnQ4YATLQX5voclO3+RNpA=; h=Date:From:To:Subject; b=oriTPg4FdO64aYLO/awOFEPLvzI8SrZPOI65hhNniyqvwpkiAhxZR44+cl3xel0fm 1gzdFJ+t2fYOFA1eBVLwf/V2dh11/UFFbSMIPsxWLPTLPf6M7MEn7V3IYz/FNmjljw +Nb6SS666TjM26tq6gwmMdnJuFrA9nDPcJj+9CT0= Received: from localhost (saper@localhost) by m.saper.info (8.15.2/8.15.2/Submit) with ESMTP id u3NMeKDd053180 for ; Sat, 23 Apr 2016 22:40:21 GMT (envelope-from saper@saper.info) X-Authentication-Warning: m.saper.info: saper owned process doing -bs Date: Sat, 23 Apr 2016 22:40:20 +0000 From: Marcin Cieslak To: freebsd-virtualization@freebsd.org Subject: Booting r298488 as Xen Dom0 may break ZFS pool? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Apr 2016 22:40:24 -0000 On a freshly installed (via upgrade from 10.3 from source) -CURRENT on this machine: FreeBSD 11.0-CURRENT #0 r298488: Sat Apr 23 11:10:01 UTC 2016 root@o.saper.info:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) WARNING: WITNESS option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E31245 @ 3.30GHz (3300.09-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206a7 Family=0x6 Model=0x2a Stepping=7 Features=0xbfebfbff Features2=0x1fbae3ff AMD Features=0x28100800 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 16475140096 (15711 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads random: unblocking device. ioapic0 irqs 0-23 on motherboard ... i am trying to boot the system as Dom0 under Xen (4.5.2_2 installed via pkg install). The kernel boots in one of four cases, mostly though I don't get a block cursor after Xen messages and the machine sits and waits. After trying Xen suddenly the system no longer boots again: Booting from local disk... PXE-M0F: Existing Intel Boot Agent. ZFS: i/o error - all block copies unavailable ZFS: i/o error - all block copies unavailable ZFS: i/o error - all block copies unavailable Can't find /boot/zfsloader FreeBSD/x86 boot Default: zroot:/boot/kernel/kernel boot: ZFS: i/o error - all block copies unavailable (...) The zpool imports without problems when booting from the rescue mfsbsd (10.3): pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors gpart layout: => 34 5860533101 ada0 GPT (2.7T) 34 1024 1 freebsd-boot (512K) 1058 4194304 2 freebsd-swap (2.0G) 4195362 5856337773 3 freebsd-zfs (2.7T) => 34 5860533101 ada1 GPT (2.7T) 34 1024 1 freebsd-boot (512K) 1058 4194304 2 freebsd-swap (2.0G) 4195362 5856337773 3 freebsd-zfs (2.7T) I have managed to make zpool boot again by doing voodoo similar to this one: [root@rescue ~]# zpool import -R /mnt zroot [root@rescue ~]# mount -t devfs devfs /mnt/dev [root@rescue ~]# chroot /mnt /bin/tcsh (... Running make install in /usr/src/sys/boot ...) root@rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada0 partcode written to ada0p1 root@rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada1 partcode written to ada1p1 root@rescue:/ # exit [root@rescue ~]# umount /mnt/dev [root@rescue ~]# zpool export zroot [root@rescue ~]# reboot Why zpool metadata get corrupted? -- Marcin Xen commandline from /boot/loader.conf: xen_kernel="/boot/xen" xen_cmdline="dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all" dmesg when started under Xen successfully: Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #0 r298488: Sat Apr 23 11:10:01 UTC 2016 root@o.saper.info:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) WARNING: WITNESS option enabled, expect reduced performance. VT(vga): text 80x25 CPU: Intel(R) Xeon(R) CPU E31245 @ 3.30GHz (3300.02-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206a7 Family=0x6 Model=0x2a Stepping=7 Features=0x1fc3ebff Features2=0x9fb82283 AMD Features=0x20100800 AMD Features2=0x1 XSAVE Features=0x1 TSC: P-state invariant, performance statistics Hypervisor: Origin = "XenVMMXenVMM" real memory = 2152079360 (2052 MB) avail memory = 2025705472 (1931 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s)