From owner-freebsd-usb@freebsd.org Sat Oct 17 17:08:52 2015 Return-Path: Delivered-To: freebsd-usb@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D90C0A17750 for ; Sat, 17 Oct 2015 17:08:52 +0000 (UTC) (envelope-from tontaubelists@gmail.com) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6F940AE1 for ; Sat, 17 Oct 2015 17:08:52 +0000 (UTC) (envelope-from tontaubelists@gmail.com) Received: by wicfv8 with SMTP id fv8so27982006wic.0 for ; Sat, 17 Oct 2015 10:08:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=to:from:subject:message-id:date:user-agent:mime-version :content-type:content-transfer-encoding; bh=d5q5qrK9szjwxWUO3zbdWSRSmeUTUYoIE3LAp4WG4Pw=; b=qeWEIe+oyakPrPjgYwXz5f7/CkbqBXXMvT/npTHrKoJ46XbdiQhWowUyDUZhOyUDWu 3wCJYOmlWM6Lxp15Wgvdr9qjggBmyc08KOgH2PuIn1MQEq1C5I1yv72AWXt44nTLJIB2 rtrbkHN5xH+xwBP13zWjq8QlcQhSsOr3mUNk9Q+cKRi0SRrKkFm8Ucb/KIGwIki5ZJDJ CdOLYjVkYw8S6p8ffj53v2H0m0N3VA6kg4eE8zN8ENcRhSAsPo/6Wal4dlHGUJNs3nIs c1nskm0vWiZGg6sasGiJQ2b4FL9k6JUWfYtyAXzhj5Yd8KY05SJcL1ao3yq801nees7N 5gpw== X-Received: by 10.180.87.102 with SMTP id w6mr11996576wiz.88.1445101730880; Sat, 17 Oct 2015 10:08:50 -0700 (PDT) Received: from nb-tontaube.local ([217.194.53.67]) by smtp.googlemail.com with ESMTPSA id ga2sm7796298wib.5.2015.10.17.10.08.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 17 Oct 2015 10:08:49 -0700 (PDT) To: freebsd-usb@freebsd.org From: Philipp Maechler Subject: External USB Harddisk ZFS boot problem X-Enigmail-Draft-Status: N1110 Message-ID: <5622809C.7020501@gmail.com> Date: Sat, 17 Oct 2015 19:08:44 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-usb@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FreeBSD support for USB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2015 17:08:53 -0000 Good evening, I have a problem and a manual an nasty work around, and like to have some hints how/where to digg deeper. FreeBSD 10.2 (GENERIC freshly installed) and root zfs pool on a external usb harddisk drive. I try to boot from this zfs pool. More details about the hardware further down. This zfs pool is a zfs send/receive copy of a working zfs boot setup (but it was before an 10.1 - note to myself: never change 2 things at the same time [duplicate]) The new harddisk is setup properly (boot-loader etc) and also the zpool cache file is correctly up-to-date and the bootfs property is set. What happens is that the boot loader works fine, then it loads the kernel and the kernel starts. At this time, the drive switch off and the kernel loads further and starts to wait for root an usbus0 and usbus1 Unsuccessful boot timeline: If I just leave it go, the drive turns on, and as soon as I read "da0 at umass xy" it turns off again and the system starts to throw some errors [1] I don't know what happens at this point, I even suspect a (firmware) crash of the drive (but how to proof? and why doesn't this happen when I connect the disk to an powered up system?). After some time, the disk powers on again (probably by some kind of watchdog) and then the prompt asking for the rootfs appears. Shortly after da0 appears again. I can then specify the zfs:zroot as root fs and it will be recognized and found, because it starts init. But then the hostid line appears and after that it doesn't continue at all (why?) Workaround: However, if I disconnect the drive after the kernel loading turns it off the first time and reconnect it immediately, the drive immediately switch on after connecting and is ready for the rootfs loaded instantly and the system boots up fine. (I stretched the time for the workaround a little bit with hw.usb.timings.extra_power_up_time set to 5000 instead of 20). Details about Hardware see [2]. UFS usb boot works fine (at least from an usb stick loaded with the FreeBSD 10.2 release). Also a patch to correctly route the device to xhci instead of ehci didn't solve the boot problem. I tried: * adding a powered usb hub to exclude power issues * changed order of usb keyboard and usb drive on the 4 ports * played around with all hw.usb.timings options (increased them one by one to 10'000 or all togher by factor 50 * enabled usb verbose outputs for boot, enabled kernel verbose output I'm wondering if you can give me any hints for tracing the problem further, as the workaround is a manual intervention (not sexy at all as it is also for a remote nas). What hints do you have? What information do you need? Please keep in mind, the system is able to boot with the workaround, so it is working at some point. Please specify if you think about the issue about why the drive is powering down again or about not booting properly after mounting the rootfs. Some additional outputs below [3]. I very appreciate any kind of link, hint or even if you like a video of a slowed-down boot (setting console to "vidconsole,comconsole" and serial_speed to 3200 ;-) ) With best regards, Philipp [1] The drive get recognized by kernel and shutdowns again (see MOMENT of power OFF) umass1: on usbus1 umass1: SCSI over Bulk-Only; quirks = 0xc101 umass1:1:1:-1: Attached to scbus1 da1 at umass-sim1 bus 1 scbus1 target 0 lun 0 da1: Fixed Direct Access SPC-4 SCSI device da1: Serial Number NA820YR2 da1: 40.000MB/s transfers da1: 3815447MB (7814037167 512 byte sectors: 255H 63S/T 486401C) da1: quirks=0x2 === MOMENT OF POWER OFF === (da1:umass-sim1:1:0:0): READ(16). CDB: 88 00 00 00 00 01 d1 c0 be ae 00 00 00 01 00 00 (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error (da1:umass-sim1:1:0:0): Retrying command (da1:umass-sim1:1:0:0): READ(16). CDB: 88 00 00 00 00 01 d1 c0 be ae 00 00 00 01 00 00 (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error (da1:umass-sim1:1:0:0): Retrying command (da1:umass-sim1:1:0:0): READ(16). CDB: 88 00 00 00 00 01 d1 c0 be ae 00 00 00 01 00 00 (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error (da1:umass-sim1:1:0:0): Retrying command (da1:umass-sim1:1:0:0): READ(16). CDB: 88 00 00 00 00 01 d1 c0 be ae 00 00 00 01 00 00 (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error (da1:umass-sim1:1:0:0): Retrying command (da1:umass-sim1:1:0:0): READ(16). CDB: 88 00 00 00 00 01 d1 c0 be ae 00 00 00 01 00 00 (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error (da1:umass-sim1:1:0:0): Error 5, Retries exhausted (this lines are from a dmesg outboot after a successfull usb stick ufs boot) [2] Intel NUC with a Samsung 4TB Drive connected - fully configured there will be 2 drives in a mirror zpool. Adding a powered usb3 hub didn't help - so probably no power issue. [3] some outputs after a successfull boot with the workaround and both disks connected (ok, I already patched the kernel at this point but just with the patch mentioned in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203650) Index: * loader.conf * usbconfig list * pciconfi -lvc * dmesg # cat /boot/loader.conf zfs_load="YES" vfs.root.mountfrom="zfs:systempool1" kern.maxusers=1024 kern.maxfiles=65536 ipfw_load="YES" ipdivert_load="YES" net.inet.ip.fw.default_to_accept="1" kern.geom.label.disk_ident.enable="0" kern.geom.label.gpt.enable="1" kern.geom.label.gptid.enable="0" #kern.cam.boot_delay=30000 # did not work #kern.cam.scsi_delay=30000 # did not work #vfs.mountroot.timeout=60 # did not work #hw.usb.no_shutdown_wait=1 # not related vfs.zfs.prefetch_disable="1" vfs.zfs.txg.timeout="5" hw.usb.timings.extra_power_up_time=5000 # making the workaround possible # those comments were added in this mail only ;-) # usbconfig list ugen1.1: at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA) ugen0.1: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA) ugen1.2: at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA) ugen0.2: at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (100mA) ugen0.3: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (36mA) ugen0.4: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (36mA) # pciconf -vlc hostb0@pci0:0:0:0: class=0x060000 card=0x20578086 chip=0x16048086 rev=0x09 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 09[e0] = vendor (length 12) Intel cap 0 version 1 vgapci0@pci0:0:2:0: class=0x030000 card=0x20578086 chip=0x16168086 rev=0x09 hdr=0x00 vendor = 'Intel Corporation' class = display subclass = VGA cap 05[90] = MSI supports 1 message cap 01[d0] = powerspec 2 supports D0 D3 current D0 cap 13[a4] = PCI Advanced Features: FLR TP hdac0@pci0:0:3:0: class=0x040300 card=0x20578086 chip=0x160c8086 rev=0x09 hdr=0x00 vendor = 'Intel Corporation' class = multimedia subclass = HDA cap 01[50] = powerspec 2 supports D0 D3 current D0 cap 05[60] = MSI supports 1 message enabled with 1 message cap 10[70] = PCI-Express 1 root endpoint max data 128(128) FLR link x0(x0) xhci0@pci0:0:20:0: class=0x0c0330 card=0x20578086 chip=0x9cb18086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' class = serial bus subclass = USB cap 01[70] = powerspec 2 supports D0 D3 current D0 cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message none0@pci0:0:22:0: class=0x078000 card=0x20578086 chip=0x9cba8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' class = simple comms cap 01[50] = powerspec 3 supports D0 D3 current D0 cap 05[8c] = MSI supports 1 message, 64 bit em0@pci0:0:25:0: class=0x020000 card=0x20578086 chip=0x15a38086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP hdac1@pci0:0:27:0: class=0x040300 card=0x20578086 chip=0x9ca08086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' class = multimedia subclass = HDA cap 01[50] = powerspec 3 supports D0 D3 current D0 cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message pcib1@pci0:0:28:0: class=0x060400 card=0x20578086 chip=0x9c908086 rev=0xe3 hdr=0x01 vendor = 'Intel Corporation' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 2 root port max data 128(128) link x0(x1) speed 0.0(5.0) ASPM L0s/L1(L0s/L1) cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x20578086 cap 01[a0] = powerspec 3 supports D0 D3 current D0 pcib2@pci0:0:28:3: class=0x060400 card=0x20578086 chip=0x9c968086 rev=0xe3 hdr=0x01 vendor = 'Intel Corporation' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 2 root port slot max data 128(128) link x1(x1) speed 2.5(5.0) ASPM L1(L0s/L1) cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x20578086 cap 01[a0] = powerspec 3 supports D0 D3 current D0 ecap 0000[100] = unknown 0 ecap 001e[200] = unknown 1 ehci0@pci0:0:29:0: class=0x0c0320 card=0x20578086 chip=0x9ca68086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' class = serial bus subclass = USB cap 01[50] = powerspec 3 supports D0 D3 current D0 cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14 cap 13[98] = PCI Advanced Features: FLR TP isab0@pci0:0:31:0: class=0x060100 card=0x20578086 chip=0x9cc38086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = PCI-ISA cap 09[e0] = vendor (length 12) Intel cap 1 version 0 features: AMT, 4 PCI-e x1 slots none1@pci0:0:31:3: class=0x0c0500 card=0x20578086 chip=0x9ca28086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' class = serial bus subclass = SMBus none2@pci0:2:0:0: class=0x028000 card=0x90108086 chip=0x095a8086 rev=0x59 hdr=0x00 vendor = 'Intel Corporation' class = network cap 01[c8] = powerspec 3 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[40] = PCI-Express 2 endpoint max data 128(128) FLR link x1(x1) speed 2.5(2.5) ASPM L1(L1) ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 0003[140] = Serial 1 3413e8ffff3e8230 ecap 0018[14c] = LTR 1 ecap 001e[154] = unknown 1 # dmesg (with workaround, means both drives reconnected physically in the right moment, marker with === ... ====) Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.2-RELEASE-p5 #1 r289059M: Sun Oct 11 08:19:20 CEST 2015 root@nas01.baar.tontaube.ch:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: Intel(R) Core(TM) i3-5010U CPU @ 2.10GHz (2095.19-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306d4 Family=0x6 Model=0x3d Stepping=4 Features=0xbfebfbff Features2=0x7ffafbbf,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800 AMD Features2=0x121 Structured Extended Features=0x21c27ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 8589934592 (8192 MB) avail memory = 8147660800 (7770 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 WARNING: Bogus Interrupt Trigger Mode. Assume CONFORMS. WARNING: Bogus Interrupt Trigger Mode. Assume CONFORMS. WARNING: Bogus Interrupt Polarity. Assume CONFORMS ioapic0 irqs 0-39 on motherboard random: initialized kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Event timer "HPET4" frequency 14318180 Hz quality 440 atrtc0: port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 vgapci0: port 0xf000-0xf03f mem 0xf6000000-0xf6ffffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 vgapci0: Boot video device hdac0: mem 0xf7134000-0xf7137fff irq 16 at device 3.0 on pci0 xhci0: mem 0xf7120000-0xf712ffff irq 21 at device 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA xhci0: Port routing mask set to 0xffffffff usbus0 on xhci0 pci0: at device 22.0 (no driver attached) em0: port 0xf060-0xf07f mem 0xf7100000-0xf711ffff,0xf713a000-0xf713afff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: b8:ae:ed:77:99:63 hdac1: mem 0xf7130000-0xf7133fff irq 22 at device 27.0 on pci0 pcib1: irq 16 at device 28.0 on pci0 pci1: on pcib1 pcib2: irq 19 at device 28.3 on pci0 pci2: on pcib2 pci2: at device 0.0 (no driver attached) ehci0: mem 0xf7139000-0xf71393ff irq 23 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 isab0: at device 31.0 on pci0 isa0: on isab0 acpi_button0: on acpi0 acpi_button1: on acpi0 acpi_tz0: on acpi0 acpi_tz1: on acpi0 orm0: at iomem 0xd0000-0xd0fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] fdc0: at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0 ppc0: cannot reserve I/O port range uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 est0: on cpu0 est1: on cpu1 est2: on cpu2 est3: on cpu3 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 3 on hdaa0 hdacc1: at cad 0 on hdac1 hdaa1: at nid 1 on hdacc1 pcm1: at nid 33 and 25 on hdaa1 random: unblocking device. usbus0: 5.0Gbps Super Speed USB v3.0 usbus1: 480Mbps High Speed USB v2.0 ugen1.1: at usbus1 uhub0: on usbus1 ugen0.1: <0x8086> at usbus0 uhub1: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC" frequency 2095194624 Hz quality 1000 ===== reconnecting the drives here for powering them on manually ====== Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 uhub0: 2 ports with 2 removable, self powered Root mount waiting for: usbus1 usbus0 ugen1.2: at usbus1 uhub2: on usbus1 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 Root mount waiting for: usbus1 usbus0 uhub2: 8 ports with 8 removable, self powered Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 uhub1: 15 ports with 15 removable, self powered ugen0.2: at usbus0 ukbd0: on usbus0 kbd2 at ukbd0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR Root mount waiting for: usbus0 usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR Root mount waiting for: usbus0 Root mount waiting for: usbus0 usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR Root mount waiting for: usbus0 usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR Root mount waiting for: usbus0 Root mount waiting for: usbus0 usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR ugen0.3: at usbus0 (disconnected) uhub_reattach_port: could not allocate new device Root mount waiting for: usbus0 Root mount waiting for: usbus0 ugen0.3: at usbus0 umass0: on usbus0 umass0: SCSI over Bulk-Only; quirks = 0x0100 umass0:0:0:-1: Attached to scbus0 da0 at umass-sim0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SPC-4 SCSI device da0: Serial Number NA820YR2 da0: 400.000MB/s transfers da0: 3815447MB (7814037167 512 byte sectors: 255H 63S/T 486401C) da0: quirks=0x2 Root mount waiting for: usbus0 ugen0.4: at usbus0 umass1: on usbus0 umass1: SCSI over Bulk-Only; quirks = 0x0100 umass1:1:1:-1: Attached to scbus1 Trying to mount root from zfs:systempool1 []... da1 at umass-sim1 bus 1 scbus1 target 0 lun 0 da1: Fixed Direct Access SPC-4 SCSI device da1: Serial Number NA821P7P da1: 400.000MB/s transfers da1: 3815447MB (7814037167 512 byte sectors: 255H 63S/T 486401C) da1: quirks=0x2 tun0: link state changed to UP