From owner-freebsd-fs@FreeBSD.ORG Thu Mar 21 15:53:11 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6D1E0AF9 for ; Thu, 21 Mar 2013 15:53:11 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-ob0-x22d.google.com (mail-ob0-x22d.google.com [IPv6:2607:f8b0:4003:c01::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 368333C9 for ; Thu, 21 Mar 2013 15:53:11 +0000 (UTC) Received: by mail-ob0-f173.google.com with SMTP id dn14so3026389obc.18 for ; Thu, 21 Mar 2013 08:53:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-received:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=DksBotVl9R+toSWnHBukgd2F+JiNCVes5cwVOxBcEnk=; b=ib9BUP6tn/tWwgvT3Gm8y/dWrmcBRvmlfFx9/AF3JSxJFLFyRc4Vc4JUBzemfGOLk7 qRRv865LzYQIBJcdwhbyvA3E1WueaDKEXT9uiIQMc57CtEoGC49gQsyUWs+nOboaGWOH mdApglCurCC0yi+ABe0U4Rhp1YiIiG4vpHYmZJ/CrOo7W1q8zeKSS7aospwDc3SIHxau wrm5vSARhFxNM/i3HyNWbt/asef+7sYxdFsTfbcT52amC9ZaRnkRBDAEs6SVOAoHqXqP AVffcwNDMHRjL0Yan7Pb/9jM11LIOjs5xOetLxcnRhQhW3ayYGVnCyTTfIM/GldRMk29 NIqQ== X-Received: by 10.60.3.233 with SMTP id f9mr7191425oef.32.1363881190567; Thu, 21 Mar 2013 08:53:10 -0700 (PDT) Received: from mail-ob0-x22b.google.com (mail-ob0-x22b.google.com [2607:f8b0:4003:c01::22b]) by mx.google.com with ESMTPS id 4sm7395326obj.7.2013.03.21.08.53.10 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 21 Mar 2013 08:53:10 -0700 (PDT) Received: by mail-ob0-f171.google.com with SMTP id x4so2993093obh.2 for ; Thu, 21 Mar 2013 08:53:09 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.60.11.8 with SMTP id m8mr7297224oeb.22.1363881189756; Thu, 21 Mar 2013 08:53:09 -0700 (PDT) Received: by 10.60.62.168 with HTTP; Thu, 21 Mar 2013 08:53:09 -0700 (PDT) Date: Thu, 21 Mar 2013 09:53:09 -0600 Message-ID: Subject: ZFS + NFS poor performance after restarting from 100 day uptime From: Josh Beard To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQlIIvyVxMt1RGcbLtF07en2LxEFhYz8Pa4EQBgXIyPOI1RCVcGPkU0T2HCtZeIVgqgNBYyF Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Mar 2013 15:53:11 -0000 Hello, I have a system with 12 disks spread between 2 raidz1. I'm using the native ("new") NFS to export a pool on this. This has worked very well all along, but since a reboot, has performed horribly - unusably under load. The system was running 9.1-rc3 and I upgraded it to 9.1-release-p1 (GENERIC kernel) after ~110 days of running (with zero performance issues). After rebooting from the upgrade, I'm finding the disks seem constantly slammed. gstat reports 90-100% busy most of the day with only ~100-130 ops/s. I didn't change any settings in /etc/sysctl.conf or /boot/loader. No ZFS tuning, etc. I've looked at the commits between 9.1-rc3 and 9.1-release-p1 and I can't see any reason why simply upgrading it would cause this. Since I had no issues at all with the same configuration before a reboot, I'm reluctant to start tweaking things too much, as I didn't have to before. As far as I can tell, the disks are fine. They're attached to a 3ware 9650 RAID card, configured to pass through the disks. The disks are WD5000AAKS-22YGA0. This system is used by many users for network home directories. I should also say - the RAID controller is scheduled to do media checks weekly. This usually takes ~10 hours. The weekend after the restart, it took ~28 hours, but came back fine. Does anyone have any recommendations as to where I might start looking? I feel like this must have something to do with the controller card, as the software changes were none that should have any impact on the performance and the mere act of restarting the system (shutdown -r, not a true power cycle) seemed to cause this. For trials, I /have/ tried some common tuning (but reverted). Such as changing the values of vfs.zfs.vdev.max_pending, vfs.zfs.write_limit_override=1073741824. Even for comparison, I set sync=disabled (not permanently) on that zpool. Performance improved, but nothing significant, and I certainly don't want to have to start tuning these if I didn't need to before. Thanks! --- /boot/loader.conf: hw.em.num_queues=1 hw.usb.no_pf=1 vfs.zfs.arc_max="13958643712" # 24 GB RAM coretemp_load="YES" aio_load="YES" loader_logo="beastie" autoboot_delay="5" Nothing in /etc/sysctl.conf Output of zpool status: pool: store state: ONLINE scan: scrub repaired 0 in 7h51m with 0 errors on Sun Mar 17 03:51:58 2013 config: NAME STATE READ WRITE CKSUM store ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 spares da13 AVAIL errors: No known data errors A snip of gstat: dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 0 0 0 0.0 0 0 0.0 0.0| cd0 0 1 0 0 0.0 1 32 0.2 0.0| da0 0 0 0 0 0.0 0 0 0.0 0.0| da0p1 0 1 0 0 0.0 1 32 0.2 0.0| da0p2 0 0 0 0 0.0 0 0 0.0 0.0| da0p3 4 160 126 1319 31.3 34 100 0.1 100.3| da1 4 146 110 1289 33.6 36 98 0.1 97.8| da2 4 142 107 1370 36.1 35 101 0.2 101.9| da3 4 121 95 1360 35.6 26 19 0.1 95.9| da4 4 151 117 1409 34.0 34 102 0.1 100.1| da5 4 141 109 1366 35.9 32 101 0.1 97.9| da6 4 136 118 1207 24.6 18 13 0.1 87.0| da7 4 118 102 1278 32.2 16 12 0.1 89.8| da8 4 138 116 1240 33.4 22 55 0.1 100.0| da9 4 133 117 1269 27.8 16 13 0.1 86.5| da10 4 121 102 1302 53.1 19 51 0.1 100.0| da11 4 120 99 1242 40.7 21 51 0.1 99.7| da12 dmesg (note the 100.000MB/s transfers. That seems low(?) However, looking at older logs, that's always been the case): Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.1-RELEASE-p1 #0 r247731: Sun Mar 3 15:11:33 MST 2013 root@topeka:/usr/obj/usr/src/sys/GENERIC amd64 CPU: Intel(R) Xeon(R) CPU E5506 @ 2.13GHz (2133.45-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a Stepping = 5 Features=0xbfebfbff Features2=0x9ce3bd AMD Features=0x28100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 25773998080 (24580 MB) avail memory = 24795668480 (23646 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <010511 APIC1122> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 cpu4 (AP): APIC ID: 16 cpu5 (AP): APIC ID: 18 cpu6 (AP): APIC ID: 20 cpu7 (AP): APIC ID: 22 ioapic0: Changing APIC ID to 1 ioapic1: Changing APIC ID to 3 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 ctl: CAM Target Layer loaded acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of 400, 100 (3) failed cpu0: on acpi0 ACPI Warning: Incorrect checksum in table [OEMB] - 0x90, should be 0x8D (20110527/tbutils-282) cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff iomem 0xfed40000-0xfed44fff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 3ware device driver for 9000 series storage controllers, version: 3.80.06.003 twa0: <3ware 9000 series Storage Controller> port 0xa800-0xa8ff mem 0xf6000000-0xf7ffffff,0xfb9de000-0xfb9defff irq 28 at device 0.0 on pci1 twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue: twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-16ML, 16 ports, Firmware FE9X 4.10.00.024, BIOS BE9X 4.08.00.004 pcib2: at device 3.0 on pci0 pci2: on pcib2 pcib3: at device 7.0 on pci0 pci3: on pcib3 pcib4: at device 9.0 on pci0 pci4: on pcib4 em0: port 0xbc00-0xbc1f mem 0xfba20000-0xfba3ffff,0xfba80000-0xfbafffff,0xfba1c000-0xfba1ffff irq 32 at device 0.0 on pci4 em0: Using MSIX interrupts with 3 vectors em0: Ethernet address: 00:1b:21:b8:cf:1b pci0: at device 20.0 (no driver attached) pci0: at device 20.1 (no driver attached) pci0: at device 20.2 (no driver attached) pci0: at device 20.3 (no driver attached) pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) pci0: at device 22.2 (no driver attached) pci0: at device 22.3 (no driver attached) pci0: at device 22.4 (no driver attached) pci0: at device 22.5 (no driver attached) pci0: at device 22.6 (no driver attached) pci0: at device 22.7 (no driver attached) uhci0: port 0x9c00-0x9c1f irq 16 at device 26.0 on pci0 uhci0: LegSup = 0x2f00 usbus0 on uhci0 uhci1: port 0x9880-0x989f irq 21 at device 26.1 on pci0 uhci1: LegSup = 0x2f00 usbus1 on uhci1 uhci2: port 0x9800-0x981f irq 19 at device 26.2 on pci0 uhci2: LegSup = 0x2f00 usbus2 on uhci2 ehci0: mem 0xfbeda000-0xfbeda3ff irq 18 at device 26.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 hdac0: mem 0xfbed4000-0xfbed7fff irq 22 at device 27.0 on pci0 pcib5: irq 17 at device 28.0 on pci0 pci5: on pcib5 em1: port 0xcc00-0xcc1f mem 0xfbb20000-0xfbb3ffff,0xfbb80000-0xfbbfffff,0xfbb1c000-0xfbb1ffff irq 16 at device 0.0 on pci5 em1: Using MSIX interrupts with 3 vectors em1: Ethernet address: 00:1b:21:b8:d0:1d pcib6: irq 17 at device 28.4 on pci0 pci6: on pcib6 em2: port 0xdc00-0xdc1f mem 0xfbce0000-0xfbcfffff,0xfbcdc000-0xfbcdffff irq 16 at device 0.0 on pci6 em2: Using MSIX interrupts with 3 vectors em2: Ethernet address: 00:25:90:32:c9:26 pcib7: irq 16 at device 28.5 on pci0 pci7: on pcib7 em3: port 0xec00-0xec1f mem 0xfbde0000-0xfbdfffff,0xfbddc000-0xfbddffff irq 17 at device 0.0 on pci7 em3: Using MSIX interrupts with 3 vectors em3: Ethernet address: 00:25:90:32:c9:27 uhci3: port 0x9480-0x949f irq 23 at device 29.0 on pci0 uhci3: LegSup = 0x2f00 usbus4 on uhci3 uhci4: port 0x9400-0x941f irq 19 at device 29.1 on pci0 uhci4: LegSup = 0x2f00 usbus5 on uhci4 uhci5: port 0x9080-0x909f irq 18 at device 29.2 on pci0 uhci5: LegSup = 0x2f00 usbus6 on uhci5 ehci1: mem 0xfbed8000-0xfbed83ff irq 23 at device 29.7 on pci0 usbus7: EHCI version 1.0 usbus7 on ehci1 pcib8: at device 30.0 on pci0 pci8: on pcib8 vgapci0: mem 0xf9000000-0xf9ffffff,0xfaffc000-0xfaffffff,0xfb000000-0xfb7fffff irq 18 at device 1.0 on pci8 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff90-0xff9f,0xffa0-0xffaf at device 31.2 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 qpi0: on motherboard pcib9: pcibus 255 on qpi0 pci255: on pcib9 pcib10: pcibus 254 on qpi0 pci254: on pcib10 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcbfff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range coretemp0: on cpu0 est0: on cpu0 p4tcc0: on cpu0 coretemp1: on cpu1 est1: on cpu1 p4tcc1: on cpu1 coretemp2: on cpu2 est2: on cpu2 p4tcc2: on cpu2 coretemp3: on cpu3 est3: on cpu3 p4tcc3: on cpu3 coretemp4: on cpu4 est4: on cpu4 p4tcc4: on cpu4 coretemp5: on cpu5 est5: on cpu5 p4tcc5: on cpu5 coretemp6: on cpu6 est6: on cpu6 p4tcc6: on cpu6 coretemp7: on cpu7 est7: on cpu7 p4tcc7: on cpu7 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered da0 at twa0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) cd0 at ata0 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed da1 at twa0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 100.000MB/s transfers da1: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da2 at twa0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da3 at twa0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SCSI-5 device da3: 100.000MB/s transfers da3: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da4 at twa0 bus 0 scbus0 target 4 lun 0 da4: Fixed Direct Access SCSI-5 device da4: 100.000MB/s transfers da4: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da5 at twa0 bus 0 scbus0 target 5 lun 0 da5: Fixed Direct Access SCSI-5 device da5: 100.000MB/s transfers da5: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da6 at twa0 bus 0 scbus0 target 6 lun 0 da6: Fixed Direct Access SCSI-5 device da6: 100.000MB/s transfers da6: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da7 at twa0 bus 0 scbus0 target 7 lun 0 da7: Fixed Direct Access SCSI-5 device da7: 100.000MB/s transfers da7: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da8 at twa0 bus 0 scbus0 target 8 lun 0 da8: Fixed Direct Access SCSI-5 device da8: 100.000MB/s transfers da8: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da9 at twa0 bus 0 scbus0 target 9 lun 0 da9: Fixed Direct Access SCSI-5 device da9: 100.000MB/s transfers da9: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da10 at twa0 bus 0 scbus0 target 10 lun 0 da10: Fixed Direct Access SCSI-5 device da10: 100.000MB/s transfers da10: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da11 at twa0 bus 0 scbus0 target 11 lun 0 da11: Fixed Direct Access SCSI-5 device da11: 100.000MB/s transfers da11: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da12 at twa0 bus 0 scbus0 target 12 lun 0 da12: Fixed Direct Access SCSI-5 device da12: 100.000MB/s transfers da12: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da13 at twa0 bus 0 scbus0 target 13 lun 0 da13: Fixed Direct Access SCSI-5 device da13: 100.000MB/s transfers da13: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da14 at twa0 bus 0 scbus0 target 14 lun 0 da14: Fixed Direct Access SCSI-5 device da14: 100.000MB/s transfers da14: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) SMP: AP CPU #2 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #1 Launched! da7: 100.000MB/s transfers da7: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da8 at twa0 bus 0 scbus0 target 8 lun 0 da8: Fixed Direct Access SCSI-5 device da8: 100.000MB/s transfers da8: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da9 at twa0 bus 0 scbus0 target 9 lun 0 da9: Fixed Direct Access SCSI-5 device da9: 100.000MB/s transfers da9: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da10 at twa0 bus 0 scbus0 target 10 lun 0 da10: Fixed Direct Access SCSI-5 device da10: 100.000MB/s transfers da10: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da11 at twa0 bus 0 scbus0 target 11 lun 0 da11: Fixed Direct Access SCSI-5 device da11: 100.000MB/s transfers da11: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da12 at twa0 bus 0 scbus0 target 12 lun 0 da12: Fixed Direct Access SCSI-5 device da12: 100.000MB/s transfers da12: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da13 at twa0 bus 0 scbus0 target 13 lun 0 da13: Fixed Direct Access SCSI-5 device da13: 100.000MB/s transfers da13: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) da14 at twa0 bus 0 scbus0 target 14 lun 0 da14: Fixed Direct Access SCSI-5 device da14: 100.000MB/s transfers da14: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C) SMP: AP CPU #2 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #5 Launched! Timecounter "TSC-low" frequency 16667583 Hz quality 1000 uhub3: 6 ports with 6 removable, self powered uhub7: 6 ports with 6 removable, self powered Trying to mount root from ufs:/dev/da0p2 [rw]... ugen4.2: at usbus4 ukbd0: on usbus4 kbd2 at ukbd0 ums0: on usbus4 ums0: 5 buttons and [XYZ] coordinates ID=1 ZFS filesystem version 5 ZFS storage pool version 28 em0: link state changed to UP lagg0: link state changed to UP em2: link state changed to UP em1: link state changed to UP em3: link state changed to UP