Date: Thu, 21 Mar 2013 09:53:09 -0600 From: Josh Beard <josh@signalboxes.net> To: freebsd-fs@freebsd.org Subject: ZFS + NFS poor performance after restarting from 100 day uptime Message-ID: <CAHDrHSsCunt9eQKjMy9epPBYTmaGs5HNgKV2%2BUKuW0RQZPpw%2BA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello,
I have a system with 12 disks spread between 2 raidz1. I'm using the
native ("new") NFS to export a pool on this. This has worked very well all
along, but since a reboot, has performed horribly - unusably under load.
The system was running 9.1-rc3 and I upgraded it to 9.1-release-p1 (GENERIC
kernel) after ~110 days of running (with zero performance issues). After
rebooting from the upgrade, I'm finding the disks seem constantly slammed.
gstat reports 90-100% busy most of the day with only ~100-130 ops/s.
I didn't change any settings in /etc/sysctl.conf or /boot/loader. No ZFS
tuning, etc. I've looked at the commits between 9.1-rc3 and 9.1-release-p1
and I can't see any reason why simply upgrading it would cause this.
Since I had no issues at all with the same configuration before a reboot,
I'm reluctant to start tweaking things too much, as I didn't have to
before. As far as I can tell, the disks are fine. They're attached to a
3ware 9650 RAID card, configured to pass through the disks. The disks
are WD5000AAKS-22YGA0.
This system is used by many users for network home directories.
I should also say - the RAID controller is scheduled to do media checks
weekly. This usually takes ~10 hours. The weekend after the restart, it
took ~28 hours, but came back fine.
Does anyone have any recommendations as to where I might start looking? I
feel like this must have something to do with the controller card, as the
software changes were none that should have any impact on the performance
and the mere act of restarting the system (shutdown -r, not a true power
cycle) seemed to cause this.
For trials, I /have/ tried some common tuning (but reverted). Such as
changing the values of
vfs.zfs.vdev.max_pending, vfs.zfs.write_limit_override=1073741824. Even
for comparison, I set sync=disabled (not permanently) on that zpool.
Performance improved, but nothing significant, and I certainly don't want
to have to start tuning these if I didn't need to before.
Thanks!
---
/boot/loader.conf:
hw.em.num_queues=1
hw.usb.no_pf=1
vfs.zfs.arc_max="13958643712" # 24 GB RAM
coretemp_load="YES"
aio_load="YES"
loader_logo="beastie"
autoboot_delay="5"
Nothing in /etc/sysctl.conf
Output of zpool status:
pool: store
state: ONLINE
scan: scrub repaired 0 in 7h51m with 0 errors on Sun Mar 17 03:51:58 2013
config:
NAME STATE READ WRITE CKSUM
store ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
spares
da13 AVAIL
errors: No known data errors
A snip of gstat:
dT: 1.002s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 0 0 0 0.0 0 0 0.0 0.0| cd0
0 1 0 0 0.0 1 32 0.2 0.0| da0
0 0 0 0 0.0 0 0 0.0 0.0| da0p1
0 1 0 0 0.0 1 32 0.2 0.0| da0p2
0 0 0 0 0.0 0 0 0.0 0.0| da0p3
4 160 126 1319 31.3 34 100 0.1 100.3| da1
4 146 110 1289 33.6 36 98 0.1 97.8| da2
4 142 107 1370 36.1 35 101 0.2 101.9| da3
4 121 95 1360 35.6 26 19 0.1 95.9| da4
4 151 117 1409 34.0 34 102 0.1 100.1| da5
4 141 109 1366 35.9 32 101 0.1 97.9| da6
4 136 118 1207 24.6 18 13 0.1 87.0| da7
4 118 102 1278 32.2 16 12 0.1 89.8| da8
4 138 116 1240 33.4 22 55 0.1 100.0| da9
4 133 117 1269 27.8 16 13 0.1 86.5| da10
4 121 102 1302 53.1 19 51 0.1 100.0| da11
4 120 99 1242 40.7 21 51 0.1 99.7| da12
dmesg (note the 100.000MB/s transfers. That seems low(?) However, looking
at older logs, that's always been the case):
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-RELEASE-p1 #0 r247731: Sun Mar 3 15:11:33 MST 2013
root@topeka:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU E5506 @ 2.13GHz (2133.45-MHz K8-class
CPU)
Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a Stepping =
5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 25773998080 (24580 MB)
avail memory = 24795668480 (23646 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <010511 APIC1122>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 6
cpu4 (AP): APIC ID: 16
cpu5 (AP): APIC ID: 18
cpu6 (AP): APIC ID: 20
cpu7 (AP): APIC ID: 22
ioapic0: Changing APIC ID to 1
ioapic1: Changing APIC ID to 3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ctl: CAM Target Layer loaded
acpi0: <SMCI > on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 400, 100 (3) failed
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0x90, should be 0x8D
(20110527/tbutils-282)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff iomem 0xfed40000-0xfed44fff
on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
3ware device driver for 9000 series storage controllers, version:
3.80.06.003
twa0: <3ware 9000 series Storage Controller> port 0xa800-0xa8ff mem
0xf6000000-0xf7ffffff,0xfb9de000-0xfb9defff irq 28 at device 0.0 on pci1
twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue:
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-16ML, 16
ports, Firmware FE9X 4.10.00.024, BIOS BE9X 4.08.00.004
pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci4: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xbc00-0xbc1f mem
0xfba20000-0xfba3ffff,0xfba80000-0xfbafffff,0xfba1c000-0xfba1ffff irq 32 at
device 0.0 on pci4
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: 00:1b:21:b8:cf:1b
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no driver
attached)
pci0: <base peripheral> at device 22.0 (no driver attached)
pci0: <base peripheral> at device 22.1 (no driver attached)
pci0: <base peripheral> at device 22.2 (no driver attached)
pci0: <base peripheral> at device 22.3 (no driver attached)
pci0: <base peripheral> at device 22.4 (no driver attached)
pci0: <base peripheral> at device 22.5 (no driver attached)
pci0: <base peripheral> at device 22.6 (no driver attached)
pci0: <base peripheral> at device 22.7 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0x9c00-0x9c1f irq
16 at device 26.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0x9880-0x989f irq
21 at device 26.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0x9800-0x981f irq
19 at device 26.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem
0xfbeda000-0xfbeda3ff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
hdac0: <Intel 82801JI HDA Controller> mem 0xfbed4000-0xfbed7fff irq 22 at
device 27.0 on pci0
pcib5: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci5: <ACPI PCI bus> on pcib5
em1: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xcc00-0xcc1f mem
0xfbb20000-0xfbb3ffff,0xfbb80000-0xfbbfffff,0xfbb1c000-0xfbb1ffff irq 16 at
device 0.0 on pci5
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 00:1b:21:b8:d0:1d
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci6: <ACPI PCI bus> on pcib6
em2: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xdc00-0xdc1f mem
0xfbce0000-0xfbcfffff,0xfbcdc000-0xfbcdffff irq 16 at device 0.0 on pci6
em2: Using MSIX interrupts with 3 vectors
em2: Ethernet address: 00:25:90:32:c9:26
pcib7: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci7: <ACPI PCI bus> on pcib7
em3: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xec00-0xec1f mem
0xfbde0000-0xfbdfffff,0xfbddc000-0xfbddffff irq 17 at device 0.0 on pci7
em3: Using MSIX interrupts with 3 vectors
em3: Ethernet address: 00:25:90:32:c9:27
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0x9480-0x949f irq
23 at device 29.0 on pci0
uhci3: LegSup = 0x2f00
usbus4 on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0x9400-0x941f irq
19 at device 29.1 on pci0
uhci4: LegSup = 0x2f00
usbus5 on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0x9080-0x909f irq
18 at device 29.2 on pci0
uhci5: LegSup = 0x2f00
usbus6 on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem
0xfbed8000-0xfbed83ff irq 23 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
pcib8: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> mem
0xf9000000-0xf9ffffff,0xfaffc000-0xfaffffff,0xfb000000-0xfb7fffff irq 18 at
device 1.0 on pci8
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH10 SATA300 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff90-0xff9f,0xffa0-0xffaf at device
31.2 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
qpi0: <QPI system bus> on motherboard
pcib9: <QPI Host-PCI bridge> pcibus 255 on qpi0
pci255: <PCI bus> on pcib9
pcib10: <QPI Host-PCI bridge> pcibus 254 on qpi0
pci254: <PCI bus> on pcib10
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcbfff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
da0 at twa0 bus 0 scbus0 target 0 lun 0
da0: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
cd0 at ata0 bus 0 scbus1 target 0 lun 0
cd0: <HL-DT-ST DVDRAM GT40N 1.00> Removable CD-ROM SCSI-0 device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present -
tray closed
da1 at twa0 bus 0 scbus0 target 1 lun 0
da1: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da1: 100.000MB/s transfers
da1: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da2 at twa0 bus 0 scbus0 target 2 lun 0
da2: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da2: 100.000MB/s transfers
da2: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da3 at twa0 bus 0 scbus0 target 3 lun 0
da3: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da3: 100.000MB/s transfers
da3: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da4 at twa0 bus 0 scbus0 target 4 lun 0
da4: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da4: 100.000MB/s transfers
da4: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da5 at twa0 bus 0 scbus0 target 5 lun 0
da5: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da5: 100.000MB/s transfers
da5: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da6 at twa0 bus 0 scbus0 target 6 lun 0
da6: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da6: 100.000MB/s transfers
da6: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da7 at twa0 bus 0 scbus0 target 7 lun 0
da7: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da7: 100.000MB/s transfers
da7: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da8 at twa0 bus 0 scbus0 target 8 lun 0
da8: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da8: 100.000MB/s transfers
da8: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da9 at twa0 bus 0 scbus0 target 9 lun 0
da9: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da9: 100.000MB/s transfers
da9: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da10 at twa0 bus 0 scbus0 target 10 lun 0
da10: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da10: 100.000MB/s transfers
da10: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da11 at twa0 bus 0 scbus0 target 11 lun 0
da11: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da11: 100.000MB/s transfers
da11: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da12 at twa0 bus 0 scbus0 target 12 lun 0
da12: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da12: 100.000MB/s transfers
da12: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da13 at twa0 bus 0 scbus0 target 13 lun 0
da13: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da13: 100.000MB/s transfers
da13: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da14 at twa0 bus 0 scbus0 target 14 lun 0
da14: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da14: 100.000MB/s transfers
da14: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #1 Launched!
da7: 100.000MB/s transfers
da7: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da8 at twa0 bus 0 scbus0 target 8 lun 0
da8: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da8: 100.000MB/s transfers
da8: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da9 at twa0 bus 0 scbus0 target 9 lun 0
da9: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da9: 100.000MB/s transfers
da9: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da10 at twa0 bus 0 scbus0 target 10 lun 0
da10: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da10: 100.000MB/s transfers
da10: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da11 at twa0 bus 0 scbus0 target 11 lun 0
da11: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da11: 100.000MB/s transfers
da11: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da12 at twa0 bus 0 scbus0 target 12 lun 0
da12: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da12: 100.000MB/s transfers
da12: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da13 at twa0 bus 0 scbus0 target 13 lun 0
da13: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da13: 100.000MB/s transfers
da13: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
da14 at twa0 bus 0 scbus0 target 14 lun 0
da14: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device
da14: 100.000MB/s transfers
da14: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
Timecounter "TSC-low" frequency 16667583 Hz quality 1000
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
Trying to mount root from ufs:/dev/da0p2 [rw]...
ugen4.2: <vendor 0x0d3d> at usbus4
ukbd0: <EP1> on usbus4
kbd2 at ukbd0
ums0: <EP2> on usbus4
ums0: 5 buttons and [XYZ] coordinates ID=1
ZFS filesystem version 5
ZFS storage pool version 28
em0: link state changed to UP
lagg0: link state changed to UP
em2: link state changed to UP
em1: link state changed to UP
em3: link state changed to UP
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHDrHSsCunt9eQKjMy9epPBYTmaGs5HNgKV2%2BUKuW0RQZPpw%2BA>