Date: Thu, 4 Jul 2013 19:36:27 +0200 From: mxb <mxb@alumni.chalmers.se> To: Jeremy Chadwick <jdc@koitsu.org> Cc: freebsd-fs@freebsd.org Subject: Re: Slow resilvering with mirrored ZIL Message-ID: <2A261BEA-4452-4F6A-8EFB-90A54D79CBB9@alumni.chalmers.se> In-Reply-To: <20130704171637.GA94539@icarus.home.lan> References: <CABBFC07-68C2-4F43-9AFC-920D8C34282E@unixconn.com> <51D42107.1050107@digsys.bg> <2EF46A8C-6908-4160-BF99-EC610B3EA771@alumni.chalmers.se> <51D437E2.4060101@digsys.bg> <E5CCC8F551CA4627A3C7376AD63A83CC@multiplay.co.uk> <CBCA1716-A3EC-4E3B-AE0A-3C8028F6AACF@alumni.chalmers.se> <20130704000405.GA75529@icarus.home.lan> <C8C696C0-2963-4868-8BB8-6987B47C3460@alumni.chalmers.se> <20130704171637.GA94539@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
zdb -C:
NAS:
version: 28
name: 'NAS'
state: 0
txg: 19039918
pool_guid: 3808946822857359331
hostid: 516334119
hostname: 'nas.home.unixconn.com'
vdev_children: 2
vdev_tree:
type: 'root'
id: 0
guid: 3808946822857359331
children[0]:
type: 'raidz'
id: 0
guid: 15126043265564363201
nparity: 1
metaslab_array: 14
metaslab_shift: 32
ashift: 9
asize: 4000799784960
is_log: 0
children[0]:
type: 'replacing'
id: 0
guid: 15772352161192928927
whole_disk: 0
children[0]:
type: 'disk'
id: 0
guid: 1160414651350745057
path: '/dev/ada0/old'
phys_path: '/dev/ada0'
whole_disk: 0
DTL: 788
children[1]:
type: 'disk'
id: 1
guid: 16857066297968094282
path: '/dev/ada0'
phys_path: '/dev/ada0'
whole_disk: 1
DTL: 4100
resilvering: 1
children[1]:
type: 'disk'
id: 1
guid: 8602024497074155869
path: '/dev/ada3'
phys_path: '/dev/ada3'
whole_disk: 1
DTL: 4097
children[2]:
type: 'disk'
id: 2
guid: 15590206669698985709
path: '/dev/ada1'
phys_path: '/dev/ada1'
whole_disk: 0
DTL: 786
children[3]:
type: 'disk'
id: 3
guid: 14741760026602900071
path: '/dev/ada2'
phys_path: '/dev/ada2'
whole_disk: 0
DTL: 785
children[1]:
type: 'mirror'
id: 1
guid: 6427576469721219307
metaslab_array: 4098
metaslab_shift: 26
ashift: 9
asize: 10732437504
is_log: 1
create_txg: 19039876
children[0]:
type: 'disk'
id: 0
guid: 14073296989893822073
path: '/dev/ada4s1'
phys_path: '/dev/ada4s1'
whole_disk: 1
create_txg: 19039876
children[1]:
type: 'disk'
id: 1
guid: 8631947194818711903
path: '/dev/ada5s1'
phys_path: '/dev/ada5s1'
whole_disk: 1
create_txg: 19039876
smartctl:
nas# smartctl -a ada3
smartctl 6.0 2012-10-10 r3643 [FreeBSD 9.1-RELEASE-p1 amd64] (local =
build)
Copyright (C) 2002-12, Bruce Allen, Christian Franke, =
www.smartmontools.org
ada3: Unable to detect device type
Please specify device type with the -d option.
Use smartctl -h to get a usage summary
THIS IS FOR ALL DISKS I HAVE INSIDE.
On 4 jul 2013, at 19:16, Jeremy Chadwick <jdc@koitsu.org> wrote:
> On Thu, Jul 04, 2013 at 06:57:09PM +0200, mxb wrote:
>>=20
>> Well, I'v got a lot of errors in dmesg regarding one of four disks, =
which is not currently replaced.
>> After resilvering dropped to 2MB/s ( :O ), it stuck. I had to reboot =
system.
>> It came up and now continues resilvering process at 9/s (9 what??? =
bytes? )
>>=20
>> Here comes requested info:
>>=20
>> dmesg(plan was to re-new disks and then lift this sys up to STABLE):
>>=20
>> Copyright (c) 1992-2012 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, =
1994
>> The Regents of the University of California. All rights =
reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 9.1-RELEASE-p1 #2: Fri Mar 8 10:30:38 CET 2013
>> root@nas.home.unixconn.com:/usr/obj/usr/src/sys/GENERIC amd64
>> module_register: module pci/em already exists!
>> Module pci/em failed to register: 17
>> module_register: module pci/lem already exists!
>> Module pci/lem failed to register: 17
>> CPU: Intel(R) Xeon(R) CPU X5460 @ 3.16GHz (3166.74-MHz =
K8-class CPU)
>> Origin =3D "GenuineIntel" Id =3D 0x10676 Family =3D 6 Model =3D =
17 Stepping =3D 6
>> =
Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE=
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>> =
Features2=3D0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDC=
M,DCA,SSE4.1>
>> AMD Features=3D0x20100800<SYSCALL,NX,LM>
>> AMD Features2=3D0x1<LAHF>
>> TSC: P-state invariant, performance statistics
>> real memory =3D 34359738368 (32768 MB)
>> avail memory =3D 33095909376 (31562 MB)
>> Event timer "LAPIC" quality 400
>> ACPI APIC Table: <PTLTD APIC >
>> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
>> FreeBSD/SMP: 1 package(s) x 4 core(s)
>> cpu0 (BSP): APIC ID: 0
>> cpu1 (AP): APIC ID: 1
>> cpu2 (AP): APIC ID: 2
>> cpu3 (AP): APIC ID: 3
>> ioapic0 <Version 2.0> irqs 0-23 on motherboard
>> ioapic1 <Version 2.0> irqs 24-47 on motherboard
>> kbd1 at kbdmux0
>> ctl: CAM Target Layer loaded
>> acpi0: <PTLTD RSDT> on motherboard
>> acpi0: Power Button (fixed)
>> cpu0: <ACPI CPU> on acpi0
>> cpu1: <ACPI CPU> on acpi0
>> cpu2: <ACPI CPU> on acpi0
>> cpu3: <ACPI CPU> on acpi0
>> atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
>> Event timer "RTC" frequency 32768 Hz quality 0
>> attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
>> Timecounter "i8254" frequency 1193182 Hz quality 0
>> Event timer "i8254" frequency 1193182 Hz quality 100
>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on =
acpi0
>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>> pci0: <ACPI PCI bus> on pcib0
>> pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
>> pci1: <ACPI PCI bus> on pcib1
>> pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
>> pci2: <ACPI PCI bus> on pcib2
>> pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
>> pci3: <ACPI PCI bus> on pcib3
>> pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
>> pci4: <ACPI PCI bus> on pcib4
>> pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
>> pci5: <ACPI PCI bus> on pcib5
>> pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
>> pci6: <ACPI PCI bus> on pcib6
>> em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0x2000-0x201f =
mem 0xdc000000-0xdc01ffff irq 18 at device 0.0 on pci6
>> em0: Using an MSI interrupt
>> em0: Ethernet address: 00:30:48:34:46:36
>> em1: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0x2020-0x203f =
mem 0xdc020000-0xdc03ffff irq 19 at device 0.1 on pci6
>> em1: Using an MSI interrupt
>> em1: Ethernet address: 00:30:48:34:46:37
>> pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
>> pci7: <ACPI PCI bus> on pcib7
>> pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
>> pci8: <ACPI PCI bus> on pcib8
>> pcib9: <PCI-PCI bridge> at device 0.0 on pci8
>> pci9: <PCI bus> on pcib9
>> bce0: <HP NC380T PCIe DP Multifunc Gig Server Adapter (A2)> mem =
0xd8000000-0xd9ffffff irq 16 at device 4.0 on pci9
>> miibus0: <MII bus> on bce0
>> brgphy0: <BCM5706 1000BASE-T/SX media interface> PHY 1 on miibus0
>> brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, =
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
>> bce0: Ethernet address: 00:1b:78:38:2f:28
>> bce0: ASIC (0x57060020); Rev (A2); Bus (PCI-X, 64-bit, 100MHz); B/C =
(1.9.6); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
>> Coal (RX:6,6,18,18; TX:20,20,80,80)
>> pcib10: <PCI-PCI bridge> at device 0.2 on pci8
>> pci10: <PCI bus> on pcib10
>> bce1: <HP NC380T PCIe DP Multifunc Gig Server Adapter (A2)> mem =
0xda000000-0xdbffffff irq 17 at device 5.0 on pci10
>> miibus1: <MII bus> on bce1
>> brgphy1: <BCM5706 1000BASE-T/SX media interface> PHY 1 on miibus1
>> brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, =
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
>> bce1: Ethernet address: 00:1b:78:38:2f:2a
>> bce1: ASIC (0x57060020); Rev (A2); Bus (PCI-X, 64-bit, 100MHz); B/C =
(1.9.6); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI)
>> Coal (RX:6,6,18,18; TX:20,20,80,80)
>> pcib11: <ACPI PCI-PCI bridge> at device 6.0 on pci0
>> pci11: <ACPI PCI bus> on pcib11
>> pci0: <base peripheral> at device 8.0 (no driver attached)
>> pcib12: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
>> pci12: <ACPI PCI bus> on pcib12
>> uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port =
0x1800-0x181f irq 17 at device 29.0 on pci0
>> usbus0 on uhci0
>> uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port =
0x1820-0x183f irq 19 at device 29.1 on pci0
>> usbus1 on uhci1
>> uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port =
0x1840-0x185f irq 18 at device 29.2 on pci0
>> usbus2 on uhci2
>> ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xdc500000-0xdc5003ff =
irq 17 at device 29.7 on pci0
>> usbus3: EHCI version 1.0
>> usbus3 on ehci0
>> pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
>> pci13: <ACPI PCI bus> on pcib13
>> vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem =
0xd0000000-0xd7ffffff,0xdc200000-0xdc20ffff irq 18 at device 1.0 on =
pci13
>> isab0: <PCI-ISA bridge> at device 31.0 on pci0
>> isa0: <ISA bus> on isab0
>> ahci0: <Intel ESB2 AHCI SATA controller> port =
0x1890-0x1897,0x1884-0x1887,0x1888-0x188f,0x1880-0x1883,0x1860-0x187f =
mem 0xdc500400-0xdc5007ff irq 19 at device 31.2 on pci0
>> ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
>> ahcich0: <AHCI channel> at channel 0 on ahci0
>> ahcich1: <AHCI channel> at channel 1 on ahci0
>> ahcich2: <AHCI channel> at channel 2 on ahci0
>> ahcich3: <AHCI channel> at channel 3 on ahci0
>> ahcich4: <AHCI channel> at channel 4 on ahci0
>> ahcich5: <AHCI channel> at channel 5 on ahci0
>> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
>> acpi_button0: <Power Button> on acpi0
>> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
>> atkbd0: <AT Keyboard> irq 1 on atkbdc0
>> kbd0 at atkbd0
>> atkbd0: [GIANT-LOCKED]
>> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on =
acpi0
>> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
>> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on =
acpi0
>> fdc0: does not respond
>> device_attach: fdc0 attach returned 6
>> ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on =
acpi0
>> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
>> ppc0: FIFO with 16/16/9 bytes threshold
>> ppbus0: <Parallel port bus> on ppc0
>> plip0: <PLIP network interface> on ppbus0
>> lpt0: <Printer> on ppbus0
>> lpt0: Interrupt-driven port
>> ppi0: <Parallel I/O> on ppbus0
>> orm0: <ISA Option ROMs> at iomem =
0xc0000-0xcafff,0xcb000-0xccfff,0xcd000-0xce7ff,0xce800-0xcffff on isa0
>> sc0: <System console> at flags 0x100 on isa0
>> sc0: VGA <16 virtual consoles, flags=3D0x300>
>> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on =
isa0
>> fdc0: No FDOUT register!
>> est0: <Enhanced SpeedStep Frequency Control> on cpu0
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 4921492106004921
>> device_attach: est0 attach returned 6
>> p4tcc0: <CPU Frequency Thermal Control> on cpu0
>> est1: <Enhanced SpeedStep Frequency Control> on cpu1
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 4921492106004921
>> device_attach: est1 attach returned 6
>> p4tcc1: <CPU Frequency Thermal Control> on cpu1
>> est2: <Enhanced SpeedStep Frequency Control> on cpu2
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 4921492106004921
>> device_attach: est2 attach returned 6
>> p4tcc2: <CPU Frequency Thermal Control> on cpu2
>> est3: <Enhanced SpeedStep Frequency Control> on cpu3
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 4921492106004921
>> device_attach: est3 attach returned 6
>> p4tcc3: <CPU Frequency Thermal Control> on cpu3
>> ZFS filesystem version 5
>> ZFS storage pool version 28
>> Timecounters tick every 1.000 msec
>> usbus0: 12Mbps Full Speed USB v1.0
>> usbus1: 12Mbps Full Speed USB v1.0
>> usbus2: 12Mbps Full Speed USB v1.0
>> usbus3: 480Mbps High Speed USB v2.0
>> ugen0.1: <Intel> at usbus0
>> uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =
usbus0
>> ugen1.1: <Intel> at usbus1
>> uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =
usbus1
>> ugen2.1: <Intel> at usbus2
>> uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =
usbus2
>> ugen3.1: <Intel> at usbus3
>> uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on =
usbus3
>> uhub0: 2 ports with 2 removable, self powered
>> uhub1: 2 ports with 2 removable, self powered
>> uhub2: 2 ports with 2 removable, self powered
>> bce0: bce_pulse(): Warning: bootcode thinks driver is absent! =
(bc_state =3D 0x00000004)
>> bce1: bce_pulse(): Warning: bootcode thinks driver is absent! =
(bc_state =3D 0x00000004)
>> uhub3: 6 ports with 6 removable, self powered
>> ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
>> ada0: <ST32000645NS 0004> ATA-8 SATA 3.x device
>> ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
>> ada0: Command Queueing enabled
>> ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
>> ada0: Previously was known as ad4
>> ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
>> ada1: <WDC WD10EARS-00Z5B1 80.00A80> ATA-8 SATA 2.x device
>> ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
>> ada1: Command Queueing enabled
>> ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
>> ada1: Previously was known as ad6
>> ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
>> ada2: <WDC WD10EARS-00Z5B1 80.00A80> ATA-8 SATA 2.x device
>> ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
>> ada2: Command Queueing enabled
>> ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
>> ada2: Previously was known as ad8
>> ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
>> ada3: <ST32000645NS 0004> ATA-8 SATA 3.x device
>> ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
>> ada3: Command Queueing enabled
>> ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
>> ada3: Previously was known as ad10
>> ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
>> ada4: <INTEL SSDSA2VP020G2 2CV102M5> ATA-7 SATA 2.x device
>> ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
>> ada4: Command Queueing enabled
>> ada4: 19087MB (39091248 512 byte sectors: 16H 63S/T 16383C)
>> ada4: Previously was known as ad12
>> ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
>> ada5: <OCZ-AGILITY3 2.15> ATA-8 SATA 3.x device
>> ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
>> ada5: Command Queueing enabled
>> ada5: 57241MB (117231408 512 byte sectors: 16H 63S/T 16383C)
>> ada5: Previously was known as ad14
>> SMP: AP CPU #3 Launched!
>> SMP: AP CPU #1 Launched!
>> SMP: AP CPU #2 Launched!
>> Timecounter "TSC-low" frequency 12370070 Hz quality 1000
>> Root mount waiting for: usbus3
>> ugen0.2: <Hewlett-Packard Company> at usbus0
>> ukbd0: <Hewlett-Packard Company HP USB Smart Card Keyboard, class =
0/0, rev 2.00/5.35, addr 2> on usbus0
>> kbd2 at ukbd0
>> Root mount waiting for: usbus3
>> ugen3.2: <Generic> at usbus3
>> umass0: <Generic USB Storage, class 0/0, rev 2.00/2.50, addr 2> on =
usbus3
>> umass0: SCSI over Bulk-Only; quirks =3D 0x4100
>> umass0:7:0:-1: Attached to scbus7
>> Trying to mount root from zfs:NAS/system/root [rw]...
>> da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
>> da0: <Generic STORAGE DEVICE 0250> Removable Direct Access SCSI-0 =
device
>> da0: 40.000MB/s transfers
>> da0: 7580MB (15523840 512 byte sectors: 255H 63S/T 966C)
>> GEOM_PART: integrity check failed (label/zfs_boot, BSD)
>> GEOM_PART: integrity check failed (label/zfs_boot, BSD)
>> bce0: Gigabit link up!
>> bce1: Gigabit link up!
>>=20
>> zpool status:
>> pool: NAS
>> state: DEGRADED
>> status: One or more devices is currently being resilvered. The pool =
will
>> continue to function, possibly in a degraded state.
>> action: Wait for the resilver to complete.
>> scan: resilver in progress since Tue Jul 2 07:31:29 2013
>> 483G scanned out of 3.49T at 10/s, (scan is slow, no estimated =
time)
>> 121G resilvered, 13.52% done
>> config:
>>=20
>> NAME STATE READ WRITE CKSUM
>> NAS DEGRADED 0 0 0
>> raidz1-0 DEGRADED 0 0 0
>> replacing-0 DEGRADED 0 0 0
>> 1160414651350745057 UNAVAIL 0 0 0 was =
/dev/ada0/old
>> ada0 ONLINE 0 0 0 =
(resilvering)
>> ada3 ONLINE 0 0 0
>> ada1 ONLINE 0 0 0
>> ada2 ONLINE 0 0 0
>> logs
>> mirror-1 ONLINE 0 0 0
>> ada4s1 ONLINE 0 0 0
>> ada5s1 ONLINE 0 0 0
>> cache
>> ada5s2 ONLINE 0 0 0
>>=20
>> errors: No known data errors
>>=20
>> zpool get all:
>> NAME PROPERTY VALUE SOURCE
>> NAS size 3.64T -
>> NAS capacity 95% -
>> NAS altroot - default
>> NAS health DEGRADED -
>> NAS guid 3808946822857359331 default
>> NAS version 28 default
>> NAS bootfs - default
>> NAS delegation on default
>> NAS autoreplace off default
>> NAS cachefile - default
>> NAS failmode wait default
>> NAS listsnapshots off default
>> NAS autoexpand off default
>> NAS dedupditto 0 default
>> NAS dedupratio 1.00x -
>> NAS free 154G -
>> NAS allocated 3.49T -
>> NAS readonly off -
>> NAS comment - default
>> NAS expandsize 0 -
>>=20
>> zfs get all:
>> NAME PROPERTY VALUE =
SOURCE
>> NAS type filesystem -
>> NAS creation Tue Aug 3 20:10 2010 -
>> NAS used 2.61T -
>> NAS available 72.0G -
>> NAS referenced 62.8K -
>> NAS compressratio 1.00x -
>> NAS mounted yes -
>> NAS quota none =
default
>> NAS reservation none =
default
>> NAS recordsize 128K =
default
>> NAS mountpoint /mnt/NAS =
local
>> NAS sharenfs off =
default
>> NAS checksum on =
default
>> NAS compression off =
default
>> NAS atime off =
local
>> NAS devices on =
default
>> NAS exec on =
default
>> NAS setuid on =
default
>> NAS readonly off =
default
>> NAS jailed off =
default
>> NAS snapdir hidden =
default
>> NAS aclmode discard =
default
>> NAS aclinherit restricted =
default
>> NAS canmount on =
default
>> NAS xattr off =
temporary
>> NAS copies 1 =
default
>> NAS version 5 -
>> NAS utf8only off -
>> NAS normalization none -
>> NAS casesensitivity sensitive -
>> NAS vscan off =
default
>> NAS nbmand off =
default
>> NAS sharesmb off =
default
>> NAS refquota none =
default
>> NAS refreservation none =
default
>> NAS primarycache all =
default
>> NAS secondarycache all =
default
>> NAS usedbysnapshots 0 -
>> NAS usedbydataset 0 -
>> NAS usedbychildren 0 -
>> NAS usedbyrefreservation 0 -
>> NAS logbias latency =
default
>> NAS dedup off =
default
>> NAS mlslabel -
>> NAS sync standard =
local
>> NAS refcompressratio 1.00x -
>> NAS written 62.8K -
>>=20
>> the rest of created - defaults. no changes or inherited from this =
one.
>>=20
>> part show -p:
>> nas# gpart show -p ada0
>> gpart: No such geom: ada0.
>> nas# gpart show -p ada1
>> gpart: No such geom: ada1.
>> nas# gpart show -p ada2
>> gpart: No such geom: ada2.
>> nas# gpart show -p ada3
>> gpart: No such geom: ada3.
>> nas# gpart show -p ada4
>> =3D> 63 39091185 ada4 MBR (18G)
>> 63 20971503 ada4s1 freebsd (10G)
>> 20971566 18119682 ada4s2 freebsd (8.7G)
>>=20
>> nas# gpart show -p ada5
>> =3D> 63 117231345 ada5 MBR (55G)
>> 63 20971503 ada5s1 freebsd (10G)
>> 20971566 96259842 ada5s2 freebsd (45G)
>>=20
>> ada4/ada5 are ZIL/L2ARC/swap
>>=20
>> cat /etc/sysct.conf:
>>=20
>> nas# cat /etc/sysctl.conf
>> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1.6.1 2010/12/21 17:09:25 =
kensmith Exp $
>> #
>> # This file is read when going to multi-user and its contents piped =
thru
>> # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for =
details.
>> #
>>=20
>> # Uncomment this to prevent users from seeing information about =
processes that
>> # are being run under another UID.
>> security.bsd.see_other_uids=3D0
>>=20
>> # Disable power button
>> hw.acpi.power_button_state=3DNONE
>> # Disable core dump
>> kern.coredump=3D0
>>=20
>> kern.ipc.maxsockbuf=3D2097152
>> kern.ipc.nmbclusters=3D32768
>> kern.ipc.somaxconn=3D8192
>> kern.maxfiles=3D65536
>> kern.maxfilesperproc=3D32768
>> kern.ipc.nmbjumbo9=3D12800
>>=20
>> # ZFS tuning
>> kern.maxvnodes=3D400000
>> kern.maxfilesperproc=3D200000
>> kern.ipc.maxsockets=3D204800
>>=20
>> =
#http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#File=
-Level_Prefetching
>> vfs.zfs.l2arc_noprefetch=3D0
>> vfs.zfs.l2arc_write_max=3D209715200 ## in bytes. 200MB/s
>> vfs.zfs.l2arc_write_boost=3D314572800 ## in bytes. 300MB/s
>>=20
>> #net.inet.icmp.icmplim=3D300
>> #net.inet.icmp.icmplim_output=3D1
>> #net.inet.ip.fastforwarding=3D1
>> #net.inet.ip.forwarding=3D1
>> #net.inet.tcp.delayed_ack=3D0
>> #net.inet.tcp.inflight.enable=3D0
>> #net.inet.tcp.path_mtu_discovery=3D0
>> #net.inet.tcp.recvspace=3D262144
>> #net.inet.tcp.rfc1323=3D1
>> #net.inet.tcp.sendspace=3D262144
>> #net.inet.udp.maxdgram=3D57344
>> #net.inet.udp.recvspace=3D65536
>> #net.local.stream.recvspace=3D65536
>> #net.local.stream.sendspace=3D65536
>>=20
>>=20
>> # Increase max command-line length showed in `ps` (e.g for =
Tomcat/Java)
>> # Default is PAGE_SIZE / 16 or 256 on x86
>> # For more info see: =
http://www.freebsd.org/cgi/query-pr.cgi?pr=3D120749
>> kern.ps_arg_cache_limit=3D4096
>>=20
>> net.inet.tcp.sendspace=3D65535
>> net.inet.tcp.recvspace=3D131072
>> net.inet.tcp.mssdflt=3D1452
>> net.inet.tcp.sendbuf_max=3D16777216
>> net.inet.tcp.sendbuf_inc=3D524288
>> net.inet.tcp.recvbuf_max=3D16777216
>> net.inet.tcp.recvbuf_inc=3D524288
>> net.inet.udp.recvspace=3D65535
>> net.inet.udp.maxdgram=3D65535
>> net.local.stream.recvspace=3D65535
>> net.local.stream.sendspace=3D65535
>> net.inet.tcp.delayed_ack=3D0
>>=20
>> cat /boot/loader.conf:
>>=20
>> nas# cat /boot/loader.conf
>> zfs_load=3D"YES"
>> kern.maxfiles=3D"20480"
>>=20
>> vfs.root.mountfrom=3D"zfs:NAS/system/root"
>> vfs.root.mountfrom.options=3D"rw"
>>=20
>> #I have 8G of Ram
>> #vfs.zfs.prefetch_disable=3D0
>> vfs.zfs.write_limit_override=3D53687091200
>>=20
>> #If Ram =3D 4GB, set the value to 512M
>> #If Ram =3D 8GB, set the value to 1024M
>> #vfs.zfs.arc_min=3D"4096M"
>>=20
>> #Ram x 0.5 - 512 MB
>> #vfs.zfs.arc_max=3D"15488"
>>=20
>> #Ram x 2
>> #vm.kmem_size_max=3D"64G"
>>=20
>> #Ram x 1.5
>> #vm.kmem_size=3D"48G"
>>=20
>> ahci_load=3D"YES"
>> aio_load=3D"YES"
>> if_em_load=3D"YES"
>> if_lagg_load=3D"YES"
>>=20
>> THIS IS ROOT ON ZFS SYS.
>=20
> Thanks for the info. Given the above, I need even more:
>=20
> - Output from: zdb -C
>=20
> - Output from: smartctl -a ada{0,5}
>=20
> And yes, "9/s" or "10/s" in resilvering means bytes per second. I =
have
> a couple feelings about what's going on (meaning right now I have 3
> separate theories), but I need to see the above first.
>=20
> Also, I swore I remember some Intel ESB2 thing that got committed to
> stable/9 recently, not sure what I'm remembering or if I'm remembering
> it right though, will go look it up later. I should be clear however: =
I
> do not believe at this time the problem you're experiencing is with =
the
> ESB2 southbridge or anything along those lines. If there were issues
> with that on FreeBSD I'd expect them to be widespread given the
> popularity of the chipset in the server realm.
>=20
> --=20
> | Jeremy Chadwick jdc@koitsu.org |
> | UNIX Systems Administrator http://jdc.koitsu.org/ |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2A261BEA-4452-4F6A-8EFB-90A54D79CBB9>