From owner-freebsd-fs@FreeBSD.ORG Thu Jul 4 17:36:39 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4AEC3E7 for ; Thu, 4 Jul 2013 17:36:39 +0000 (UTC) (envelope-from mxb@alumni.chalmers.se) Received: from mail-la0-f44.google.com (mail-la0-f44.google.com [209.85.215.44]) by mx1.freebsd.org (Postfix) with ESMTP id AE3D71305 for ; Thu, 4 Jul 2013 17:36:38 +0000 (UTC) Received: by mail-la0-f44.google.com with SMTP id er20so1451325lab.31 for ; Thu, 04 Jul 2013 10:36:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=V3+h3DHqWswKtjjXU7r/RAMxbVgiAk7EddoGDigEMPE=; b=kBw+Xq3VfATHV8BoJWlEOix5FnMGxJQEzOoS9wi/0vYoj2l+f1uoc09PLIG7RNaXLY 2k35cW1fwaCVrDkmqaTOp6h3DM7DpalK7yh27AvXmHE/XHeKGZeIzYVcYrpff2ONABHx b64cKkVJ7WYulLieH3FPkKgv0wyYS/iCI+vRMgG2bKfrKKiXgtSY8W92zJKFBDAoQvr8 AoQFcQGvaq8CQntdobkGAQuWeM9QVPSrR74HNJwU5yUcBK103idafInGOHxAXR2Rf6CX 9GYo+NR0w/eEjOvwln/Fw/W5HRWi9Q5PXcOPsQveAKEeWezQk3nIsOrKsaONkqNlQioW 0lPg== X-Received: by 10.152.44.170 with SMTP id f10mr3256619lam.68.1372959390591; Thu, 04 Jul 2013 10:36:30 -0700 (PDT) Received: from grey.home.unixconn.com (h-74-23.a183.priv.bahnhof.se. [46.59.74.23]) by mx.google.com with ESMTPSA id y5sm1507202lae.2.2013.07.04.10.36.28 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 04 Jul 2013 10:36:29 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: Slow resilvering with mirrored ZIL From: mxb In-Reply-To: <20130704171637.GA94539@icarus.home.lan> Date: Thu, 4 Jul 2013 19:36:27 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <2A261BEA-4452-4F6A-8EFB-90A54D79CBB9@alumni.chalmers.se> References: <51D42107.1050107@digsys.bg> <2EF46A8C-6908-4160-BF99-EC610B3EA771@alumni.chalmers.se> <51D437E2.4060101@digsys.bg> <20130704000405.GA75529@icarus.home.lan> <20130704171637.GA94539@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1508) X-Gm-Message-State: ALoCoQkox10Nga9IjFZbGRo8ljwuRaYcMrKeCqBGNaC6g0D7IYdsv4pg9/rWpWl1xvDroK5dB4cB Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2013 17:36:39 -0000 zdb -C: NAS: version: 28 name: 'NAS' state: 0 txg: 19039918 pool_guid: 3808946822857359331 hostid: 516334119 hostname: 'nas.home.unixconn.com' vdev_children: 2 vdev_tree: type: 'root' id: 0 guid: 3808946822857359331 children[0]: type: 'raidz' id: 0 guid: 15126043265564363201 nparity: 1 metaslab_array: 14 metaslab_shift: 32 ashift: 9 asize: 4000799784960 is_log: 0 children[0]: type: 'replacing' id: 0 guid: 15772352161192928927 whole_disk: 0 children[0]: type: 'disk' id: 0 guid: 1160414651350745057 path: '/dev/ada0/old' phys_path: '/dev/ada0' whole_disk: 0 DTL: 788 children[1]: type: 'disk' id: 1 guid: 16857066297968094282 path: '/dev/ada0' phys_path: '/dev/ada0' whole_disk: 1 DTL: 4100 resilvering: 1 children[1]: type: 'disk' id: 1 guid: 8602024497074155869 path: '/dev/ada3' phys_path: '/dev/ada3' whole_disk: 1 DTL: 4097 children[2]: type: 'disk' id: 2 guid: 15590206669698985709 path: '/dev/ada1' phys_path: '/dev/ada1' whole_disk: 0 DTL: 786 children[3]: type: 'disk' id: 3 guid: 14741760026602900071 path: '/dev/ada2' phys_path: '/dev/ada2' whole_disk: 0 DTL: 785 children[1]: type: 'mirror' id: 1 guid: 6427576469721219307 metaslab_array: 4098 metaslab_shift: 26 ashift: 9 asize: 10732437504 is_log: 1 create_txg: 19039876 children[0]: type: 'disk' id: 0 guid: 14073296989893822073 path: '/dev/ada4s1' phys_path: '/dev/ada4s1' whole_disk: 1 create_txg: 19039876 children[1]: type: 'disk' id: 1 guid: 8631947194818711903 path: '/dev/ada5s1' phys_path: '/dev/ada5s1' whole_disk: 1 create_txg: 19039876 smartctl: nas# smartctl -a ada3 smartctl 6.0 2012-10-10 r3643 [FreeBSD 9.1-RELEASE-p1 amd64] (local = build) Copyright (C) 2002-12, Bruce Allen, Christian Franke, = www.smartmontools.org ada3: Unable to detect device type Please specify device type with the -d option. Use smartctl -h to get a usage summary THIS IS FOR ALL DISKS I HAVE INSIDE. On 4 jul 2013, at 19:16, Jeremy Chadwick wrote: > On Thu, Jul 04, 2013 at 06:57:09PM +0200, mxb wrote: >>=20 >> Well, I'v got a lot of errors in dmesg regarding one of four disks, = which is not currently replaced. >> After resilvering dropped to 2MB/s ( :O ), it stuck. I had to reboot = system. >> It came up and now continues resilvering process at 9/s (9 what??? = bytes? ) >>=20 >> Here comes requested info: >>=20 >> dmesg(plan was to re-new disks and then lift this sys up to STABLE): >>=20 >> Copyright (c) 1992-2012 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, = 1994 >> The Regents of the University of California. All rights = reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.1-RELEASE-p1 #2: Fri Mar 8 10:30:38 CET 2013 >> root@nas.home.unixconn.com:/usr/obj/usr/src/sys/GENERIC amd64 >> module_register: module pci/em already exists! >> Module pci/em failed to register: 17 >> module_register: module pci/lem already exists! >> Module pci/lem failed to register: 17 >> CPU: Intel(R) Xeon(R) CPU X5460 @ 3.16GHz (3166.74-MHz = K8-class CPU) >> Origin =3D "GenuineIntel" Id =3D 0x10676 Family =3D 6 Model =3D = 17 Stepping =3D 6 >> = Features=3D0xbfebfbff >> = Features2=3D0xce3bd >> AMD Features=3D0x20100800 >> AMD Features2=3D0x1 >> TSC: P-state invariant, performance statistics >> real memory =3D 34359738368 (32768 MB) >> avail memory =3D 33095909376 (31562 MB) >> Event timer "LAPIC" quality 400 >> ACPI APIC Table: >> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs >> FreeBSD/SMP: 1 package(s) x 4 core(s) >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> cpu2 (AP): APIC ID: 2 >> cpu3 (AP): APIC ID: 3 >> ioapic0 irqs 0-23 on motherboard >> ioapic1 irqs 24-47 on motherboard >> kbd1 at kbdmux0 >> ctl: CAM Target Layer loaded >> acpi0: on motherboard >> acpi0: Power Button (fixed) >> cpu0: on acpi0 >> cpu1: on acpi0 >> cpu2: on acpi0 >> cpu3: on acpi0 >> atrtc0: port 0x70-0x71 irq 8 on acpi0 >> Event timer "RTC" frequency 32768 Hz quality 0 >> attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> Event timer "i8254" frequency 1193182 Hz quality 100 >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on = acpi0 >> pcib0: port 0xcf8-0xcff on acpi0 >> pci0: on pcib0 >> pcib1: at device 2.0 on pci0 >> pci1: on pcib1 >> pcib2: irq 16 at device 0.0 on pci1 >> pci2: on pcib2 >> pcib3: irq 16 at device 0.0 on pci2 >> pci3: on pcib3 >> pcib4: at device 0.0 on pci3 >> pci4: on pcib4 >> pcib5: at device 0.2 on pci3 >> pci5: on pcib5 >> pcib6: irq 18 at device 2.0 on pci2 >> pci6: on pcib6 >> em0: port 0x2000-0x201f = mem 0xdc000000-0xdc01ffff irq 18 at device 0.0 on pci6 >> em0: Using an MSI interrupt >> em0: Ethernet address: 00:30:48:34:46:36 >> em1: port 0x2020-0x203f = mem 0xdc020000-0xdc03ffff irq 19 at device 0.1 on pci6 >> em1: Using an MSI interrupt >> em1: Ethernet address: 00:30:48:34:46:37 >> pcib7: at device 0.3 on pci1 >> pci7: on pcib7 >> pcib8: at device 4.0 on pci0 >> pci8: on pcib8 >> pcib9: at device 0.0 on pci8 >> pci9: on pcib9 >> bce0: mem = 0xd8000000-0xd9ffffff irq 16 at device 4.0 on pci9 >> miibus0: on bce0 >> brgphy0: PHY 1 on miibus0 >> brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, = 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow >> bce0: Ethernet address: 00:1b:78:38:2f:28 >> bce0: ASIC (0x57060020); Rev (A2); Bus (PCI-X, 64-bit, 100MHz); B/C = (1.9.6); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI) >> Coal (RX:6,6,18,18; TX:20,20,80,80) >> pcib10: at device 0.2 on pci8 >> pci10: on pcib10 >> bce1: mem = 0xda000000-0xdbffffff irq 17 at device 5.0 on pci10 >> miibus1: on bce1 >> brgphy1: PHY 1 on miibus1 >> brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, = 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow >> bce1: Ethernet address: 00:1b:78:38:2f:2a >> bce1: ASIC (0x57060020); Rev (A2); Bus (PCI-X, 64-bit, 100MHz); B/C = (1.9.6); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI) >> Coal (RX:6,6,18,18; TX:20,20,80,80) >> pcib11: at device 6.0 on pci0 >> pci11: on pcib11 >> pci0: at device 8.0 (no driver attached) >> pcib12: irq 17 at device 28.0 on pci0 >> pci12: on pcib12 >> uhci0: port = 0x1800-0x181f irq 17 at device 29.0 on pci0 >> usbus0 on uhci0 >> uhci1: port = 0x1820-0x183f irq 19 at device 29.1 on pci0 >> usbus1 on uhci1 >> uhci2: port = 0x1840-0x185f irq 18 at device 29.2 on pci0 >> usbus2 on uhci2 >> ehci0: mem 0xdc500000-0xdc5003ff = irq 17 at device 29.7 on pci0 >> usbus3: EHCI version 1.0 >> usbus3 on ehci0 >> pcib13: at device 30.0 on pci0 >> pci13: on pcib13 >> vgapci0: port 0x3000-0x30ff mem = 0xd0000000-0xd7ffffff,0xdc200000-0xdc20ffff irq 18 at device 1.0 on = pci13 >> isab0: at device 31.0 on pci0 >> isa0: on isab0 >> ahci0: port = 0x1890-0x1897,0x1884-0x1887,0x1888-0x188f,0x1880-0x1883,0x1860-0x187f = mem 0xdc500400-0xdc5007ff irq 19 at device 31.2 on pci0 >> ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported >> ahcich0: at channel 0 on ahci0 >> ahcich1: at channel 1 on ahci0 >> ahcich2: at channel 2 on ahci0 >> ahcich3: at channel 3 on ahci0 >> ahcich4: at channel 4 on ahci0 >> ahcich5: at channel 5 on ahci0 >> pci0: at device 31.3 (no driver attached) >> acpi_button0: on acpi0 >> atkbdc0: port 0x60,0x64 irq 1 on acpi0 >> atkbd0: irq 1 on atkbdc0 >> kbd0 at atkbd0 >> atkbd0: [GIANT-LOCKED] >> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on = acpi0 >> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 >> fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on = acpi0 >> fdc0: does not respond >> device_attach: fdc0 attach returned 6 >> ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on = acpi0 >> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode >> ppc0: FIFO with 16/16/9 bytes threshold >> ppbus0: on ppc0 >> plip0: on ppbus0 >> lpt0: on ppbus0 >> lpt0: Interrupt-driven port >> ppi0: on ppbus0 >> orm0: at iomem = 0xc0000-0xcafff,0xcb000-0xccfff,0xcd000-0xce7ff,0xce800-0xcffff on isa0 >> sc0: at flags 0x100 on isa0 >> sc0: VGA <16 virtual consoles, flags=3D0x300> >> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 >> fdc0: No FDOUT register! >> est0: on cpu0 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 4921492106004921 >> device_attach: est0 attach returned 6 >> p4tcc0: on cpu0 >> est1: on cpu1 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 4921492106004921 >> device_attach: est1 attach returned 6 >> p4tcc1: on cpu1 >> est2: on cpu2 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 4921492106004921 >> device_attach: est2 attach returned 6 >> p4tcc2: on cpu2 >> est3: on cpu3 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 4921492106004921 >> device_attach: est3 attach returned 6 >> p4tcc3: on cpu3 >> ZFS filesystem version 5 >> ZFS storage pool version 28 >> Timecounters tick every 1.000 msec >> usbus0: 12Mbps Full Speed USB v1.0 >> usbus1: 12Mbps Full Speed USB v1.0 >> usbus2: 12Mbps Full Speed USB v1.0 >> usbus3: 480Mbps High Speed USB v2.0 >> ugen0.1: at usbus0 >> uhub0: on = usbus0 >> ugen1.1: at usbus1 >> uhub1: on = usbus1 >> ugen2.1: at usbus2 >> uhub2: on = usbus2 >> ugen3.1: at usbus3 >> uhub3: on = usbus3 >> uhub0: 2 ports with 2 removable, self powered >> uhub1: 2 ports with 2 removable, self powered >> uhub2: 2 ports with 2 removable, self powered >> bce0: bce_pulse(): Warning: bootcode thinks driver is absent! = (bc_state =3D 0x00000004) >> bce1: bce_pulse(): Warning: bootcode thinks driver is absent! = (bc_state =3D 0x00000004) >> uhub3: 6 ports with 6 removable, self powered >> ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 >> ada0: ATA-8 SATA 3.x device >> ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada0: Command Queueing enabled >> ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) >> ada0: Previously was known as ad4 >> ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 >> ada1: ATA-8 SATA 2.x device >> ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada1: Command Queueing enabled >> ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) >> ada1: Previously was known as ad6 >> ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 >> ada2: ATA-8 SATA 2.x device >> ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada2: Command Queueing enabled >> ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) >> ada2: Previously was known as ad8 >> ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 >> ada3: ATA-8 SATA 3.x device >> ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada3: Command Queueing enabled >> ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) >> ada3: Previously was known as ad10 >> ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 >> ada4: ATA-7 SATA 2.x device >> ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada4: Command Queueing enabled >> ada4: 19087MB (39091248 512 byte sectors: 16H 63S/T 16383C) >> ada4: Previously was known as ad12 >> ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 >> ada5: ATA-8 SATA 3.x device >> ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada5: Command Queueing enabled >> ada5: 57241MB (117231408 512 byte sectors: 16H 63S/T 16383C) >> ada5: Previously was known as ad14 >> SMP: AP CPU #3 Launched! >> SMP: AP CPU #1 Launched! >> SMP: AP CPU #2 Launched! >> Timecounter "TSC-low" frequency 12370070 Hz quality 1000 >> Root mount waiting for: usbus3 >> ugen0.2: at usbus0 >> ukbd0: on usbus0 >> kbd2 at ukbd0 >> Root mount waiting for: usbus3 >> ugen3.2: at usbus3 >> umass0: on = usbus3 >> umass0: SCSI over Bulk-Only; quirks =3D 0x4100 >> umass0:7:0:-1: Attached to scbus7 >> Trying to mount root from zfs:NAS/system/root [rw]... >> da0 at umass-sim0 bus 0 scbus7 target 0 lun 0 >> da0: Removable Direct Access SCSI-0 = device >> da0: 40.000MB/s transfers >> da0: 7580MB (15523840 512 byte sectors: 255H 63S/T 966C) >> GEOM_PART: integrity check failed (label/zfs_boot, BSD) >> GEOM_PART: integrity check failed (label/zfs_boot, BSD) >> bce0: Gigabit link up! >> bce1: Gigabit link up! >>=20 >> zpool status: >> pool: NAS >> state: DEGRADED >> status: One or more devices is currently being resilvered. The pool = will >> continue to function, possibly in a degraded state. >> action: Wait for the resilver to complete. >> scan: resilver in progress since Tue Jul 2 07:31:29 2013 >> 483G scanned out of 3.49T at 10/s, (scan is slow, no estimated = time) >> 121G resilvered, 13.52% done >> config: >>=20 >> NAME STATE READ WRITE CKSUM >> NAS DEGRADED 0 0 0 >> raidz1-0 DEGRADED 0 0 0 >> replacing-0 DEGRADED 0 0 0 >> 1160414651350745057 UNAVAIL 0 0 0 was = /dev/ada0/old >> ada0 ONLINE 0 0 0 = (resilvering) >> ada3 ONLINE 0 0 0 >> ada1 ONLINE 0 0 0 >> ada2 ONLINE 0 0 0 >> logs >> mirror-1 ONLINE 0 0 0 >> ada4s1 ONLINE 0 0 0 >> ada5s1 ONLINE 0 0 0 >> cache >> ada5s2 ONLINE 0 0 0 >>=20 >> errors: No known data errors >>=20 >> zpool get all: >> NAME PROPERTY VALUE SOURCE >> NAS size 3.64T - >> NAS capacity 95% - >> NAS altroot - default >> NAS health DEGRADED - >> NAS guid 3808946822857359331 default >> NAS version 28 default >> NAS bootfs - default >> NAS delegation on default >> NAS autoreplace off default >> NAS cachefile - default >> NAS failmode wait default >> NAS listsnapshots off default >> NAS autoexpand off default >> NAS dedupditto 0 default >> NAS dedupratio 1.00x - >> NAS free 154G - >> NAS allocated 3.49T - >> NAS readonly off - >> NAS comment - default >> NAS expandsize 0 - >>=20 >> zfs get all: >> NAME PROPERTY VALUE = SOURCE >> NAS type filesystem - >> NAS creation Tue Aug 3 20:10 2010 - >> NAS used 2.61T - >> NAS available 72.0G - >> NAS referenced 62.8K - >> NAS compressratio 1.00x - >> NAS mounted yes - >> NAS quota none = default >> NAS reservation none = default >> NAS recordsize 128K = default >> NAS mountpoint /mnt/NAS = local >> NAS sharenfs off = default >> NAS checksum on = default >> NAS compression off = default >> NAS atime off = local >> NAS devices on = default >> NAS exec on = default >> NAS setuid on = default >> NAS readonly off = default >> NAS jailed off = default >> NAS snapdir hidden = default >> NAS aclmode discard = default >> NAS aclinherit restricted = default >> NAS canmount on = default >> NAS xattr off = temporary >> NAS copies 1 = default >> NAS version 5 - >> NAS utf8only off - >> NAS normalization none - >> NAS casesensitivity sensitive - >> NAS vscan off = default >> NAS nbmand off = default >> NAS sharesmb off = default >> NAS refquota none = default >> NAS refreservation none = default >> NAS primarycache all = default >> NAS secondarycache all = default >> NAS usedbysnapshots 0 - >> NAS usedbydataset 0 - >> NAS usedbychildren 0 - >> NAS usedbyrefreservation 0 - >> NAS logbias latency = default >> NAS dedup off = default >> NAS mlslabel - >> NAS sync standard = local >> NAS refcompressratio 1.00x - >> NAS written 62.8K - >>=20 >> the rest of created - defaults. no changes or inherited from this = one. >>=20 >> part show -p: >> nas# gpart show -p ada0 >> gpart: No such geom: ada0. >> nas# gpart show -p ada1 >> gpart: No such geom: ada1. >> nas# gpart show -p ada2 >> gpart: No such geom: ada2. >> nas# gpart show -p ada3 >> gpart: No such geom: ada3. >> nas# gpart show -p ada4 >> =3D> 63 39091185 ada4 MBR (18G) >> 63 20971503 ada4s1 freebsd (10G) >> 20971566 18119682 ada4s2 freebsd (8.7G) >>=20 >> nas# gpart show -p ada5 >> =3D> 63 117231345 ada5 MBR (55G) >> 63 20971503 ada5s1 freebsd (10G) >> 20971566 96259842 ada5s2 freebsd (45G) >>=20 >> ada4/ada5 are ZIL/L2ARC/swap >>=20 >> cat /etc/sysct.conf: >>=20 >> nas# cat /etc/sysctl.conf >> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1.6.1 2010/12/21 17:09:25 = kensmith Exp $ >> # >> # This file is read when going to multi-user and its contents piped = thru >> # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for = details. >> # >>=20 >> # Uncomment this to prevent users from seeing information about = processes that >> # are being run under another UID. >> security.bsd.see_other_uids=3D0 >>=20 >> # Disable power button >> hw.acpi.power_button_state=3DNONE >> # Disable core dump >> kern.coredump=3D0 >>=20 >> kern.ipc.maxsockbuf=3D2097152 >> kern.ipc.nmbclusters=3D32768 >> kern.ipc.somaxconn=3D8192 >> kern.maxfiles=3D65536 >> kern.maxfilesperproc=3D32768 >> kern.ipc.nmbjumbo9=3D12800 >>=20 >> # ZFS tuning >> kern.maxvnodes=3D400000 >> kern.maxfilesperproc=3D200000 >> kern.ipc.maxsockets=3D204800 >>=20 >> = #http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#File= -Level_Prefetching >> vfs.zfs.l2arc_noprefetch=3D0 >> vfs.zfs.l2arc_write_max=3D209715200 ## in bytes. 200MB/s >> vfs.zfs.l2arc_write_boost=3D314572800 ## in bytes. 300MB/s >>=20 >> #net.inet.icmp.icmplim=3D300 >> #net.inet.icmp.icmplim_output=3D1 >> #net.inet.ip.fastforwarding=3D1 >> #net.inet.ip.forwarding=3D1 >> #net.inet.tcp.delayed_ack=3D0 >> #net.inet.tcp.inflight.enable=3D0 >> #net.inet.tcp.path_mtu_discovery=3D0 >> #net.inet.tcp.recvspace=3D262144 >> #net.inet.tcp.rfc1323=3D1 >> #net.inet.tcp.sendspace=3D262144 >> #net.inet.udp.maxdgram=3D57344 >> #net.inet.udp.recvspace=3D65536 >> #net.local.stream.recvspace=3D65536 >> #net.local.stream.sendspace=3D65536 >>=20 >>=20 >> # Increase max command-line length showed in `ps` (e.g for = Tomcat/Java) >> # Default is PAGE_SIZE / 16 or 256 on x86 >> # For more info see: = http://www.freebsd.org/cgi/query-pr.cgi?pr=3D120749 >> kern.ps_arg_cache_limit=3D4096 >>=20 >> net.inet.tcp.sendspace=3D65535 >> net.inet.tcp.recvspace=3D131072 >> net.inet.tcp.mssdflt=3D1452 >> net.inet.tcp.sendbuf_max=3D16777216 >> net.inet.tcp.sendbuf_inc=3D524288 >> net.inet.tcp.recvbuf_max=3D16777216 >> net.inet.tcp.recvbuf_inc=3D524288 >> net.inet.udp.recvspace=3D65535 >> net.inet.udp.maxdgram=3D65535 >> net.local.stream.recvspace=3D65535 >> net.local.stream.sendspace=3D65535 >> net.inet.tcp.delayed_ack=3D0 >>=20 >> cat /boot/loader.conf: >>=20 >> nas# cat /boot/loader.conf >> zfs_load=3D"YES" >> kern.maxfiles=3D"20480" >>=20 >> vfs.root.mountfrom=3D"zfs:NAS/system/root" >> vfs.root.mountfrom.options=3D"rw" >>=20 >> #I have 8G of Ram >> #vfs.zfs.prefetch_disable=3D0 >> vfs.zfs.write_limit_override=3D53687091200 >>=20 >> #If Ram =3D 4GB, set the value to 512M >> #If Ram =3D 8GB, set the value to 1024M >> #vfs.zfs.arc_min=3D"4096M" >>=20 >> #Ram x 0.5 - 512 MB >> #vfs.zfs.arc_max=3D"15488" >>=20 >> #Ram x 2 >> #vm.kmem_size_max=3D"64G" >>=20 >> #Ram x 1.5 >> #vm.kmem_size=3D"48G" >>=20 >> ahci_load=3D"YES" >> aio_load=3D"YES" >> if_em_load=3D"YES" >> if_lagg_load=3D"YES" >>=20 >> THIS IS ROOT ON ZFS SYS. >=20 > Thanks for the info. Given the above, I need even more: >=20 > - Output from: zdb -C >=20 > - Output from: smartctl -a ada{0,5} >=20 > And yes, "9/s" or "10/s" in resilvering means bytes per second. I = have > a couple feelings about what's going on (meaning right now I have 3 > separate theories), but I need to see the above first. >=20 > Also, I swore I remember some Intel ESB2 thing that got committed to > stable/9 recently, not sure what I'm remembering or if I'm remembering > it right though, will go look it up later. I should be clear however: = I > do not believe at this time the problem you're experiencing is with = the > ESB2 southbridge or anything along those lines. If there were issues > with that on FreeBSD I'd expect them to be widespread given the > popularity of the chipset in the server realm. >=20 > --=20 > | Jeremy Chadwick jdc@koitsu.org | > | UNIX Systems Administrator http://jdc.koitsu.org/ | > | Making life hard for others since 1977. PGP 4BD6C0CB | >=20