From owner-freebsd-stable@FreeBSD.ORG Wed Sep 29 18:04:20 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1C58106566C for ; Wed, 29 Sep 2010 18:04:20 +0000 (UTC) (envelope-from dan@langille.org) Received: from nyi.unixathome.org (nyi.unixathome.org [64.147.113.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6C6FB8FC14 for ; Wed, 29 Sep 2010 18:04:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by nyi.unixathome.org (Postfix) with ESMTP id 2976C509A8 for ; Wed, 29 Sep 2010 19:04:19 +0100 (BST) X-Virus-Scanned: amavisd-new at unixathome.org Received: from nyi.unixathome.org ([127.0.0.1]) by localhost (nyi.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XHZClX6szfLZ for ; Wed, 29 Sep 2010 19:04:18 +0100 (BST) Received: from nyi.unixathome.org (localhost [127.0.0.1]) by nyi.unixathome.org (Postfix) with ESMTP id 8864A509A3 for ; Wed, 29 Sep 2010 19:04:18 +0100 (BST) Received: from 68.64.144.221 (SquirrelMail authenticated user dan) by nyi.unixathome.org with HTTP; Wed, 29 Sep 2010 14:04:18 -0400 Message-ID: Date: Wed, 29 Sep 2010 14:04:18 -0400 From: "Dan Langille" To: freebsd-stable@freebsd.org User-Agent: SquirrelMail/1.4.20-RC2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Subject: zfs send/receive: is this slow? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Sep 2010 18:04:20 -0000 It's taken about 15 hours to copy 800GB. I'm sure there's some tuning I can do. The system is now running: # zfs send storage/bacula@transfer | zfs receive storage/compressed/bacula All the drives are ATA-8 SATA 2.x device from systat: 1 users Load 0.36 0.58 0.57 Sep 29 13:47 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 42012 7584 544044 11028 204492 count All 962356 8736 1074363k 18220 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 141 cow 9951 total 42 23k 668 3094 1951 2166 657 288 zfod ohci0 ohci ozfod ohci2 ohci 13.6%Sys 0.8%Intr 0.2%User 0.0%Nice 85.5%Idle %ozfod ahc0 irq20 | | | | | | | | | | | daefr ahci0 22 ======= 366 prcfr 2000 cpu0: time 26 dtbuf 47129 totfr 3 em0 irq256 Namei Name-cache Dir-cache 100000 desvn react 892 siis0 257 Calls hits % hits % 87983 numvn pdwak 1056 siis1 259 4608 4608 100 24981 frevn pdpgs 2000 cpu3: time intrn 2000 cpu1: time Disks ada0 ada1 ada2 ada3 ada4 ada5 ada6 1355484 wire 2000 cpu2: time KB/t 35.95 37.00 36.75 41.44 40.05 40.86 41.11 25936 act tps 306 299 301 267 276 271 269 2452756 inact MB/s 10.75 10.82 10.79 10.79 10.80 10.80 10.81 76664 cache %busy 27 50 25 37 27 27 27 127828 free 427728 buf $ zpool iostat 10 capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- storage 7.67T 5.02T 358 38 43.1M 1.96M storage 7.67T 5.02T 317 475 39.4M 30.9M storage 7.67T 5.02T 357 533 44.3M 34.4M storage 7.67T 5.02T 371 556 46.0M 35.8M storage 7.67T 5.02T 313 521 38.9M 28.7M storage 7.67T 5.02T 309 457 38.4M 30.4M storage 7.67T 5.02T 388 589 48.2M 37.8M storage 7.67T 5.02T 377 581 46.8M 36.5M storage 7.67T 5.02T 310 559 38.4M 30.4M storage 7.67T 5.02T 430 611 53.4M 41.3M $ zfs get all storage/compressed NAME PROPERTY VALUE SOURCE storage/compressed type filesystem - storage/compressed creation Tue Sep 28 20:35 2010 - storage/compressed used 856G - storage/compressed available 3.38T - storage/compressed referenced 44.8K - storage/compressed compressratio 1.60x - storage/compressed mounted yes - storage/compressed quota none default storage/compressed reservation none default storage/compressed recordsize 128K default storage/compressed mountpoint /storage/compressed default storage/compressed sharenfs off default storage/compressed checksum on default storage/compressed compression on local storage/compressed atime on default storage/compressed devices on default storage/compressed exec on default storage/compressed setuid on default storage/compressed readonly off default storage/compressed jailed off default storage/compressed snapdir hidden default storage/compressed aclmode groupmask default storage/compressed aclinherit restricted default storage/compressed canmount on default storage/compressed shareiscsi off default storage/compressed xattr off temporary storage/compressed copies 1 default storage/compressed version 4 - storage/compressed utf8only off - storage/compressed normalization none - storage/compressed casesensitivity sensitive - storage/compressed vscan off default storage/compressed nbmand off default storage/compressed sharesmb off default storage/compressed refquota none default storage/compressed refreservation none default storage/compressed primarycache all default storage/compressed secondarycache all default storage/compressed usedbysnapshots 0 - storage/compressed usedbydataset 44.8K - storage/compressed usedbychildren 856G - storage/compressed usedbyrefreservation 0 - $ less /var/run/dmesg.boot Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-STABLE #0: Sat Sep 18 23:43:48 EDT 2010 dan@kraken.example.org:/usr/obj/usr/src/sys/KRAKEN amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) II X4 945 Processor (3010.17-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f42 Family = 10 Model = 4 Stepping = 2 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x37ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 4100673536 (3910 MB) ACPI APIC Table: <111909 APIC1708> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ACPI Warning: Optional field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20100331/tbfadt-655) ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <111909 RSDT1708> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of fec10000, 20 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dfe00000 (3) failed ACPI HPET table warning: Sequence is non-zero (2) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 18 at device 2.0 on pci0 pci7: on pcib1 em0: port 0xec00-0xec1f mem 0xfbfe0000-0xfbffffff,0xfbf00000-0xfbf7ffff,0xfbfdc000-0xfbfdffff irq 18 at device 0.0 on pci7 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1b:21:51:ab:2d pcib2: irq 17 at device 5.0 on pci0 pci5: on pcib2 pcib3: irq 17 at device 0.0 on pci5 pci6: on pcib3 siis0: port 0xdc00-0xdc0f mem 0xfbeffc00-0xfbeffc7f,0xfbef0000-0xfbef7fff irq 17 at device 4.0 on pci6 siis0: [ITHREAD] siisch0: at channel 0 on siis0 siisch0: [ITHREAD] siisch1: at channel 1 on siis0 siisch1: [ITHREAD] siisch2: at channel 2 on siis0 siisch2: [ITHREAD] siisch3: at channel 3 on siis0 siisch3: [ITHREAD] pcib4: irq 18 at device 6.0 on pci0 pci4: on pcib4 re0: port 0xc800-0xc8ff mem 0xfbdff000-0xfbdfffff irq 18 at device 0.0 on pci4 re0: Using 1 MSI messages re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto re0: Ethernet address: e0:cb:4e:42:f0:ff re0: [FILTER] pcib5: irq 19 at device 11.0 on pci0 pci2: on pcib5 pcib6: irq 19 at device 0.0 on pci2 pci3: on pcib6 siis1: port 0xbc00-0xbc0f mem 0xfbcffc00-0xfbcffc7f,0xfbcf0000-0xfbcf7fff irq 19 at device 4.0 on pci3 siis1: [ITHREAD] siisch4: at channel 0 on siis1 siisch4: [ITHREAD] siisch5: at channel 1 on siis1 siisch5: [ITHREAD] siisch6: at channel 2 on siis1 siisch6: [ITHREAD] siisch7: at channel 3 on siis1 siisch7: [ITHREAD] ahci0: port 0x9000-0x9007,0x8000-0x8003,0x7000-0x7007,0x6000-0x6003,0x5000-0x500f mem 0xfb6fe400-0xfb6fe7ff irq 22 at device 17.0 on pci0 ahci0: [ITHREAD] ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: at channel 1 on ahci0 ahcich1: [ITHREAD] ahcich2: at channel 2 on ahci0 ahcich2: [ITHREAD] ahcich3: at channel 3 on ahci0 ahcich3: [ITHREAD] ohci0: mem 0xfb6fa000-0xfb6fafff irq 16 at device 18.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ohci1: mem 0xfb6fb000-0xfb6fbfff irq 16 at device 18.1 on pci0 ohci1: [ITHREAD] usbus1: on ohci1 ehci0: mem 0xfb6fe800-0xfb6fe8ff irq 17 at device 18.2 on pci0 ehci0: [ITHREAD] ehci0: AMD SB600/700 quirk applied usbus2: EHCI version 1.0 usbus2: on ehci0 ohci2: mem 0xfb6fc000-0xfb6fcfff irq 18 at device 19.0 on pci0 ohci2: [ITHREAD] usbus3: on ohci2 ohci3: mem 0xfb6fd000-0xfb6fdfff irq 18 at device 19.1 on pci0 ohci3: [ITHREAD] usbus4: on ohci3 ehci1: mem 0xfb6fec00-0xfb6fecff irq 19 at device 19.2 on pci0 ehci1: [ITHREAD] ehci1: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5: on ehci1 pci0: at device 20.0 (no driver attached) atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] isab0: at device 20.3 on pci0 isa0: on isab0 pcib7: at device 20.4 on pci0 pci1: on pcib7 ahc0: port 0xa800-0xa8ff mem 0xfb7df000-0xfb7dffff irq 20 at device 5.0 on pci1 ahc0: [ITHREAD] aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs vgapci0: mem 0xfb800000-0xfbbfffff,0xfb7f0000-0xfb7fffff irq 21 at device 6.0 on pci1 ohci4: mem 0xfb6ff000-0xfb6fffff irq 18 at device 20.5 on pci0 ohci4: [ITHREAD] usbus6: on ohci4 acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc87ff,0xc8800-0xc97ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: cannot reserve I/O port range acpi_throttle0: on cpu0 hwpstate0: on cpu0 Timecounters tick every 1.000 msec (noperiph:siisch0:0:-1:-1): rescan already queued (noperiph:siisch2:0:-1:-1): rescan already queued (noperiph:siisch3:0:-1:-1): rescan already queued (noperiph:siisch4:0:-1:-1): rescan already queued (noperiph:siisch5:0:-1:-1): rescan already queued (noperiph:siisch6:0:-1:-1): rescan already queued (noperiph:siisch7:0:-1:-1): rescan already queued usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 usbus6: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 uhub6: 2 ports with 2 removable, self powered uhub0: 3 ports with 3 removable, self powered uhub1: 3 ports with 3 removable, self powered uhub3: 3 ports with 3 removable, self powered uhub4: 3 ports with 3 removable, self powered uhub2: 6 ports with 6 removable, self powered uhub5: 6 ports with 6 removable, self powered ada0 at siisch0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada1 at siisch2 bus 0 scbus2 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2 at siisch3 bus 0 scbus3 target 0 lun 0 ada2: ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3 at siisch4 bus 0 scbus4 target 0 lun 0 ada3: ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada4 at siisch5 bus 0 scbus5 target 0 lun 0 ada4: ATA-8 SATA 2.x device ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada5 at siisch6 bus 0 scbus6 target 0 lun 0 ada5: ATA-8 SATA 2.x device ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada6 at siisch7 bus 0 scbus7 target 0 lun 0 ada6: ATA-8 SATA 2.x device ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada7 at ahcich0 bus 0 scbus8 target 0 lun 0 ada7: ATA-7 SATA 2.x device ada7: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada7: Command Queueing enabled ada7: 76319MB (156301488 512 byte sectors: 16H 63S/T 16383C) ada8 at ahcich2 bus 0 scbus10 target 0 lun 0 ada8: ATA-8 SATA 2.x device ada8: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada8: Command Queueing enabled ada8: 152587MB (312500000 512 byte sectors: 16H 63S/T 16383C) SMP: AP CPU #3 Launched! cd0 at ahcich1 bus 0 scbus9 target 0 lun 0SMP: AP CPU #1 Launched! cd0: Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)SMP: AP CPU #2 Launched! cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed GEOM_MIRROR: Device mirror/gm0 launched (1/2). GEOM_MIRROR: Device gm0: rebuilding provider ada7. GEOM: mirror/gm0s1: geometry does not match label (16h,63s != 255h,63s). Trying to mount root from ufs:/dev/mirror/gm0s1a WARNING: / was not properly dismounted ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 4 ZFS storage pool version 15 WARNING: /tmp was not properly dismounted WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted -- Dan Langille -- http://langille.org/