Date: Wed, 29 Sep 2010 14:04:18 -0400 From: "Dan Langille" <dan@langille.org> To: freebsd-stable@freebsd.org Subject: zfs send/receive: is this slow? Message-ID: <a263c3beaeb0fa3acd82650775e31ee3.squirrel@nyi.unixathome.org>
next in thread | raw e-mail | index | archive | help
It's taken about 15 hours to copy 800GB. I'm sure there's some tuning I
can do.
The system is now running:
# zfs send storage/bacula@transfer | zfs receive storage/compressed/bacula
All the drives are <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
from systat:
1 users Load 0.36 0.58 0.57 Sep 29 13:47
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 42012 7584 544044 11028 204492 count
All 962356 8736 1074363k 18220 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt 141 cow 9951 total
42 23k 668 3094 1951 2166 657 288 zfod
ohci0 ohci
ozfod
ohci2 ohci
13.6%Sys 0.8%Intr 0.2%User 0.0%Nice 85.5%Idle %ozfod ahc0
irq20
| | | | | | | | | | | daefr
ahci0 22
======= 366 prcfr 2000
cpu0: time
26 dtbuf 47129 totfr 3 em0
irq256
Namei Name-cache Dir-cache 100000 desvn react 892
siis0 257
Calls hits % hits % 87983 numvn pdwak 1056
siis1 259
4608 4608 100 24981 frevn pdpgs 2000
cpu3: time
intrn 2000
cpu1: time
Disks ada0 ada1 ada2 ada3 ada4 ada5 ada6 1355484 wire 2000
cpu2: time
KB/t 35.95 37.00 36.75 41.44 40.05 40.86 41.11 25936 act
tps 306 299 301 267 276 271 269 2452756 inact
MB/s 10.75 10.82 10.79 10.79 10.80 10.80 10.81 76664 cache
%busy 27 50 25 37 27 27 27 127828 free
427728 buf
$ zpool iostat 10
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
storage 7.67T 5.02T 358 38 43.1M 1.96M
storage 7.67T 5.02T 317 475 39.4M 30.9M
storage 7.67T 5.02T 357 533 44.3M 34.4M
storage 7.67T 5.02T 371 556 46.0M 35.8M
storage 7.67T 5.02T 313 521 38.9M 28.7M
storage 7.67T 5.02T 309 457 38.4M 30.4M
storage 7.67T 5.02T 388 589 48.2M 37.8M
storage 7.67T 5.02T 377 581 46.8M 36.5M
storage 7.67T 5.02T 310 559 38.4M 30.4M
storage 7.67T 5.02T 430 611 53.4M 41.3M
$ zfs get all storage/compressed
NAME PROPERTY VALUE SOURCE
storage/compressed type filesystem -
storage/compressed creation Tue Sep 28 20:35 2010 -
storage/compressed used 856G -
storage/compressed available 3.38T -
storage/compressed referenced 44.8K -
storage/compressed compressratio 1.60x -
storage/compressed mounted yes -
storage/compressed quota none default
storage/compressed reservation none default
storage/compressed recordsize 128K default
storage/compressed mountpoint /storage/compressed default
storage/compressed sharenfs off default
storage/compressed checksum on default
storage/compressed compression on local
storage/compressed atime on default
storage/compressed devices on default
storage/compressed exec on default
storage/compressed setuid on default
storage/compressed readonly off default
storage/compressed jailed off default
storage/compressed snapdir hidden default
storage/compressed aclmode groupmask default
storage/compressed aclinherit restricted default
storage/compressed canmount on default
storage/compressed shareiscsi off default
storage/compressed xattr off temporary
storage/compressed copies 1 default
storage/compressed version 4 -
storage/compressed utf8only off -
storage/compressed normalization none -
storage/compressed casesensitivity sensitive -
storage/compressed vscan off default
storage/compressed nbmand off default
storage/compressed sharesmb off default
storage/compressed refquota none default
storage/compressed refreservation none default
storage/compressed primarycache all default
storage/compressed secondarycache all default
storage/compressed usedbysnapshots 0 -
storage/compressed usedbydataset 44.8K -
storage/compressed usedbychildren 856G -
storage/compressed usedbyrefreservation 0 -
$ less /var/run/dmesg.boot
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-STABLE #0: Sat Sep 18 23:43:48 EDT 2010
dan@kraken.example.org:/usr/obj/usr/src/sys/KRAKEN amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Phenom(tm) II X4 945 Processor (3010.17-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0x100f42 Family = 10 Model = 4 Stepping
= 2
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x802009<SSE3,MON,CX16,POPCNT>
AMD
Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
AMD
Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
TSC: P-state invariant
real memory = 4294967296 (4096 MB)
avail memory = 4100673536 (3910 MB)
ACPI APIC Table: <111909 APIC1708>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
ACPI Warning: Optional field Pm2ControlBlock has zero address or length:
0x0000000000000000/0x1 (20100331/tbfadt-655)
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <111909 RSDT1708> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of ffb80000, 80000 (3) failed
acpi0: reservation of fec10000, 20 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dfe00000 (3) failed
ACPI HPET table warning: Sequence is non-zero (2)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci0
pci7: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0xec00-0xec1f mem
0xfbfe0000-0xfbffffff,0xfbf00000-0xfbf7ffff,0xfbfdc000-0xfbfdffff irq 18
at device 0.0 on pci7
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:1b:21:51:ab:2d
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib2
pcib3: <PCI-PCI bridge> irq 17 at device 0.0 on pci5
pci6: <PCI bus> on pcib3
siis0: <SiI3124 SATA controller> port 0xdc00-0xdc0f mem
0xfbeffc00-0xfbeffc7f,0xfbef0000-0xfbef7fff irq 17 at device 4.0 on pci6
siis0: [ITHREAD]
siisch0: <SIIS channel> at channel 0 on siis0
siisch0: [ITHREAD]
siisch1: <SIIS channel> at channel 1 on siis0
siisch1: [ITHREAD]
siisch2: <SIIS channel> at channel 2 on siis0
siisch2: [ITHREAD]
siisch3: <SIIS channel> at channel 3 on siis0
siisch3: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 6.0 on pci0
pci4: <ACPI PCI bus> on pcib4
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port
0xc800-0xc8ff mem 0xfbdff000-0xfbdfffff irq 18 at device 0.0 on pci4
re0: Using 1 MSI messages
re0: Chip rev. 0x38000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
re0: Ethernet address: e0:cb:4e:42:f0:ff
re0: [FILTER]
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 11.0 on pci0
pci2: <ACPI PCI bus> on pcib5
pcib6: <PCI-PCI bridge> irq 19 at device 0.0 on pci2
pci3: <PCI bus> on pcib6
siis1: <SiI3124 SATA controller> port 0xbc00-0xbc0f mem
0xfbcffc00-0xfbcffc7f,0xfbcf0000-0xfbcf7fff irq 19 at device 4.0 on pci3
siis1: [ITHREAD]
siisch4: <SIIS channel> at channel 0 on siis1
siisch4: [ITHREAD]
siisch5: <SIIS channel> at channel 1 on siis1
siisch5: [ITHREAD]
siisch6: <SIIS channel> at channel 2 on siis1
siisch6: [ITHREAD]
siisch7: <SIIS channel> at channel 3 on siis1
siisch7: [ITHREAD]
ahci0: <ATI IXP700 AHCI SATA controller> port
0x9000-0x9007,0x8000-0x8003,0x7000-0x7007,0x6000-0x6003,0x5000-0x500f mem
0xfb6fe400-0xfb6fe7ff irq 22 at device 17.0 on pci0
ahci0: [ITHREAD]
ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich3: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xfb6fa000-0xfb6fafff irq 16 at
device 18.0 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
ohci1: <OHCI (generic) USB controller> mem 0xfb6fb000-0xfb6fbfff irq 16 at
device 18.1 on pci0
ohci1: [ITHREAD]
usbus1: <OHCI (generic) USB controller> on ohci1
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfb6fe800-0xfb6fe8ff irq
17 at device 18.2 on pci0
ehci0: [ITHREAD]
ehci0: AMD SB600/700 quirk applied
usbus2: EHCI version 1.0
usbus2: <EHCI (generic) USB 2.0 controller> on ehci0
ohci2: <OHCI (generic) USB controller> mem 0xfb6fc000-0xfb6fcfff irq 18 at
device 19.0 on pci0
ohci2: [ITHREAD]
usbus3: <OHCI (generic) USB controller> on ohci2
ohci3: <OHCI (generic) USB controller> mem 0xfb6fd000-0xfb6fdfff irq 18 at
device 19.1 on pci0
ohci3: [ITHREAD]
usbus4: <OHCI (generic) USB controller> on ohci3
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfb6fec00-0xfb6fecff irq
19 at device 19.2 on pci0
ehci1: [ITHREAD]
ehci1: AMD SB600/700 quirk applied
usbus5: EHCI version 1.0
usbus5: <EHCI (generic) USB 2.0 controller> on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
atapci0: <ATI IXP700/800 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib7: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci1: <ACPI PCI bus> on pcib7
ahc0: <Adaptec 2944 Ultra SCSI adapter> port 0xa800-0xa8ff mem
0xfb7df000-0xfb7dffff irq 20 at device 5.0 on pci1
ahc0: [ITHREAD]
aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
vgapci0: <VGA-compatible display> mem
0xfb800000-0xfbbfffff,0xfb7f0000-0xfb7fffff irq 21 at device 6.0 on pci1
ohci4: <OHCI (generic) USB controller> mem 0xfb6ff000-0xfb6fffff irq 18 at
device 20.5 on pci0
ohci4: [ITHREAD]
usbus6: <OHCI (generic) USB controller> on ohci4
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
on acpi0
fdc0: [FILTER]
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xc87ff,0xc8800-0xc97ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: cannot reserve I/O port range
acpi_throttle0: <ACPI CPU Throttling> on cpu0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
(noperiph:siisch0:0:-1:-1): rescan already queued
(noperiph:siisch2:0:-1:-1): rescan already queued
(noperiph:siisch3:0:-1:-1): rescan already queued
(noperiph:siisch4:0:-1:-1): rescan already queued
(noperiph:siisch5:0:-1:-1): rescan already queued
(noperiph:siisch6:0:-1:-1): rescan already queued
(noperiph:siisch7:0:-1:-1): rescan already queued
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
usbus6: 12Mbps Full Speed USB v1.0
ugen0.1: <ATI> at usbus0
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <ATI> at usbus1
uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <ATI> at usbus2
uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen3.1: <ATI> at usbus3
uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <ATI> at usbus4
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <ATI> at usbus5
uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen6.1: <ATI> at usbus6
uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub6: 2 ports with 2 removable, self powered
uhub0: 3 ports with 3 removable, self powered
uhub1: 3 ports with 3 removable, self powered
uhub3: 3 ports with 3 removable, self powered
uhub4: 3 ports with 3 removable, self powered
uhub2: 6 ports with 6 removable, self powered
uhub5: 6 ports with 6 removable, self powered
ada0 at siisch0 bus 0 scbus0 target 0 lun 0
ada0: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1 at siisch2 bus 0 scbus2 target 0 lun 0
ada1: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at siisch3 bus 0 scbus3 target 0 lun 0
ada2: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at siisch4 bus 0 scbus4 target 0 lun 0
ada3: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4 at siisch5 bus 0 scbus5 target 0 lun 0
ada4: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at siisch6 bus 0 scbus6 target 0 lun 0
ada5: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada6 at siisch7 bus 0 scbus7 target 0 lun 0
ada6: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada7 at ahcich0 bus 0 scbus8 target 0 lun 0
ada7: <ST380815AS 4.AAB> ATA-7 SATA 2.x device
ada7: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada7: Command Queueing enabled
ada7: 76319MB (156301488 512 byte sectors: 16H 63S/T 16383C)
ada8 at ahcich2 bus 0 scbus10 target 0 lun 0
ada8: <WDC WD1600AAJS-75M0A0 02.03E02> ATA-8 SATA 2.x device
ada8: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada8: Command Queueing enabled
ada8: 152587MB (312500000 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #3 Launched!
cd0 at ahcich1 bus 0 scbus9 target 0 lun 0SMP: AP CPU #1 Launched!
cd0:
<TSSTcorp CDDVDW SH-S223C SB01> Removable CD-ROM SCSI-0 device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO
8192bytes)SMP: AP CPU #2 Launched!
cd0: Attempt to query device size failed: NOT READY, Medium not present -
tray closed
GEOM_MIRROR: Device mirror/gm0 launched (1/2).
GEOM_MIRROR: Device gm0: rebuilding provider ada7.
GEOM: mirror/gm0s1: geometry does not match label (16h,63s != 255h,63s).
Trying to mount root from ufs:/dev/mirror/gm0s1a
WARNING: / was not properly dismounted
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is
present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 4
ZFS storage pool version 15
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
--
Dan Langille -- http://langille.org/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a263c3beaeb0fa3acd82650775e31ee3.squirrel>