From owner-freebsd-current@FreeBSD.ORG Wed Nov 11 18:50:43 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAB6A1065670 for ; Wed, 11 Nov 2009 18:50:43 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-pw0-f44.google.com (mail-pw0-f44.google.com [209.85.160.44]) by mx1.freebsd.org (Postfix) with ESMTP id B47A28FC15 for ; Wed, 11 Nov 2009 18:50:43 +0000 (UTC) Received: by pwj15 with SMTP id 15so941705pwj.3 for ; Wed, 11 Nov 2009 10:50:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=FuRrcyFX4bXhwxui2sPSrw6v+TNMDjnmDouX7ZIiL1E=; b=jatysFtn7/aO48qzBy0+mGij0r1UApfnUHMJyUNGLclCfnat2adbqbDoBbtEhCRegW YkML8trGeCTPWcSGy62zoE7oaDWKbCSsvTEKaz1Xc/CKqUC3TmQ7++BaSxiwgqijw9zk cx1DGwtEZpAUZjWEWwerz1KbcTPhncVfYPQDk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=fQo+ufSagEIzr6b2duDnqzw087UTQdy1XjxHmfzEhxUn5qtBC24HL9LpvV9wmSjcRX DdunSpL2XdGtTXH9+J/LciCSbH8oyxp+RFxTZClQF3cF8z8l+E7XGjHtK7OgQL8Mk6Ee wiipPHnucIKUNBFhAhjQeoCsqaSUVw/JCLJ8Q= MIME-Version: 1.0 Received: by 10.142.122.6 with SMTP id u6mr201736wfc.95.1257965443262; Wed, 11 Nov 2009 10:50:43 -0800 (PST) Date: Wed, 11 Nov 2009 12:50:43 -0600 Message-ID: <11167f520911111050j36dd94far667c81e6f5c18e69@mail.gmail.com> From: "Sam Fourman Jr." To: FreeBSD Current Content-Type: text/plain; charset=ISO-8859-1 Subject: Help ZFS FreeBSD 8.0 RC2 Write performance issue X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Nov 2009 18:50:43 -0000 Hello list, I am running FreeBSD 8.0RC2 and I dont understand why my ZFS/NFS is acting weird on writes. I get ~150mbit writes idk if this is good or not? but it paused for a few seconds every once and awhile. I have cut and pasted a bunch of stuff below. does anyone have any insight? this is a recent machine with a AMD processor and 4GB of memory. the switch in the middle is a Dell 5224 gigabit switch if that maters. and the Clients performing the writes are also FreeBSD 8RC2 I will say one thing... While I am performing writes.. if I was to do a Read from a different client I only can get like ~3mbit reads. if I do nothing but a read on a single client I can get ~300mbit reads any help would be appreciated Sam Fourman Jr. Fourman Networks. # zpool status pool: Network state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM Network ONLINE 0 0 0 raidz1 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 errors: No known data errors # ifconfig re0: flags=8843 metric 0 mtu 1500 options=389b ether 90:e6:ba:10:0e:2d inet 192.168.12.10 netmask 0xffffff00 broadcast 192.168.12.255 inet 192.168.12.11 netmask 0xffffff00 broadcast 192.168.12.255 inet 192.168.12.12 netmask 0xffffff00 broadcast 192.168.12.255 inet 192.168.12.13 netmask 0xffffff00 broadcast 192.168.12.255 media: Ethernet autoselect (1000baseT ) status: active plip0: flags=8810 metric 0 mtu 1500 lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 # ifstat -b -i re0 re0 Kbps in Kbps out 3.37 5.18 3.37 5.05 3.37 5.05 42.21 48.22 5.89 5.05 3.37 5.05 3.37 5.05 41.68 48.20 110.44 101.74 3.37 5.05 3.37 5.05 5.03 6.37 10.83 12.84 62.05 62.50 3.37 5.05 3.37 5.05 5.21 6.47 3.37 5.05 56521.85 2052.38 133949.5 4723.58 141345.1 4968.34 154383.0 5433.63 154244.1 5423.14 121558.2 4306.39 155455.9 5475.89 142424.3 5015.22 159326.7 5609.79 152141.6 5281.99 146702.9 7199.11 151088.5 5312.14 151621.7 5346.30 112848.5 3939.65 4.80 5.05 46.20 49.68 5.21 6.47 3.37 5.05 3.37 5.05 5.21 6.47 44.72 48.20 132888.9 4652.04 153707.7 5402.98 152234.6 5343.49 136510.0 4814.00 124268.0 4437.97 158612.9 5590.24 156464.4 5514.69 142010.5 5010.10 148333.6 5211.41 re0 Kbps in Kbps out 155707.6 5525.03 156187.8 5504.36 121838.5 4282.46 16874.25 621.48 5.21 6.47 46.17 49.61 3.37 5.05 5.21 6.47 3.37 5.05 30120.71 1058.03 156991.7 5639.63 147261.1 5311.55 131024.8 4725.30 151883.2 5353.18 155146.7 5456.85 161816.2 5727.37 144066.7 5138.94 155767.4 5507.38 142959.7 5043.90 152714.3 5428.75 110144.8 3987.22 3.37 5.05 3.37 5.05 5.03 6.37 3.37 5.05 64.62 68.86 11302.01 367.31 154841.7 5489.48 160238.3 5635.34 151150.4 5323.96 160726.1 5696.35 152421.8 5381.29 161405.3 5751.44 152095.4 5349.11 116161.9 4104.15 158146.7 5573.70 147790.0 5247.66 12551.27 462.41 3.37 5.05 3.37 5.05 3.37 5.05 2160.03 2117.73 11297.74 376.93 158567.7 5610.60 147388.6 5186.73 151555.5 5349.12 157579.2 5587.67 149001.2 5268.48 re0 Kbps in Kbps out 154172.6 5507.73 162795.7 5753.79 151039.5 5315.01 153744.4 5478.71 88269.23 3137.44 3.37 5.05 3.37 5.05 25.07 24.70 44.72 48.20 3.37 5.05 107780.3 3813.35 151123.3 5326.76 153103.2 5440.45 149461.4 5318.25 154987.6 5465.26 147789.8 5223.30 140811.6 4981.23 158819.1 5658.08 156550.1 5542.69 117199.0 4096.97 3.37 5.05 3.37 5.05 3.37 5.05 44.70 48.17 3.37 5.05 116172.3 4084.82 159684.2 5595.67 155388.3 5470.82 153778.8 5471.12 160333.3 5650.18 151555.5 5346.59 153912.8 5427.27 146753.1 5234.99 151624.0 5337.10 41923.68 1478.39 3.37 5.05 3.37 5.05 66.24 67.47 3.37 5.05 61305.44 2143.38 157087.6 5549.02 155647.7 5553.67 157510.1 5605.24 146786.6 5180.34 135555.9 4780.34 158633.3 5578.98 157735.7 5538.46 151445.5 5390.60 re0 Kbps in Kbps out 88655.49 3138.99 5.21 7.61 3.37 5.05 3.37 5.05 314.11 309.87 35855.92 3019.12 126837.6 4492.57 136117.2 4806.87 147924.3 5229.35 150553.5 5315.17 158044.2 5609.08 159151.6 5627.26 147855.5 5178.02 156533.7 5495.67 125278.2 4390.51 44.72 48.20 3.37 5.05 3.37 5.05 3.37 5.05 12400.47 420.22 141994.2 5045.35 157535.1 5537.08 154312.9 5412.69 159218.2 5633.57 152698.8 5365.41 155403.4 5563.36 150104.5 5317.83 156397.1 5517.41 86013.25 3048.44 3.37 5.05 44.72 48.20 3.37 5.05 3.37 5.05 47334.93 1647.06 149320.0 5347.42 149435.6 5308.92 156021.1 5474.01 152563.6 5375.73 155523.0 5473.40 152630.9 5385.65 152203.3 5394.63 142484.9 5030.69 56701.45 2052.13 3.37 5.05 3.37 5.05 46.17 49.62 5.03 6.37 94137.63 3297.45 re0 Kbps in Kbps out 164023.3 5785.23 147958.4 5279.17 152165.2 5356.31 147259.5 5208.91 155024.8 5456.01 158034.7 5552.35 147619.9 5249.18 135962.4 4821.04 3.37 5.05 3.37 5.05 3.37 5.05 44.54 48.00 24643.59 808.53 155887.8 5499.69 152622.6 5374.52 90909.52 3216.71 150315.9 6516.74 149293.3 6100.74 157711.7 5644.70 142586.9 5036.40 146279.9 5149.50 126335.6 4510.90 3.37 5.05 3.37 5.05 3.37 5.05 3.37 5.05 31785.91 1137.64 152113.2 5361.44 149366.4 5262.94 141578.1 4984.64 147796.4 5205.79 154105.2 5549.08 161407.3 5713.83 149983.9 5294.44 145774.1 5148.62 58054.58 2079.92 3.37 5.05 45.64 48.75 3.37 5.05 10.38 41.29 115633.4 4057.17 145172.9 5109.34 160267.6 5680.67 154851.5 5455.03 157873.1 5621.18 150019.6 5295.47 160156.1 5655.63 146290.4 5200.17 re0 Kbps in Kbps out 97345.63 3477.19 3.37 6.19 3.37 5.05 3.37 5.05 46.56 49.61 73127.16 2544.00 150076.1 5293.10 152638.9 5381.02 153774.1 5412.67 155966.5 5522.56 161909.6 5662.65 158143.3 5553.83 150476.0 5284.16 115083.2 4048.69 11872.56 459.17 3.37 5.05 3.37 5.05 5.10 6.69 48749.13 1788.02 160387.1 5730.22 151622.1 5301.40 158541.1 5553.69 163521.1 5719.97 155853.6 5478.11 159261.4 5626.13 ^C # /boot/loader.conf kern.ipc.semaem=32767 kern.ipc.semvmx=65534 kern.ipc.semusz=184 kern.ipc.semume=80 kern.ipc.semopm=200 kern.ipc.semmsl=120 kern.ipc.semmnu=4096 kern.ipc.semmns=8192 kern.ipc.semmni=32767 kern.ipc.semmap=60 /etc/sysctl.conf # Jail Settings security.jail.jailed: 0 security.jail.mount_allowed: 0 security.jail.chflags_allowed: 0 security.jail.allow_raw_sockets: 1 security.jail.enforce_statfs: 2 security.jail.sysvipc_allowed: 1 security.jail.socket_unixiproute_only: 1 security.jail.set_hostname_allowed: 1 # More Shared memory for jails/PostgreSQL kern.ipc.shmall=65536 kern.ipc.shmmax=134217728 kern.ipc.semmap=4096 # dmesg Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC2 #2: Tue Nov 10 21:05:03 CST 2009 sfourman@.PuffyBSD.Com:/usr/obj/usr/src/sys/GENERIC amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) II X2 550 Processor (3100.02-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f42 Stepping = 2 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x37ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3847766016 (3669 MB) ACPI APIC Table: <092809 APIC1459> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ACPI Warning: Optional field Pm2ControlBlock has zero address or length: 0 0/1 20090521 tbfadt-655 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <092809 XSDT1459> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of fec10000, 20 (3) failed acpi0: reservation of fed40000, 5000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, cff00000 (3) failed ACPI HPET table warning: Sequence is non-zero (2) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 acpi_hpet0: HPET never increments, disabling device_attach: acpi_hpet0 attach returned 6 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xd000-0xd0ff mem 0xd0000000-0xdfffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 pci1: at device 5.1 (no driver attached) pcib2: irq 18 at device 10.0 on pci0 pci2: on pcib2 re0: port 0xe800-0xe8ff mem 0xfdbff000-0xfdbfffff,0xfdbf8000-0xfdbfbfff irq 18 at device 0.0 on pci2 re0: Using 1 MSI messages re0: Chip rev. 0x28000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto re0: Ethernet address: 90:e6:ba:10:0e:2d re0: [FILTER] atapci0: port 0xc000-0xc007,0xb000-0xb003,0xa000-0xa007,0x9000-0x9003,0x8000-0x800f mem 0xfe7ffc00-0xfe7fffff irq 22 at device 17.0 on pci0 atapci0: [ITHREAD] atapci0: AHCI v1.10 controller with 4 3Gbps ports, PM supported ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] ata4: on atapci0 ata4: [ITHREAD] ata5: on atapci0 ata5: [ITHREAD] ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 18.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 16 at device 18.1 on pci0 ohci1: [ITHREAD] usbus1: on ohci1 ehci0: mem 0xfe7ff800-0xfe7ff8ff irq 17 at device 18.2 on pci0 ehci0: [ITHREAD] usbus2: EHCI version 1.0 usbus2: on ehci0 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.0 on pci0 ohci2: [ITHREAD] usbus3: on ohci2 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 18 at device 19.1 on pci0 ohci3: [ITHREAD] usbus4: on ohci3 ehci1: mem 0xfe7ff400-0xfe7ff4ff irq 19 at device 19.2 on pci0 ehci1: [ITHREAD] usbus5: EHCI version 1.0 usbus5: on ehci1 pci0: at device 20.0 (no driver attached) atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] ata1: on atapci1 ata1: [ITHREAD] pci0: at device 20.2 (no driver attached) isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 pcib4: at device 5.0 on pci3 pci4: on pcib4 arcmsr0: mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 22 at device 14.0 on pci4 ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.47 2009-06-25 arcmsr0: [ITHREAD] ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 20.5 on pci0 ohci4: [ITHREAD] usbus6: on ohci4 acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] cpu0: on acpi0 acpi_throttle0: on cpu0 hwpstate0: on cpu0 cpu1: on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 orm0: at iomem 0xcf000-0xcffff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 usbus6: 12Mbps Full Speed USB v1.0 ad4: 38146MB at ata2-master SATA150 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 Waiting 5 seconds for SCSI devices to settle uhub6: 2 ports with 2 removable, self powered uhub0: 3 ports with 3 removable, self powered uhub1: 3 ports with 3 removable, self powered uhub3: 3 ports with 3 removable, self powered uhub4: 3 ports with 3 removable, self powered uhub2: 6 ports with 6 removable, self powered uhub5: 6 ports with 6 removable, self powered (probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step da0 at arcmsr0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit) da0: Command Queueing enabled da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da1 at arcmsr0 bus 0 target 0 lun 1 da1: Fixed Direct Access SCSI-5 device da1: 166.666MB/s transfers (83.333MHz, offset 32, 16bit) da1: Command Queueing enabled da1: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da2 at arcmsr0 bus 0 target 0 lun 2 da2: Fixed Direct Access SCSI-5 device da2: 166.666MB/s transfers (83.333MHz, offset 32, 16bit) da2: Command Queueing enabled da2: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da3 at arcmsr0 bus 0 target 0 lun 3 da3: Fixed Direct Access SCSI-5 device da3: 166.666MB/s transfers (83.333MHz, offset 32, 16bit) da3: Command Queueing enabled da3: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da4 at arcmsr0 bus 0 target 0 lun 4 da4: Fixed Direct Access SCSI-5 device da4: 166.666MB/s transfers (83.333MHz, offset 32, 16bit) da4: Command Queueing enabled da4: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da5 at arcmsr0 bus 0 target 0 lun 5 da5: Fixed Direct Access SCSI-5 device da5: 166.666MB/s transfers (83.333MHz, offset 32, 16bit) da5: Command Queueing enabled da5: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/ad4s1a ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 13 ZFS storage pool version 13 re0: link state changed to UP #