From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 08:17:45 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2358B106564A for ; Fri, 18 Nov 2011 08:17:45 +0000 (UTC) (envelope-from bane.ivosev@pmf.uns.ac.rs) Received: from mail.pmf.uns.ac.rs (mail.pmf.uns.ac.rs [147.91.177.13]) by mx1.freebsd.org (Postfix) with ESMTP id AE28F8FC14 for ; Fri, 18 Nov 2011 08:17:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.pmf.uns.ac.rs (Postfix) with ESMTP id BB8821AF41A; Fri, 18 Nov 2011 09:17:43 +0100 (CET) X-Virus-Scanned: amavisd-new at pmf.uns.ac.rs Received: from mail.pmf.uns.ac.rs ([127.0.0.1]) by localhost (mail.pmf.uns.ac.rs [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1vioASm-FTvj; Fri, 18 Nov 2011 09:17:13 +0100 (CET) Received: from [10.0.4.20] (unknown [10.0.4.20]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: bane.ivosev@pmf.uns.ac.rs) by mail.pmf.uns.ac.rs (Postfix) with ESMTPSA id CBB791AF46D; Fri, 18 Nov 2011 09:17:13 +0100 (CET) Message-ID: <4EC61489.8080304@pmf.uns.ac.rs> Date: Fri, 18 Nov 2011 09:17:13 +0100 From: Bane Ivosev Organization: PMF DMI User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 MIME-Version: 1.0 To: Daryl Sayers , freebsd-stable@freebsd.org References: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> In-Reply-To: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> X-Enigmail-Version: 1.3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: Re: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 08:17:45 -0000 and if you use zfs also try this zfs set sync=disabled On 11/18/11 04:10, Daryl Sayers wrote: > Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) > > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) > > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount. > > > A copy of dmesg: > ================ > > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 > root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 > Features=0xbfebfbff > Features2=0xe3fd > AMD Features=0x20100000 > AMD Features2=0x1 > TSC: P-state invariant > real memory = 4294967296 (4096 MB) > avail memory = 3141234688 (2995 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 1 package(s) x 2 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0 irqs 0-23 on motherboard > kbd1 at kbdmux0 > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, bff00000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > cpu0: on acpi0 > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354) > cpu1: on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: irq 16 at device 1.0 on pci0 > pci1: on pcib1 > mpt0: port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1 > mpt0: [ITHREAD] > mpt0: MPI Version=1.5.18.0 > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > mpt0: 0 Active Volumes (2 Max) > mpt0: 0 Hidden Drive Members (14 Max) > uhci0: port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0 > uhci0: [ITHREAD] > uhci0: LegSup = 0x2f00 > usbus0: on uhci0 > uhci1: port 0xe000-0xe01f irq 17 at device 26.1 on pci0 > uhci1: [ITHREAD] > uhci1: LegSup = 0x2f00 > usbus1: on uhci1 > ehci0: mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 > ehci0: [ITHREAD] > usbus2: EHCI version 1.0 > usbus2: on ehci0 > pci0: at device 27.0 (no driver attached) > pcib2: irq 16 at device 28.0 on pci0 > pci5: on pcib2 > atapci0: port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5 > atapci0: [ITHREAD] > ata2: on atapci0 > ata2: [ITHREAD] > ata3: on atapci0 > ata3: [ITHREAD] > pcib3: irq 17 at device 28.1 on pci0 > pci4: on pcib3 > em0: port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4 > em0: Using an MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:1b:21:04:ac:11 > pcib4: irq 19 at device 28.3 on pci0 > pci3: on pcib4 > age0: mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 > age0: 1280 Tx FIFO, 2364 Rx FIFO > age0: Using 1 MSI messages. > miibus0: on age0 > atphy0: PHY 0 on miibus0 > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto > age0: Ethernet address: 00:1a:92:d2:de:cc > age0: [FILTER] > pcib5: irq 16 at device 28.4 on pci0 > pci2: on pcib5 > atapci1: port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 > atapci1: [ITHREAD] > atapci2: on atapci1 > atapci2: [ITHREAD] > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported > ata4: on atapci2 > ata4: [ITHREAD] > ata5: on atapci2 > ata5: [ITHREAD] > ata6: on atapci1 > ata6: [ITHREAD] > uhci2: port 0xd480-0xd49f irq 23 at device 29.0 on pci0 > uhci2: [ITHREAD] > uhci2: LegSup = 0x2f00 > usbus3: on uhci2 > uhci3: port 0xd800-0xd81f irq 19 at device 29.1 on pci0 > uhci3: [ITHREAD] > uhci3: LegSup = 0x2f00 > usbus4: on uhci3 > uhci4: port 0xd880-0xd89f irq 18 at device 29.2 on pci0 > uhci4: [ITHREAD] > uhci4: LegSup = 0x2f00 > usbus5: on uhci4 > ehci1: mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 > ehci1: [ITHREAD] > usbus6: EHCI version 1.0 > usbus6: on ehci1 > pcib6: at device 30.0 on pci0 > pci6: on pcib6 > vgapci0: mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6 > fwohci0: port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 > fwohci0: [ITHREAD] > fwohci0: OHCI version 1.10 (ROM=1) > fwohci0: No. of Isochronous channels is 4. > fwohci0: EUI64 00:11:d8:00:01:39:52:d3 > fwohci0: Phy 1394a available S400, 2 ports. > fwohci0: Link S400, max_rec 2048 bytes. > firewire0: on fwohci0 > dcons_crom0: on firewire0 > dcons_crom0: bus_addr 0x1494000 > fwe0: on firewire0 > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 > fwe0: Ethernet address: 02:11:d8:39:52:d3 > fwip0: on firewire0 > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048 > fwohci0: Initiate bus reset > fwohci0: fwohci_intr_core: BUS reset > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci3: port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0 > atapci3: [ITHREAD] > ata7: on atapci3 > ata7: [ITHREAD] > ata8: on atapci3 > ata8: [ITHREAD] > pci0: at device 31.3 (no driver attached) > atapci4: port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0 > atapci4: [ITHREAD] > ata9: on atapci4 > ata9: [ITHREAD] > ata10: on atapci4 > ata10: [ITHREAD] > acpi_button0: on acpi0 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart0: [FILTER] > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > fdc0: [FILTER] > acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppc0: [ITHREAD] > ppbus0: on ppc0 > plip0: on ppbus0 > plip0: [ITHREAD] > lpt0: on ppbus0 > lpt0: [ITHREAD] > lpt0: Interrupt-driven port > ppi0: on ppbus0 > pmtimer0 on isa0 > orm0: at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 > ata0: [ITHREAD] > ata1 at port 0x170-0x177,0x376 irq 15 on isa0 > ata1: [ITHREAD] > atkbdc0: at port 0x60,0x64 on isa0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > est0: on cpu0 > p4tcc0: on cpu0 > est1: on cpu1 > p4tcc1: on cpu1 > Timecounters tick every 1.000 msec > IPsec: Initialized Security Association Processing. > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) > firewire0: bus manager 0 > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 480Mbps High Speed USB v2.0 > usbus3: 12Mbps Full Speed USB v1.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 12Mbps Full Speed USB v1.0 > usbus6: 480Mbps High Speed USB v2.0 > ugen0.1: at usbus0 > uhub0: on usbus0 > ugen1.1: at usbus1 > uhub1: on usbus1 > ugen2.1: at usbus2 > uhub2: on usbus2 > ugen3.1: at usbus3 > uhub3: on usbus3 > ugen4.1: at usbus4 > uhub4: on usbus4 > ugen5.1: at usbus5 > uhub5: on usbus5 > ugen6.1: at usbus6 > uhub6: on usbus6 > acd0: DVDR at ata7-slave UDMA100 SATA 1.5Gb/s > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub3: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > uhub5: 2 ports with 2 removable, self powered > uhub2: 4 ports with 4 removable, self powered > uhub6: 6 ports with 6 removable, self powered > da1 at mpt0 bus 0 scbus0 target 1 lun 0 > da1: Fixed Direct Access SCSI-5 device > da1: 300.000MB/s transfers > da1: Command Queueing enabled > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da2 at mpt0 bus 0 scbus0 target 2 lun 0 > da2: Fixed Direct Access SCSI-5 device > da2: 300.000MB/s transfers > da2: Command Queueing enabled > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da0 at mpt0 bus 0 scbus0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 300.000MB/s transfers > da0: Command Queueing enabled > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) > SMP: AP CPU #1 Launched! > ugen1.2: at usbus1 > ugen3.2: at usbus3 > ukbd1: on usbus1 > ums0: on usbus3 > ums0: 3 buttons and [XYZ] coordinates ID=0 > kbd2 at ukbd1 > uhid0: on usbus1 > Trying to mount root from ufs:/dev/da0s1a > >