Date: Fri, 18 Nov 2011 09:17:13 +0100 From: Bane Ivosev <bane.ivosev@pmf.uns.ac.rs> To: Daryl Sayers <daryl@ci.com.au>, freebsd-stable@freebsd.org Subject: Re: Low nfs write throughput Message-ID: <4EC61489.8080304@pmf.uns.ac.rs> In-Reply-To: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> References: <201111180310.pAI3ARbZ075115@mippet.ci.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
and if you use zfs also try this zfs set sync=disabled On 11/18/11 04:10, Daryl Sayers wrote: > Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) > > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) > > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount. > > > A copy of dmesg: > ================ > > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 > root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> > AMD Features=0x20100000<NX,LM> > AMD Features2=0x1<LAHF> > TSC: P-state invariant > real memory = 4294967296 (4096 MB) > avail memory = 3141234688 (2995 MB) > ACPI APIC Table: <MSTEST OEMAPIC > > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 1 package(s) x 2 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0 <Version 2.0> irqs 0-23 on motherboard > kbd1 at kbdmux0 > cryptosoft0: <software crypto> on motherboard > acpi0: <MSTEST TESTONLY> on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, bff00000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > cpu0: <ACPI CPU> on acpi0 > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354) > cpu1: <ACPI CPU> on acpi0 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 > pci1: <ACPI PCI bus> on pcib1 > mpt0: <LSILogic SAS/SATA Adapter> port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1 > mpt0: [ITHREAD] > mpt0: MPI Version=1.5.18.0 > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > mpt0: 0 Active Volumes (2 Max) > mpt0: 0 Hidden Drive Members (14 Max) > uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0 > uhci0: [ITHREAD] > uhci0: LegSup = 0x2f00 > usbus0: <Intel 82801H (ICH8) USB controller USB-D> on uhci0 > uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0xe000-0xe01f irq 17 at device 26.1 on pci0 > uhci1: [ITHREAD] > uhci1: LegSup = 0x2f00 > usbus1: <Intel 82801H (ICH8) USB controller USB-E> on uhci1 > ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 > ehci0: [ITHREAD] > usbus2: EHCI version 1.0 > usbus2: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> on ehci0 > pci0: <multimedia, HDA> at device 27.0 (no driver attached) > pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 > pci5: <ACPI PCI bus> on pcib2 > atapci0: <SiI 3132 SATA300 controller> port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5 > atapci0: [ITHREAD] > ata2: <ATA channel 0> on atapci0 > ata2: [ITHREAD] > ata3: <ATA channel 1> on atapci0 > ata3: [ITHREAD] > pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0 > pci4: <ACPI PCI bus> on pcib3 > em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4 > em0: Using an MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:1b:21:04:ac:11 > pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 > pci3: <ACPI PCI bus> on pcib4 > age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 > age0: 1280 Tx FIFO, 2364 Rx FIFO > age0: Using 1 MSI messages. > miibus0: <MII bus> on age0 > atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0 > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto > age0: Ethernet address: 00:1a:92:d2:de:cc > age0: [FILTER] > pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 > pci2: <ACPI PCI bus> on pcib5 > atapci1: <JMicron JMB363 UDMA133 controller> port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 > atapci1: [ITHREAD] > atapci2: <AHCI SATA controller> on atapci1 > atapci2: [ITHREAD] > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported > ata4: <ATA channel 0> on atapci2 > ata4: [ITHREAD] > ata5: <ATA channel 1> on atapci2 > ata5: [ITHREAD] > ata6: <ATA channel 0> on atapci1 > ata6: [ITHREAD] > uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0xd480-0xd49f irq 23 at device 29.0 on pci0 > uhci2: [ITHREAD] > uhci2: LegSup = 0x2f00 > usbus3: <Intel 82801H (ICH8) USB controller USB-A> on uhci2 > uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0xd800-0xd81f irq 19 at device 29.1 on pci0 > uhci3: [ITHREAD] > uhci3: LegSup = 0x2f00 > usbus4: <Intel 82801H (ICH8) USB controller USB-B> on uhci3 > uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0xd880-0xd89f irq 18 at device 29.2 on pci0 > uhci4: [ITHREAD] > uhci4: LegSup = 0x2f00 > usbus5: <Intel 82801H (ICH8) USB controller USB-C> on uhci4 > ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 > ehci1: [ITHREAD] > usbus6: EHCI version 1.0 > usbus6: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> on ehci1 > pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0 > pci6: <ACPI PCI bus> on pcib6 > vgapci0: <VGA-compatible display> mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6 > fwohci0: <VIA Fire II (VT6306)> port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 > fwohci0: [ITHREAD] > fwohci0: OHCI version 1.10 (ROM=1) > fwohci0: No. of Isochronous channels is 4. > fwohci0: EUI64 00:11:d8:00:01:39:52:d3 > fwohci0: Phy 1394a available S400, 2 ports. > fwohci0: Link S400, max_rec 2048 bytes. > firewire0: <IEEE1394(FireWire) bus> on fwohci0 > dcons_crom0: <dcons configuration ROM> on firewire0 > dcons_crom0: bus_addr 0x1494000 > fwe0: <Ethernet over FireWire> on firewire0 > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 > fwe0: Ethernet address: 02:11:d8:39:52:d3 > fwip0: <IP over FireWire> on firewire0 > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048 > fwohci0: Initiate bus reset > fwohci0: fwohci_intr_core: BUS reset > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode > isab0: <PCI-ISA bridge> at device 31.0 on pci0 > isa0: <ISA bus> on isab0 > atapci3: <Intel ICH8 SATA300 controller> port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0 > atapci3: [ITHREAD] > ata7: <ATA channel 0> on atapci3 > ata7: [ITHREAD] > ata8: <ATA channel 1> on atapci3 > ata8: [ITHREAD] > pci0: <serial bus, SMBus> at device 31.3 (no driver attached) > atapci4: <Intel ICH8 SATA300 controller> port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0 > atapci4: [ITHREAD] > ata9: <ATA channel 0> on atapci4 > ata9: [ITHREAD] > ata10: <ATA channel 1> on atapci4 > ata10: [ITHREAD] > acpi_button0: <Power Button> on acpi0 > atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart0: [FILTER] > fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > fdc0: [FILTER] > acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppc0: [ITHREAD] > ppbus0: <Parallel port bus> on ppc0 > plip0: <PLIP network interface> on ppbus0 > plip0: [ITHREAD] > lpt0: <Printer> on ppbus0 > lpt0: [ITHREAD] > lpt0: Interrupt-driven port > ppi0: <Parallel I/O> on ppbus0 > pmtimer0 on isa0 > orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 > sc0: <System console> at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 > ata0: [ITHREAD] > ata1 at port 0x170-0x177,0x376 irq 15 on isa0 > ata1: [ITHREAD] > atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 > atkbd0: <AT Keyboard> irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > est0: <Enhanced SpeedStep Frequency Control> on cpu0 > p4tcc0: <CPU Frequency Thermal Control> on cpu0 > est1: <Enhanced SpeedStep Frequency Control> on cpu1 > p4tcc1: <CPU Frequency Thermal Control> on cpu1 > Timecounters tick every 1.000 msec > IPsec: Initialized Security Association Processing. > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) > firewire0: bus manager 0 > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 480Mbps High Speed USB v2.0 > usbus3: 12Mbps Full Speed USB v1.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 12Mbps Full Speed USB v1.0 > usbus6: 480Mbps High Speed USB v2.0 > ugen0.1: <Intel> at usbus0 > uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 > ugen1.1: <Intel> at usbus1 > uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 > ugen2.1: <Intel> at usbus2 > uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 > ugen3.1: <Intel> at usbus3 > uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 > ugen4.1: <Intel> at usbus4 > uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 > ugen5.1: <Intel> at usbus5 > uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 > ugen6.1: <Intel> at usbus6 > uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6 > acd0: DVDR <HL-DT-ST DVDRAM GH22NS50/TN02> at ata7-slave UDMA100 SATA 1.5Gb/s > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub3: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > uhub5: 2 ports with 2 removable, self powered > uhub2: 4 ports with 4 removable, self powered > uhub6: 6 ports with 6 removable, self powered > da1 at mpt0 bus 0 scbus0 target 1 lun 0 > da1: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device > da1: 300.000MB/s transfers > da1: Command Queueing enabled > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da2 at mpt0 bus 0 scbus0 target 2 lun 0 > da2: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device > da2: 300.000MB/s transfers > da2: Command Queueing enabled > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da0 at mpt0 bus 0 scbus0 target 0 lun 0 > da0: <FUJITSU MBC2073RC 0104> Fixed Direct Access SCSI-5 device > da0: 300.000MB/s transfers > da0: Command Queueing enabled > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) > SMP: AP CPU #1 Launched! > ugen1.2: <NOVATEK> at usbus1 > ugen3.2: <Microsoft> at usbus3 > ukbd1: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1 > ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev 1.10/1.99, addr 2> on usbus3 > ums0: 3 buttons and [XYZ] coordinates ID=0 > kbd2 at ukbd1 > uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1 > Trying to mount root from ufs:/dev/da0s1a > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EC61489.8080304>