Date: Fri, 18 Nov 2011 14:10:27 +1100 (EST) From: Daryl Sayers <daryl@ci.com.au> To: freebsd-stable@freebsd.org Subject: Low nfs write throughput Message-ID: <201111180310.pAI3ARbZ075115@mippet.ci.com.au>
next in thread | raw e-mail | index | archive | help
Can anyone suggest why I am getting poor write performance from my nfs setup.
I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards,
4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with
onboard Gb network cards connected to an idle network. The results below show
that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It
improves if I use async but a smbfs mount still beats it. I am using the same
file, source and destinations for all tests. I have tried alternate Network
cards with no resulting benefit.
oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2
1950511+1 records in
1950511+1 records out
998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec)
1950477+74 records in
1950511+1 records out
998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s)
oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt
oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
7619+1 records in
7619+1 records out
998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s)
oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt
oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
7619+1 records in
7619+1 records out
998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s)
oguido# mount -t smbfs //gemini/ufs /mnt
oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
7619+1 records in
7619+1 records out
998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s)
Looking at a systat -v on the destination I see that the nfs test does not
exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t.
For the record I get reads of 22Mb/s without and 77Mb/s with async turned on
for the nfs mount.
A copy of dmesg:
================
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011
root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 4294967296 (4096 MB)
avail memory = 3141234688 (2995 MB)
ACPI APIC Table: <MSTEST OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <MSTEST TESTONLY> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, bff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354)
cpu1: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mpt0: <LSILogic SAS/SATA Adapter> port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.18.0
mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt0: 0 Active Volumes (2 Max)
mpt0: 0 Hidden Drive Members (14 Max)
uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801H (ICH8) USB controller USB-D> on uhci0
uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0xe000-0xe01f irq 17 at device 26.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801H (ICH8) USB controller USB-E> on uhci1
ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus2: EHCI version 1.0
usbus2: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> on ehci0
pci0: <multimedia, HDA> at device 27.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci5: <ACPI PCI bus> on pcib2
atapci0: <SiI 3132 SATA300 controller> port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci4: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4
em0: Using an MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:1b:21:04:ac:11
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci3: <ACPI PCI bus> on pcib4
age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3
age0: 1280 Tx FIFO, 2364 Rx FIFO
age0: Using 1 MSI messages.
miibus0: <MII bus> on age0
atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0
atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
age0: Ethernet address: 00:1a:92:d2:de:cc
age0: [FILTER]
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci2: <ACPI PCI bus> on pcib5
atapci1: <JMicron JMB363 UDMA133 controller> port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2
atapci1: [ITHREAD]
atapci2: <AHCI SATA controller> on atapci1
atapci2: [ITHREAD]
atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported
ata4: <ATA channel 0> on atapci2
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci2
ata5: [ITHREAD]
ata6: <ATA channel 0> on atapci1
ata6: [ITHREAD]
uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0xd480-0xd49f irq 23 at device 29.0 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus3: <Intel 82801H (ICH8) USB controller USB-A> on uhci2
uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0xd800-0xd81f irq 19 at device 29.1 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x2f00
usbus4: <Intel 82801H (ICH8) USB controller USB-B> on uhci3
uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0xd880-0xd89f irq 18 at device 29.2 on pci0
uhci4: [ITHREAD]
uhci4: LegSup = 0x2f00
usbus5: <Intel 82801H (ICH8) USB controller USB-C> on uhci4
ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus6: EHCI version 1.0
usbus6: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> on ehci1
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6
fwohci0: <VIA Fire II (VT6306)> port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:11:d8:00:01:39:52:d3
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x1494000
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3
fwe0: Ethernet address: 02:11:d8:39:52:d3
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci3: <Intel ICH8 SATA300 controller> port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0
atapci3: [ITHREAD]
ata7: <ATA channel 0> on atapci3
ata7: [ITHREAD]
ata8: <ATA channel 1> on atapci3
ata8: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci4: <Intel ICH8 SATA300 controller> port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0
atapci4: [ITHREAD]
ata9: <ATA channel 0> on atapci4
ata9: [ITHREAD]
ata10: <ATA channel 1> on atapci4
ata10: [ITHREAD]
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata0: [ITHREAD]
ata1 at port 0x170-0x177,0x376 irq 15 on isa0
ata1: [ITHREAD]
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
acd0: DVDR <HL-DT-ST DVDRAM GH22NS50/TN02> at ata7-slave UDMA100 SATA 1.5Gb/s
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub2: 4 ports with 4 removable, self powered
uhub6: 6 ports with 6 removable, self powered
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
da2 at mpt0 bus 0 scbus0 target 2 lun 0
da2: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <FUJITSU MBC2073RC 0104> Fixed Direct Access SCSI-5 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C)
SMP: AP CPU #1 Launched!
ugen1.2: <NOVATEK> at usbus1
ugen3.2: <Microsoft> at usbus3
ukbd1: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1
ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev 1.10/1.99, addr 2> on usbus3
ums0: 3 buttons and [XYZ] coordinates ID=0
kbd2 at ukbd1
uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1
Trying to mount root from ufs:/dev/da0s1a
--
Daryl Sayers Direct: +612 95525510
Corinthian Engineering Office: +612 95525500
Suite 54, Jones Bay Wharf Fax: +612 95525549
26-32 Pirrama Rd email: daryl@ci.com.au
Pyrmont NSW 2009 Australia www: http://www.ci.com.au
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201111180310.pAI3ARbZ075115>