From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 17:23:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9138106567A for ; Fri, 18 Nov 2011 17:23:49 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 373EC8FC0A for ; Fri, 18 Nov 2011 17:23:48 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsIEACqNxk6DaFvO/2dsb2JhbAA7CA6Ec6JAgRSCYYFyAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggCBQQBGgIEh2IIpGWSBoEwgimDDYIbgRYEiBeKA4IfkUpV X-IronPort-AV: E=Sophos;i="4.69,533,1315195200"; d="scan'208";a="144557803" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 18 Nov 2011 11:54:43 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 197FEB3F20; Fri, 18 Nov 2011 11:54:43 -0500 (EST) Date: Fri, 18 Nov 2011 11:54:43 -0500 (EST) From: Rick Macklem To: Bane Ivosev Message-ID: <917971298.2052425.1321635283084.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4EC61489.8080304@pmf.uns.ac.rs> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-stable@freebsd.org, Daryl Sayers Subject: Re: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 17:23:49 -0000 Bane Ivosev wrote: > and if you use zfs also try this > > zfs set sync=disabled > I know diddly about zfs, but I believe some others have improved zfs performance for NFS writing by moving the ZIL log to a dedicated device, sometimes an SSD. Apparently (again I'm not knowledgible) you do have to be careful what SSD you use and how full you make it, if you want good write performance on the SSD. I should also note that use of these options (vfs.nfsrv.async=1 and the above for zfs) is risky in the sense that recently written data can be lost when a server crashes/reboots because the NFS clients don't know to hold onto the data and re-write it after a server crash/reboot. rick ps: NFS write performance has been an issue since SUN released their first implementation of it in 1985. The "big" server vendors typically solve the problem with lots of non-volatile RAM in the server boxes. (This solution requires server code that specifically knows how to use this non-volatile RAM. Such code is not in the FreeBSD servers.) > On 11/18/11 04:10, Daryl Sayers wrote: > > Can anyone suggest why I am getting poor write performance from my > > nfs setup. > > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus > > mother boards, > > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives > > with > > onboard Gb network cards connected to an idle network. The results > > below show > > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using > > nfs. It > > improves if I use async but a smbfs mount still beats it. I am using > > the same > > file, source and destinations for all tests. I have tried alternate > > Network > > cards with no resulting benefit. > > > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > > 1950511+1 records in > > 1950511+1 records out > > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > > 1950477+74 records in > > 1950511+1 records out > > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) > > (98Mb/s) > > > > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs > > /mnt > > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > > 7619+1 records in > > 7619+1 records out > > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) > > (15Mb/s) > > > > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async > > gemini:/dsk/ufs /mnt > > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > > 7619+1 records in > > 7619+1 records out > > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) > > (19Mb/s) > > > > > > oguido# mount -t smbfs //gemini/ufs /mnt > > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > > 7619+1 records in > > 7619+1 records out > > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) > > (33Mb/s) > > > > Looking at a systat -v on the destination I see that the nfs test > > does not > > exceed 16KB/t with 100% busy where the other tests reach up to > > 128KB/t. > > For the record I get reads of 22Mb/s without and 77Mb/s with async > > turned on > > for the nfs mount. > > > > > > A copy of dmesg: > > ================ > > > > Copyright (c) 1992-2011 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, > > 1994 > > The Regents of the University of California. All rights > > reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 > > root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 > > Timecounter "i8254" frequency 1193182 Hz quality 0 > > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz > > 686-class CPU) > > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = > > 11 > > Features=0xbfebfbff > > Features2=0xe3fd > > AMD Features=0x20100000 > > AMD Features2=0x1 > > TSC: P-state invariant > > real memory = 4294967296 (4096 MB) > > avail memory = 3141234688 (2995 MB) > > ACPI APIC Table: > > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > > FreeBSD/SMP: 1 package(s) x 2 core(s) > > cpu0 (BSP): APIC ID: 0 > > cpu1 (AP): APIC ID: 1 > > ioapic0 irqs 0-23 on motherboard > > kbd1 at kbdmux0 > > cryptosoft0: on motherboard > > acpi0: on motherboard > > acpi0: [ITHREAD] > > acpi0: Power Button (fixed) > > acpi0: reservation of 0, a0000 (3) failed > > acpi0: reservation of 100000, bff00000 (3) failed > > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > > cpu0: on acpi0 > > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be > > 0xB1 (20101013/tbutils-354) > > cpu1: on acpi0 > > pcib0: port 0xcf8-0xcff on acpi0 > > pci0: on pcib0 > > pcib1: irq 16 at device 1.0 on pci0 > > pci1: on pcib1 > > mpt0: port 0x7800-0x78ff mem > > 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on > > pci1 > > mpt0: [ITHREAD] > > mpt0: MPI Version=1.5.18.0 > > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > > mpt0: 0 Active Volumes (2 Max) > > mpt0: 0 Hidden Drive Members (14 Max) > > uhci0: port 0xdc00-0xdc1f > > irq 16 at device 26.0 on pci0 > > uhci0: [ITHREAD] > > uhci0: LegSup = 0x2f00 > > usbus0: on uhci0 > > uhci1: port 0xe000-0xe01f > > irq 17 at device 26.1 on pci0 > > uhci1: [ITHREAD] > > uhci1: LegSup = 0x2f00 > > usbus1: on uhci1 > > ehci0: mem > > 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 > > ehci0: [ITHREAD] > > usbus2: EHCI version 1.0 > > usbus2: on ehci0 > > pci0: at device 27.0 (no driver attached) > > pcib2: irq 16 at device 28.0 on pci0 > > pci5: on pcib2 > > atapci0: port 0xac00-0xac7f mem > > 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on > > pci5 > > atapci0: [ITHREAD] > > ata2: on atapci0 > > ata2: [ITHREAD] > > ata3: on atapci0 > > ata3: [ITHREAD] > > pcib3: irq 17 at device 28.1 on pci0 > > pci4: on pcib3 > > em0: port 0x9c00-0x9c1f > > mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 > > on pci4 > > em0: Using an MSI interrupt > > em0: [FILTER] > > em0: Ethernet address: 00:1b:21:04:ac:11 > > pcib4: irq 19 at device 28.3 on pci0 > > pci3: on pcib4 > > age0: mem > > 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 > > age0: 1280 Tx FIFO, 2364 Rx FIFO > > age0: Using 1 MSI messages. > > miibus0: on age0 > > atphy0: PHY 0 on miibus0 > > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, > > 1000baseT-FDX, 1000baseT-FDX-master, auto > > age0: Ethernet address: 00:1a:92:d2:de:cc > > age0: [FILTER] > > pcib5: irq 16 at device 28.4 on pci0 > > pci2: on pcib5 > > atapci1: port > > 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f > > mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 > > atapci1: [ITHREAD] > > atapci2: on atapci1 > > atapci2: [ITHREAD] > > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported > > ata4: on atapci2 > > ata4: [ITHREAD] > > ata5: on atapci2 > > ata5: [ITHREAD] > > ata6: on atapci1 > > ata6: [ITHREAD] > > uhci2: port 0xd480-0xd49f > > irq 23 at device 29.0 on pci0 > > uhci2: [ITHREAD] > > uhci2: LegSup = 0x2f00 > > usbus3: on uhci2 > > uhci3: port 0xd800-0xd81f > > irq 19 at device 29.1 on pci0 > > uhci3: [ITHREAD] > > uhci3: LegSup = 0x2f00 > > usbus4: on uhci3 > > uhci4: port 0xd880-0xd89f > > irq 18 at device 29.2 on pci0 > > uhci4: [ITHREAD] > > uhci4: LegSup = 0x2f00 > > usbus5: on uhci4 > > ehci1: mem > > 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 > > ehci1: [ITHREAD] > > usbus6: EHCI version 1.0 > > usbus6: on ehci1 > > pcib6: at device 30.0 on pci0 > > pci6: on pcib6 > > vgapci0: mem > > 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff > > irq 22 at device 1.0 on pci6 > > fwohci0: port 0xbc00-0xbc7f mem > > 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 > > fwohci0: [ITHREAD] > > fwohci0: OHCI version 1.10 (ROM=1) > > fwohci0: No. of Isochronous channels is 4. > > fwohci0: EUI64 00:11:d8:00:01:39:52:d3 > > fwohci0: Phy 1394a available S400, 2 ports. > > fwohci0: Link S400, max_rec 2048 bytes. > > firewire0: on fwohci0 > > dcons_crom0: on firewire0 > > dcons_crom0: bus_addr 0x1494000 > > fwe0: on firewire0 > > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 > > fwe0: Ethernet address: 02:11:d8:39:52:d3 > > fwip0: on firewire0 > > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, > > S400, maxrec 2048 > > fwohci0: Initiate bus reset > > fwohci0: fwohci_intr_core: BUS reset > > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, > > CYCLEMASTER mode > > isab0: at device 31.0 on pci0 > > isa0: on isab0 > > atapci3: port > > 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f > > irq 19 at device 31.2 on pci0 > > atapci3: [ITHREAD] > > ata7: on atapci3 > > ata7: [ITHREAD] > > ata8: on atapci3 > > ata8: [ITHREAD] > > pci0: at device 31.3 (no driver attached) > > atapci4: port > > 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f > > irq 19 at device 31.5 on pci0 > > atapci4: [ITHREAD] > > ata9: on atapci4 > > ata9: [ITHREAD] > > ata10: on atapci4 > > ata10: [ITHREAD] > > acpi_button0: on acpi0 > > atrtc0: port 0x70-0x71 irq 8 on acpi0 > > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on > > acpi0 > > uart0: [FILTER] > > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 > > drq 2 on acpi0 > > fdc0: [FILTER] > > acpi_hpet0: iomem 0xfed00000-0xfed003ff > > on acpi0 > > Timecounter "HPET" frequency 14318180 Hz quality 900 > > ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on > > acpi0 > > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > > ppc0: FIFO with 16/16/9 bytes threshold > > ppc0: [ITHREAD] > > ppbus0: on ppc0 > > plip0: on ppbus0 > > plip0: [ITHREAD] > > lpt0: on ppbus0 > > lpt0: [ITHREAD] > > lpt0: Interrupt-driven port > > ppi0: on ppbus0 > > pmtimer0 on isa0 > > orm0: at iomem 0xc0000-0xc7fff pnpid ORM0000 on > > isa0 > > sc0: at flags 0x100 on isa0 > > sc0: VGA <16 virtual consoles, flags=0x300> > > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on > > isa0 > > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 > > ata0: [ITHREAD] > > ata1 at port 0x170-0x177,0x376 irq 15 on isa0 > > ata1: [ITHREAD] > > atkbdc0: at port 0x60,0x64 on isa0 > > atkbd0: irq 1 on atkbdc0 > > kbd0 at atkbd0 > > atkbd0: [GIANT-LOCKED] > > atkbd0: [ITHREAD] > > est0: on cpu0 > > p4tcc0: on cpu0 > > est1: on cpu1 > > p4tcc1: on cpu1 > > Timecounters tick every 1.000 msec > > IPsec: Initialized Security Association Processing. > > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) > > firewire0: bus manager 0 > > usbus0: 12Mbps Full Speed USB v1.0 > > usbus1: 12Mbps Full Speed USB v1.0 > > usbus2: 480Mbps High Speed USB v2.0 > > usbus3: 12Mbps Full Speed USB v1.0 > > usbus4: 12Mbps Full Speed USB v1.0 > > usbus5: 12Mbps Full Speed USB v1.0 > > usbus6: 480Mbps High Speed USB v2.0 > > ugen0.1: at usbus0 > > uhub0: on > > usbus0 > > ugen1.1: at usbus1 > > uhub1: on > > usbus1 > > ugen2.1: at usbus2 > > uhub2: on > > usbus2 > > ugen3.1: at usbus3 > > uhub3: on > > usbus3 > > ugen4.1: at usbus4 > > uhub4: on > > usbus4 > > ugen5.1: at usbus5 > > uhub5: on > > usbus5 > > ugen6.1: at usbus6 > > uhub6: on > > usbus6 > > acd0: DVDR at ata7-slave UDMA100 > > SATA 1.5Gb/s > > uhub0: 2 ports with 2 removable, self powered > > uhub1: 2 ports with 2 removable, self powered > > uhub3: 2 ports with 2 removable, self powered > > uhub4: 2 ports with 2 removable, self powered > > uhub5: 2 ports with 2 removable, self powered > > uhub2: 4 ports with 4 removable, self powered > > uhub6: 6 ports with 6 removable, self powered > > da1 at mpt0 bus 0 scbus0 target 1 lun 0 > > da1: Fixed Direct Access SCSI-5 device > > da1: 300.000MB/s transfers > > da1: Command Queueing enabled > > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > > da2 at mpt0 bus 0 scbus0 target 2 lun 0 > > da2: Fixed Direct Access SCSI-5 device > > da2: 300.000MB/s transfers > > da2: Command Queueing enabled > > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > > da0 at mpt0 bus 0 scbus0 target 0 lun 0 > > da0: Fixed Direct Access SCSI-5 device > > da0: 300.000MB/s transfers > > da0: Command Queueing enabled > > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) > > SMP: AP CPU #1 Launched! > > ugen1.2: at usbus1 > > ugen3.2: at usbus3 > > ukbd1: on > > usbus1 > > ums0: > 1.10/1.99, addr 2> on usbus3 > > ums0: 3 buttons and [XYZ] coordinates ID=0 > > kbd2 at ukbd1 > > uhid0: on > > usbus1 > > Trying to mount root from ufs:/dev/da0s1a > > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org"