From owner-freebsd-stable Tue Jul 23 21: 9:16 2002 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A0C2037B401 for ; Tue, 23 Jul 2002 21:08:59 -0700 (PDT) Received: from bronkowitz.keghouse.net (esoteric.keghouse.net [203.38.186.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2175043E67 for ; Tue, 23 Jul 2002 21:07:29 -0700 (PDT) (envelope-from olaf@bronkowitz.keghouse.net) Received: from bronkowitz.keghouse.net (localhost.keghouse.net [127.0.0.1]) by bronkowitz.keghouse.net (8.12.2/bronkowitz.keghouse.net) with ESMTP id g6O3w9Ux006349 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO) for ; Wed, 24 Jul 2002 13:28:11 +0930 (CST) Received: (from olaf@localhost) by bronkowitz.keghouse.net (8.12.2/8.12.2/Submit) id g6O3w90G006348 for stable@freebsd.org; Wed, 24 Jul 2002 13:28:09 +0930 (CST) Message-Id: <200207240358.g6O3w90G006348@bronkowitz.keghouse.net> Subject: Abominable NFSv3 read performance / FreeBSD server / Solaris client To: stable@freebsd.org Date: Wed, 24 Jul 2002 03:58:09 +0000 (GMT) From: Olaf R X-Mailer: ELM [version 2.4ME+ PL95a (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I'm experiencing terrible NFSv3 read performance between Solaris 8 clients and a FreeBSD 4.6-STABLE server, which ran CVSup/make world a day ago and thus includes the 'em' driver update (though that made no difference). Other FreeBSD clients and Mac G3 boxes running MacOS X 10.1.x can do reads at fairly respectable rate - several MB/s over 100 Mbps ethernet. The hardware: Client: Sun E250, 2 x 300 MHz UltraSPARC II, 1 GB RAM, light load. (Other Suns exhibit the same behaviour, though - this is an example.) Server: Dell PowerEdge 1650, 2 x 1300 MHz PIII, 1 GB RAM, no load. Network: all ports and cards at full-duplex, the Sun at 100 Mbps and the FreeBSD box at 1 Gbps, switched through a ridiculously underutilised cisco 6509. The numbers: NFSv3/TCP/32KB reads are going no higher than 0.5 MB/s. NFSv3/TCP/32KB writes are nearly getting the full 100 Mbps. To clear the network of blame, I've verified that ttcp can readily transfer nearly 100 Mbps in both directions between these machines. I've also used ttcp to blast several 100 Mbps streams in both directions (ie. one2many and many2one) between the FreeBSD box and several other 100Mbps Suns and everything scales just fine. The Solaris client can read from other vendors' NFS servers at several MB/s. Note that we're not using any jumbo frames, port channeling, firewalling, etc. I'm using the default make.conf for the FreeBSD build. Whilst performing "dd if=foo of=/dev/null bs=64k" on the client to read a several MB file, the following is seen on the server: freebsd# netstat -I em0 1 input (em0) output packets errs bytes packets errs bytes colls 0 0 22140 0 0 478660 0 369 0 22422 637 0 449416 0 0 0 20693 0 0 444298 0 386 0 24301 635 0 482818 0 0 0 20346 0 0 410936 0 ...and so on. (No, I don't know why the packet counts alternate between zero and non-zero.) Looking through the archives I see there have been some issues and/or concerns with the effects of delayed ACKs, New Reno, etc. Trying the various combinations of net.inet.tcp.{delayed_ack,newreno} has not helped. Varying net.inet.tcp.{send,recv}space also does little, though for the numbers in this post they're both at 64 KB. I'm not savvy enough with TCP's intricacies or the code to dig deeper at this point so I thought I'd see if those who do know more have ideas. For your information, the dmesg output follows. Copyright (c) 1992-2002 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.6-STABLE #9: Tue Jul 23 14:33:02 GMT 2002 root@freebsd:/usr/obj/usr/src/sys/DELLPE1650 Timecounter "i8254" frequency 1193182 Hz CPU: Pentium III/Pentium III Xeon/Celeron (1263.45-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6b1 Stepping = 1 Features=0x383fbff real memory = 1073676288 (1048512K bytes) avail memory = 1041997824 (1017576K bytes) Changing APIC ID for IO APIC #0 from 0 to 2 on chip Changing APIC ID for IO APIC #1 from 0 to 3 on chip Programming 16 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 Programming 16 pins in IOAPIC #1 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 1, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 2, version: 0x000f0011, at 0xfec00000 io1 (APIC): apic id: 3, version: 0x000f0011, at 0xfec01000 Preloaded elf kernel "kernel" at 0xc02d4000. Pentium Pro MTRR support enabled Using $PIR table, 9 entries at 0xc00fc220 npx0: on motherboard npx0: INT 16 interface pcib1: on motherboard IOAPIC #1 intpin 1 -> irq 2 IOAPIC #1 intpin 0 -> irq 3 IOAPIC #1 intpin 2 -> irq 5 pci1: on pcib1 em0: port 0xcce0-0xccff mem 0xfeb40000-0xfeb5ffff,0xfeb60000-0xfeb7ffff irq 2 at device 2.0 on pci1 em0: Speed:1000 Mbps Duplex:Full em1: port 0xccc0-0xccdf mem 0xfeb00000-0xfeb1ffff,0xfeb20000-0xfeb3ffff irq 3 at device 4.0 on pci1 em1: Speed:1000 Mbps Duplex:Full pcib5: at device 8.0 on pci1 IOAPIC #1 intpin 3 -> irq 7 pci2: on pcib5 pci2: (vendor=0x9005, dev=0x00c5) at 6.0 irq 5 pci2: (vendor=0x9005, dev=0x00c5) at 6.1 irq 7 aac0: mem 0xf0000000-0xf7ffffff irq 5 at device 8.1 on pci1 aac0: i960RX 100MHz, 118MB cache memory, optional battery present aac0: Kernel 2.7-0, Build 3153, S/N 1838d3 pcib0: on motherboard pci0: on pcib0 pci0: at 12.0 atapci0: port 0x8b0-0x8bf,0x374-0x377,0x170-0x177,0x3f4-0x3f7,0x1f0-0x1f7 at device 15.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata1: at 0x170 irq 15 on atapci0 pci0: at 15.2 irq 11 isab0: at device 15.3 on pci0 isa0: on isab0 pcib2: on motherboard IOAPIC #1 intpin 8 -> irq 13 IOAPIC #1 intpin 9 -> irq 16 pci3: on pcib2 ahc0: port 0xbc00-0xbcff mem 0xfe701000-0xfe701fff irq 13 at device 10.0 on pci3 aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs ahc1: port 0xb800-0xb8ff mem 0xfe700000-0xfe700fff irq 16 at device 10.1 on pci3 aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs pcib3: on motherboard pci4: on pcib3 pcib4: on motherboard pci5: on pcib4 orm0: