From owner-freebsd-hackers@FreeBSD.ORG Sat Aug 16 20:57:14 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 282811065673 for ; Sat, 16 Aug 2008 20:57:14 +0000 (UTC) (envelope-from usleepless@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.174]) by mx1.freebsd.org (Postfix) with ESMTP id EC4688FC0A for ; Sat, 16 Aug 2008 20:57:13 +0000 (UTC) (envelope-from usleepless@gmail.com) Received: by wf-out-1314.google.com with SMTP id 24so1386817wfg.7 for ; Sat, 16 Aug 2008 13:57:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=gRd6p3B5gR+nnjScltMMjkzD5/QslClfnw1xz3jcXuY=; b=G8LW0knoTjhc7tUVG5dMz0tWivAse118PnPkp1n7cERheZYEegqfj10WghfcFMI4Y0 3Pw/IiimhnOUX5XLoqchVSPv9H8D5sS5LAeCK+Ht82XIkyYlbxy4uE04sEbYs7ZQY7eJ 9xHRgbTTjOJtn/W84oETyNZQSfVJRh4lMtNr8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=BtPthlCA+59cQvD9q6ZQAdlknswfsuIFolUtjIV9T2MGN3ZndImDQL1JRq3jXY/hsS xRIWJBUkhWUJubeC7Wb3RNZ6jiRyRgPwHJiCmQetDmndy+Jb45omi5sJoskJWxZP7n5/ VFsTAHjHmxukQsWQ/AbD7fatkyAZtPRoRgdXI= Received: by 10.142.58.5 with SMTP id g5mr1474308wfa.224.1218918691264; Sat, 16 Aug 2008 13:31:31 -0700 (PDT) Received: by 10.142.174.7 with HTTP; Sat, 16 Aug 2008 13:31:31 -0700 (PDT) Message-ID: Date: Sat, 16 Aug 2008 22:31:31 +0200 From: usleepless@gmail.com To: rick-freebsd2008@kiwi-computer.com In-Reply-To: <20080816194311.GA67723@keira.kiwi-computer.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080816194311.GA67723@keira.kiwi-computer.com> Cc: freebsd-hackers@freebsd.org Subject: Re: local network throughput issues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2008 20:57:14 -0000 On 8/16/08, Rick C. Petty wrote: > Hello. I've been having some serious network throughput issues recently. > Prior to these issues, I was running 7.0-STABLE from a few months back. I > recently build a world/kernel from csup(1) on 2008-Jul-25 and started > noticing the issues. I'm currently on a RELENG_7 from 2008-Aug-02 and am > building another one today. > > The server is running samba, dnsmasq, an NFS server, and is the gateway for > a couple of freebsd machines. I first noticed the issue when I was unable > to play mp3s served over samba. I was getting a throughput of about 3 > KBytes/sec over my gigabit switched network. After a few weeks of trying > different versions of samba and watching tcpdump/trafshow, I decided to try > another test: > > % ssh gateway cat /dev/zero | dd of=/dev/null > load: 0.00 cmd: dd 68286 [runnable] 0.00u 0.00s 0% 844k > 3680+0 records in > 3680+0 records out > 1884160 bytes transferred in 512.856684 secs (3674 bytes/sec) > ^C3680+0 records in > 3680+0 records out > 1884160 bytes transferred in 513.392858 secs (3670 bytes/sec) > Killed by signal 2. > > Again, not even 4 KB/s throughput over ssh. What's weird is that not all > network traffic is slow. I'm able to download at almost my full DSL speed > (7168 Kbps) and here's the strange bit: NFS is fast! From the same > machine as above, I have an NFS mountpoint: > > % dd if=/nfs/some_large_file of=/dev/null > 55980+0 records in > 55980+0 records out > 28661760 bytes transferred in 0.147159 secs (194767476 bytes/sec) > > Which is what I would expect for a gigabit network. Even if I perform the > same test over ssh: > > % ssh workstation cat /nfs/some_large_file | dd of=/dev/null > 55980+0 records in > 55980+0 records out > 28661760 bytes transferred in 2.791392 secs (10267909 bytes/sec) > > This is reasonable considering ssh overhead. The kernel config for the > server contains: > > include GENERIC > ident DDB > options KDB > options KDB_TRACE > options DDB > options DDB_NUMSYM > > My internal network is on 172.23.20.x and the DSL modem is connected to the > same NIC on 192.168.0.1: > > # ifconfig nfe0 > nfe0: flags=8843 metric 0 mtu 1500 > options=10b > ether 00:15:f2:17:0c:20 > inet 172.23.20.1 netmask 0xffffff00 broadcast 172.23.20.255 > inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255 > media: Ethernet autoselect (1000baseTX ) > status: active > > # ipfw show > 00100 62968 8170100 allow ip from any to any via lo0 > 00110 0 0 deny ip from any to 127.0.0.0/8 > 00120 0 0 deny ip from 127.0.0.0/8 to any > 00130 25444031 23419621364 divert 8668 ip from any to any via 192.168.0.3 > 00140 0 0 deny ip from not 172.23.20.0/24 to 172.23.20.0/24 dst-port 137-149,445 > 00150 20639141 15067620538 allow tcp from any to any established > 00160 21 10683 allow ip from any to any frag > 00170 79791 4696696 allow tcp from any to any setup > 65530 5713367 8864760207 allow ip from any to any > 65535 0 0 deny ip from any to any > > Other than that, I'm not doing anything out of the ordinary. Why is NFS > behaving correctly and why are ssh/smbd connections so slow? I've pasted > my dmesg output below. I've used this configuration for years and it > wasn't until a recent RELENG_7 upgrade that I've had any problems. The box > was 99-100% idle during those tests and I don't see an interrupt storm or > anything funny like that. Any ideas? > > -- Rick C. Petty > > > >From /var/run/dmesg.boot: > > > Copyright (c) 1992-2008 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.0-STABLE #5: Sat Aug 2 19:43:25 CDT 2008 > rick@gateway:/usr/obj/usr/src/sys/DDB > module_register: module sk/miibus already exists! > Module sk/miibus failed to register: 17 > module_register: module skc/sk already exists! > Module skc/sk failed to register: 17 > module_register: module pci/skc already exists! > Module pci/skc failed to register: 17 > module_register: module rl/miibus already exists! > Module rl/miibus failed to register: 17 > module_register: module cardbus/rl already exists! > Module cardbus/rl failed to register: 17 > module_register: module pci/rl already exists! > Module pci/rl failed to register: 17 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ (2211.34-MHz 686-class CPU) > Origin = "AuthenticAMD" Id = 0x20f32 Stepping = 2 > Features=0x178bfbff > Features2=0x1 > AMD Features=0xe2500800 > AMD Features2=0x3 > Cores per package: 2 > real memory = 1073676288 (1023 MB) > avail memory = 1037037568 (988 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0: Changing APIC ID to 2 > ioapic0 irqs 0-23 on motherboard > kbd1 at kbdmux0 > ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) > acpi0: on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, 3fef0000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pci0: at device 0.0 (no driver attached) > isab0: at device 1.0 on pci0 > isa0: on isab0 > pci0: at device 1.1 (no driver attached) > atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 6.0 on pci0 > ata0: on atapci0 > ata0: [ITHREAD] > ata1: on atapci0 > ata1: [ITHREAD] > atapci1: port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xd800-0xd80f mem 0xd0002000-0xd0002fff irq 20 at device 7.0 on pci0 > atapci1: [ITHREAD] > ata2: on atapci1 > ata2: [ITHREAD] > ata3: on atapci1 > ata3: [ITHREAD] > atapci2: port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xc400-0xc40f mem 0xd0001000-0xd0001fff irq 21 at device 8.0 on pci0 > atapci2: [ITHREAD] > ata4: on atapci2 > ata4: [ITHREAD] > ata5: on atapci2 > ata5: [ITHREAD] > pcib1: at device 9.0 on pci0 > pci5: on pcib1 > vgapci0: mem 0xc0000000-0xc7ffffff irq 16 at device 6.0 on pci5 > atapci3: port 0xa000-0xa07f,0xa400-0xa4ff mem 0xc9041000-0xc9041fff,0xc9000000-0xc901ffff irq 17 at device 7.0 on pci5 > atapci3: [ITHREAD] > atapci3: [ITHREAD] > ata6: on atapci3 > ata6: [ITHREAD] > ata7: on atapci3 > ata7: [ITHREAD] > ata8: on atapci3 > ata8: [ITHREAD] > ata9: on atapci3 > ata9: [ITHREAD] > atapci4: port 0xa800-0xa87f,0xac00-0xacff mem 0xc9040000-0xc9040fff,0xc9020000-0xc903ffff irq 18 at device 8.0 on pci5 > atapci4: [ITHREAD] > atapci4: [ITHREAD] > ata10: on atapci4 > ata10: [ITHREAD] > ata11: on atapci4 > ata11: [ITHREAD] > ata12: on atapci4 > ata12: [ITHREAD] > ata13: on atapci4 > ata13: [ITHREAD] > nfe0: port 0xb000-0xb007 mem 0xd0000000-0xd0000fff irq 22 at device 10.0 on pci0 > miibus0: on nfe0 > e1000phy0: PHY 9 on miibus0 > e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto > nfe0: Ethernet address: 00:15:f2:17:0c:20 > nfe0: [FILTER] > pcib2: at device 11.0 on pci0 > pci4: on pcib2 > pcib3: at device 12.0 on pci0 > pci3: on pcib3 > pcib4: at device 13.0 on pci0 > pci2: on pcib4 > pcib5: at device 14.0 on pci0 > pci1: on pcib5 > cpu0: on acpi0 > powernow0: on cpu0 > device_attach: powernow0 attach returned 6 > cpu1: on acpi0 > powernow1: on cpu1 > device_attach: powernow1 attach returned 6 > acpi_button0: on acpi0 > acpi_tz0: on acpi0 > pmtimer0 on isa0 > orm0: at iomem 0xc0000-0xccfff pnpid ORM0000 on isa0 > atkbdc0: at port 0x60,0x64 on isa0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > ppc0: parallel port not found. > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 > sio0: type 8250 or not responding > sio0: [FILTER] > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Timecounters tick every 1.000 msec > ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to deny, logging disabled > ad0: 157066MB at ata0-master UDMA100 > ad12: 715404MB at ata6-master SATA150 > ad14: 715404MB at ata7-master SATA150 > ad16: 715404MB at ata8-master SATA150 > ad18: 715404MB at ata9-master SATA150 > ad20: 476940MB at ata10-master SATA300 > ad22: 476940MB at ata11-master SATA300 > ad24: 476940MB at ata12-master SATA300 > ad26: 476940MB at ata13-master SATA300 > SMP: AP CPU #1 Launched! > GEOM_LABEL: Label for provider ad0s1g is ufs/cache. > Trying to mount root from ufs:/dev/ad0s1a > GEOM_LABEL: Label ufs/cache removed. > GEOM_LABEL: Label for provider ad0s1g is ufs/cache. > GEOM_LABEL: Label ufs/cache removed. > nfe0: link state changed to UP > > Additional messages from dmesg -a: > > Aug 16 14:02:20 gateway login: ROOT LOGIN (root) ON ttyv3 > nfe0: promiscuous mode enabled > nfe0: promiscuous mode disabled i immediatly think dns issues if samba and ssh are slow. are your hosts reverse lookup-able? regards, usleep