Date: Sat, 16 Aug 2008 22:31:31 +0200 From: usleepless@gmail.com To: rick-freebsd2008@kiwi-computer.com Cc: freebsd-hackers@freebsd.org Subject: Re: local network throughput issues Message-ID: <c39ec84c0808161331y3656dc1av10dd3d3208b2548e@mail.gmail.com> In-Reply-To: <20080816194311.GA67723@keira.kiwi-computer.com> References: <20080816194311.GA67723@keira.kiwi-computer.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8/16/08, Rick C. Petty <rick-freebsd2008@kiwi-computer.com> wrote: > Hello. I've been having some serious network throughput issues recently. > Prior to these issues, I was running 7.0-STABLE from a few months back. I > recently build a world/kernel from csup(1) on 2008-Jul-25 and started > noticing the issues. I'm currently on a RELENG_7 from 2008-Aug-02 and am > building another one today. > > The server is running samba, dnsmasq, an NFS server, and is the gateway for > a couple of freebsd machines. I first noticed the issue when I was unable > to play mp3s served over samba. I was getting a throughput of about 3 > KBytes/sec over my gigabit switched network. After a few weeks of trying > different versions of samba and watching tcpdump/trafshow, I decided to try > another test: > > % ssh gateway cat /dev/zero | dd of=/dev/null > load: 0.00 cmd: dd 68286 [runnable] 0.00u 0.00s 0% 844k > 3680+0 records in > 3680+0 records out > 1884160 bytes transferred in 512.856684 secs (3674 bytes/sec) > ^C3680+0 records in > 3680+0 records out > 1884160 bytes transferred in 513.392858 secs (3670 bytes/sec) > Killed by signal 2. > > Again, not even 4 KB/s throughput over ssh. What's weird is that not all > network traffic is slow. I'm able to download at almost my full DSL speed > (7168 Kbps) and here's the strange bit: NFS is fast! From the same > machine as above, I have an NFS mountpoint: > > % dd if=/nfs/some_large_file of=/dev/null > 55980+0 records in > 55980+0 records out > 28661760 bytes transferred in 0.147159 secs (194767476 bytes/sec) > > Which is what I would expect for a gigabit network. Even if I perform the > same test over ssh: > > % ssh workstation cat /nfs/some_large_file | dd of=/dev/null > 55980+0 records in > 55980+0 records out > 28661760 bytes transferred in 2.791392 secs (10267909 bytes/sec) > > This is reasonable considering ssh overhead. The kernel config for the > server contains: > > include GENERIC > ident DDB > options KDB > options KDB_TRACE > options DDB > options DDB_NUMSYM > > My internal network is on 172.23.20.x and the DSL modem is connected to the > same NIC on 192.168.0.1: > > # ifconfig nfe0 > nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=10b<RXCSUM,TXCSUM,VLAN_MTU,TSO4> > ether 00:15:f2:17:0c:20 > inet 172.23.20.1 netmask 0xffffff00 broadcast 172.23.20.255 > inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255 > media: Ethernet autoselect (1000baseTX <full-duplex,flag0,flag1>) > status: active > > # ipfw show > 00100 62968 8170100 allow ip from any to any via lo0 > 00110 0 0 deny ip from any to 127.0.0.0/8 > 00120 0 0 deny ip from 127.0.0.0/8 to any > 00130 25444031 23419621364 divert 8668 ip from any to any via 192.168.0.3 > 00140 0 0 deny ip from not 172.23.20.0/24 to 172.23.20.0/24 dst-port 137-149,445 > 00150 20639141 15067620538 allow tcp from any to any established > 00160 21 10683 allow ip from any to any frag > 00170 79791 4696696 allow tcp from any to any setup > 65530 5713367 8864760207 allow ip from any to any > 65535 0 0 deny ip from any to any > > Other than that, I'm not doing anything out of the ordinary. Why is NFS > behaving correctly and why are ssh/smbd connections so slow? I've pasted > my dmesg output below. I've used this configuration for years and it > wasn't until a recent RELENG_7 upgrade that I've had any problems. The box > was 99-100% idle during those tests and I don't see an interrupt storm or > anything funny like that. Any ideas? > > -- Rick C. Petty > > > >From /var/run/dmesg.boot: > > > Copyright (c) 1992-2008 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.0-STABLE #5: Sat Aug 2 19:43:25 CDT 2008 > rick@gateway:/usr/obj/usr/src/sys/DDB > module_register: module sk/miibus already exists! > Module sk/miibus failed to register: 17 > module_register: module skc/sk already exists! > Module skc/sk failed to register: 17 > module_register: module pci/skc already exists! > Module pci/skc failed to register: 17 > module_register: module rl/miibus already exists! > Module rl/miibus failed to register: 17 > module_register: module cardbus/rl already exists! > Module cardbus/rl failed to register: 17 > module_register: module pci/rl already exists! > Module pci/rl failed to register: 17 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ (2211.34-MHz 686-class CPU) > Origin = "AuthenticAMD" Id = 0x20f32 Stepping = 2 > Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> > Features2=0x1<SSE3> > AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!> > AMD Features2=0x3<LAHF,CMP> > Cores per package: 2 > real memory = 1073676288 (1023 MB) > avail memory = 1037037568 (988 MB) > ACPI APIC Table: <Nvidia AWRDACPI> > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0: Changing APIC ID to 2 > ioapic0 <Version 1.1> irqs 0-23 on motherboard > kbd1 at kbdmux0 > ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) > acpi0: <Nvidia AWRDACPI> on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, 3fef0000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pci0: <memory> at device 0.0 (no driver attached) > isab0: <PCI-ISA bridge> at device 1.0 on pci0 > isa0: <ISA bus> on isab0 > pci0: <serial bus, SMBus> at device 1.1 (no driver attached) > atapci0: <nVidia nForce CK804 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 6.0 on pci0 > ata0: <ATA channel 0> on atapci0 > ata0: [ITHREAD] > ata1: <ATA channel 1> on atapci0 > ata1: [ITHREAD] > atapci1: <nVidia nForce CK804 SATA300 controller> port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xd800-0xd80f mem 0xd0002000-0xd0002fff irq 20 at device 7.0 on pci0 > atapci1: [ITHREAD] > ata2: <ATA channel 0> on atapci1 > ata2: [ITHREAD] > ata3: <ATA channel 1> on atapci1 > ata3: [ITHREAD] > atapci2: <nVidia nForce CK804 SATA300 controller> port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xc400-0xc40f mem 0xd0001000-0xd0001fff irq 21 at device 8.0 on pci0 > atapci2: [ITHREAD] > ata4: <ATA channel 0> on atapci2 > ata4: [ITHREAD] > ata5: <ATA channel 1> on atapci2 > ata5: [ITHREAD] > pcib1: <ACPI PCI-PCI bridge> at device 9.0 on pci0 > pci5: <ACPI PCI bus> on pcib1 > vgapci0: <VGA-compatible display> mem 0xc0000000-0xc7ffffff irq 16 at device 6.0 on pci5 > atapci3: <Promise PDC40718 SATA300 controller> port 0xa000-0xa07f,0xa400-0xa4ff mem 0xc9041000-0xc9041fff,0xc9000000-0xc901ffff irq 17 at device 7.0 on pci5 > atapci3: [ITHREAD] > atapci3: [ITHREAD] > ata6: <ATA channel 0> on atapci3 > ata6: [ITHREAD] > ata7: <ATA channel 1> on atapci3 > ata7: [ITHREAD] > ata8: <ATA channel 2> on atapci3 > ata8: [ITHREAD] > ata9: <ATA channel 3> on atapci3 > ata9: [ITHREAD] > atapci4: <Promise PDC40718 SATA300 controller> port 0xa800-0xa87f,0xac00-0xacff mem 0xc9040000-0xc9040fff,0xc9020000-0xc903ffff irq 18 at device 8.0 on pci5 > atapci4: [ITHREAD] > atapci4: [ITHREAD] > ata10: <ATA channel 0> on atapci4 > ata10: [ITHREAD] > ata11: <ATA channel 1> on atapci4 > ata11: [ITHREAD] > ata12: <ATA channel 2> on atapci4 > ata12: [ITHREAD] > ata13: <ATA channel 3> on atapci4 > ata13: [ITHREAD] > nfe0: <NVIDIA nForce4 CK804 MCP9 Networking Adapter> port 0xb000-0xb007 mem 0xd0000000-0xd0000fff irq 22 at device 10.0 on pci0 > miibus0: <MII bus> on nfe0 > e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 9 on miibus0 > e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto > nfe0: Ethernet address: 00:15:f2:17:0c:20 > nfe0: [FILTER] > pcib2: <ACPI PCI-PCI bridge> at device 11.0 on pci0 > pci4: <ACPI PCI bus> on pcib2 > pcib3: <ACPI PCI-PCI bridge> at device 12.0 on pci0 > pci3: <ACPI PCI bus> on pcib3 > pcib4: <ACPI PCI-PCI bridge> at device 13.0 on pci0 > pci2: <ACPI PCI bus> on pcib4 > pcib5: <ACPI PCI-PCI bridge> at device 14.0 on pci0 > pci1: <ACPI PCI bus> on pcib5 > cpu0: <ACPI CPU> on acpi0 > powernow0: <Cool`n'Quiet K8> on cpu0 > device_attach: powernow0 attach returned 6 > cpu1: <ACPI CPU> on acpi0 > powernow1: <Cool`n'Quiet K8> on cpu1 > device_attach: powernow1 attach returned 6 > acpi_button0: <Power Button> on acpi0 > acpi_tz0: <Thermal Zone> on acpi0 > pmtimer0 on isa0 > orm0: <ISA Option ROM> at iomem 0xc0000-0xccfff pnpid ORM0000 on isa0 > atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 > atkbd0: <AT Keyboard> irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > ppc0: parallel port not found. > sc0: <System console> at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 > sio0: type 8250 or not responding > sio0: [FILTER] > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Timecounters tick every 1.000 msec > ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to deny, logging disabled > ad0: 157066MB <Hitachi HDS722516VLAT80 V34OA6MA> at ata0-master UDMA100 > ad12: 715404MB <Seagate ST3750330AS SD04> at ata6-master SATA150 > ad14: 715404MB <Seagate ST3750330AS SD04> at ata7-master SATA150 > ad16: 715404MB <Seagate ST3750330AS SD04> at ata8-master SATA150 > ad18: 715404MB <Seagate ST3750330AS SD04> at ata9-master SATA150 > ad20: 476940MB <WDC WD5000AAKS-65TMA0 12.01C01> at ata10-master SATA300 > ad22: 476940MB <WDC WD5000AAKS-65TMA0 12.01C01> at ata11-master SATA300 > ad24: 476940MB <WDC WD5000AAKS-00TMA0 12.01C01> at ata12-master SATA300 > ad26: 476940MB <WDC WD5000AAKS-65TMA0 12.01C01> at ata13-master SATA300 > SMP: AP CPU #1 Launched! > GEOM_LABEL: Label for provider ad0s1g is ufs/cache. > Trying to mount root from ufs:/dev/ad0s1a > GEOM_LABEL: Label ufs/cache removed. > GEOM_LABEL: Label for provider ad0s1g is ufs/cache. > GEOM_LABEL: Label ufs/cache removed. > nfe0: link state changed to UP > > Additional messages from dmesg -a: > > Aug 16 14:02:20 gateway login: ROOT LOGIN (root) ON ttyv3 > nfe0: promiscuous mode enabled > nfe0: promiscuous mode disabled i immediatly think dns issues if samba and ssh are slow. are your hosts reverse lookup-able? regards, usleep
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c39ec84c0808161331y3656dc1av10dd3d3208b2548e>