From owner-freebsd-net@FreeBSD.ORG Mon May 12 18:50:07 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 235D9106564A for ; Mon, 12 May 2008 18:50:07 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from gtcomm.net (web.gtcomm.net [72.10.164.67]) by mx1.freebsd.org (Postfix) with ESMTP id B38BE8FC14 for ; Mon, 12 May 2008 18:50:06 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from [192.168.1.7] (c-76-108-179-28.hsd1.fl.comcast.net [76.108.179.28]) by gtcomm.net (8.12.20/8.12.10) with ESMTP id m4CInlY4022249; Mon, 12 May 2008 15:49:47 -0300 Message-ID: <482891C7.1030603@gtcomm.net> Date: Mon, 12 May 2008 14:51:51 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Stefan Lambrev References: <4827E79C.8050608@gtcomm.net> <48280B97.1020708@moneybookers.com> In-Reply-To: <48280B97.1020708@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Discrepancy on netstat -w x -I and what Cisco reports X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 18:50:07 -0000 Yes it was a typo I meant -w -I It just doesn't add up to what the switch is seeing, and it's directly connected and the traffic can't be going anywhere else. Stefan Lambrev wrote: > Greetings, > > I just have a question: > > is "netstat -w 100 lagg0" a typo ? > > If you want to see the traffic only on single interface you should use -I > > I do not know if this is bug, but netstat -w 100 > something_non_existing works on my bsd and just shows "Total" > > So may be from here comes the confusion. You think, that netstat count > only traffic on lagg0, > but it shows you the Total traffic? > > Paul wrote: >> This is very strange.. I can do: >> >> netstat -w 10 -I lagg0 >> input (lagg0) output >> packets errs bytes packets errs bytes colls >> 57806 0 41751685 232442 0 51062425 0 >> 56459 0 38341591 225146 0 48865209 0 >> 60687 0 43552946 227987 0 52008241 0 >> >> which is roughly 23,000 pps and the Cisco switch reports >> 30 second input rate 44544000 bits/sec, 16198 packets/sec >> >> >> Another example: >> >> netstat -w 10 -I lagg0 >> input (lagg0) output >> packets errs bytes packets errs bytes colls >> 71111 0 52180947 89734 0 25304669 0 >> 66847 0 49028588 81737 0 21614941 0 >> 63530 0 43502418 83419 0 24599547 0 >> >> 8,300 or so pps >> >> Cisco: >> 30 second input rate 19230000 bits/sec, 4594 packets/sec >> >> >> In some cases it's pretty close, cisco says 6500 and bsd says 7500.. >> but sometimes it is way off >> I even checked the em interfaces directly to see if it was a problem >> with the lagg interface code and they report weird issues, check below. >> Kind of weird.. I'm not sure if this is a Cisco issue or Fbsd issue >> with the counters. >> It's two different Cisco switches and two fbsd machines that have the >> same kernel, etc. >> >> Maybe this is a side effect of setting my kernel HZ at 4000? >> Gets even worse like this: >> netstat -w 100 lagg0 >> input (Total) output >> packets errs bytes packets errs bytes colls >> 9229322 0 3337789024 9424932 0 3510341158 0 >> >> 100 seconds, 9.4 million packets? That's 94,000 pps and cisco reports >> 2 minute input rate 44130000 bits/sec, 14892 packets/sec > Again this is Total not lagg0 >> and even check this out: >> >> netstat -w 1 lagg0 >> input *(Total) * output >> packets errs bytes packets errs bytes colls >> 92481 0 31630795 94952 0 34193131 0 >> 89078 0 32498082 91460 0 35094821 0 >> 87540 0 34526292 89992 0 37159101 0 >> 88987 0 32391984 91765 0 35394351 0 >> >> >> netstat -w 1 em0 >> input *(Total) * output >> packets errs bytes packets errs bytes colls >> 96802 0 39474036 99897 0 42814245 0 >> 93277 0 37018533 95943 0 39860879 0 >> 95916 0 37559076 99032 0 40739640 0 >> >> netstat -w 1 em1 >> input * (Total) * output >> packets errs bytes packets errs bytes colls >> 97102 0 38369949 99508 0 40841183 0 >> 92321 0 35375169 94305 0 37384073 0 >> 92225 0 33171455 94253 0 35209658 0 >> >> What in the world?? em0 + em1 is almost 200k pps but lagg0 says 100k >> and i KNOW it's not doing anywhere near.. >> cisco report >> 2 minute output rate 32928000 bits/sec, 5823 packets/sec > Again missing -I :) >> >> Now all lagg interfaces are reporting >> netstat -w 1 lagg1 >> input * (Total) * output >> packets errs bytes packets errs bytes colls >> 89324 0 30824353 91518 0 32770482 0 >> 85875 0 31924738 87813 0 33552137 0 >> 84105 0 31176932 85666 0 32393051 0 >> 83617 0 32175677 84871 0 33120271 0 >> 90611 0 37313093 92403 0 38818721 0 >> >> lagg1 goes directly to another freebsd box and on the other freebsd >> box I do: >> netstat -w 1 -I lagg1 >> input (lagg1) output >> packets errs bytes packets errs bytes colls >> 45 0 3078 2213 0 1890198 0 >> 48 0 3245 1958 0 1545642 0 >> 43 0 3186 1975 0 1628916 0 >> 43 0 2905 2169 0 1918250 0 >> 46 0 3464 1859 0 1729764 0 >> 46 0 3134 1873 0 1739662 0 >> >> and the other one >> netstat -w 1 lagg1 >> input * (Total)* output >> packets errs bytes packets errs bytes colls >> 92149 0 31706183 93523 0 32673138 0 >> 89737 0 28119643 91323 0 28958816 0 >> >> >> Doing all these reports now seems to have the interfaces stuck at >> packets errs bytes packets errs bytes colls >> 96937 0 31749525 98551 0 32678863 0 >> 85892 0 29411078 87233 0 30182355 0 >> 90435 0 31628680 91620 0 32215244 0 >> 87383 0 30616741 88278 0 31026608 0 >> >> >> every interface on the machine is reporting the same PPS and bytes.. >> lol :) >> >> So something is extremely fishy about the counters.. I'm going to try >> and update to -STABLE to see if it makes any difference. It's not >> just the lagg interface either because all the em's are showing it as >> well. >> >> This is using 4 port Intel Server PCI Express NIC >> >> ifstat seems to report correct usage in Kbps and seems to report >> correct packet count. Maybe it's just a netstat problem? >> >> I will see if stable fixes it. Also, feel free to make any comments >> on my config file for routing. >> >> FreeBSD 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #6: Thu Apr 17 >> 18:11:49 EDT 2008 amd64 >> >> >> UPDATE.. Changed 1 router to stable: >> FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT >> 2008 amd64 >> >> Still see: >> 2 minute input rate 10463000 bits/sec, 2481 packets/sec >> 2 minute output rate 40075000 bits/sec, 6847 packets/sec >> >> and >> input (lagg0) output >> packets errs bytes packets errs bytes colls >> 6940 0 5172153 4841 0 1345660 0 >> 5922 0 4252074 3963 0 1087205 0 >> 6673 0 4982394 4116 0 1056933 0 >> 6659 0 4467398 4140 0 1068919 0 >> 7085 0 4692973 4777 0 1665109 0 >> 7140 0 4654486 4713 0 1658303 0 >> 7070 0 4558384 5078 0 1994666 0 >> 6375 0 4575464 4037 0 1121385 0 >> 6257 0 3932910 4321 0 1607862 0 >> 6504 0 4345014 4370 0 1278819 0 >> >> >> Hmmmm.. >> >> >> >> >> em0: port >> 0xece0-0xecff mem 0xd5ee0000-0xd5efffff,0xd5ec0000-0xd5edffff irq 17 >> at device 0.0 on pci12 >> em0: Using MSI interrupt >> em0: Ethernet address: 00:15:17:5d:18:00 >> em0: [FILTER] >> em1: port >> 0xecc0-0xecdf mem 0xd5ea0000-0xd5ebffff,0xd5e80000-0xd5e9ffff irq 18 >> at device 0.1 on pci12 >> em1: Using MSI interrupt >> em1: Ethernet address: 00:15:17:5d:18:01 >> em1: [FILTER] >> .......etc.. to em7 >> >> >> Copyright (c) 1992-2008 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 7.0-RELEASE-p1 #6: Thu Apr 17 18:11:49 EDT 2008 >> root@CR1.MTL3.Gtcomm.net:/usr/obj/usr/src/sys/ROUTER >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2329.28-MHz >> K8-class CPU) >> Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 >> >> Features=0xbfebfbff >> >> >> Features2=0x4e3bd >> >> AMD Features=0x20100800 >> AMD Features2=0x1 >> Cores per package: 2 >> usable memory = 4285833216 (4087 MB) >> avail memory = 4124545024 (3933 MB) >> ACPI APIC Table: >> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> cpu2 (AP): APIC ID: 6 >> cpu3 (AP): APIC ID: 7 >> ioapic0: Changing APIC ID to 8 >> ioapic1: Changing APIC ID to 9 >> >> >> >> Kernel config file: (if you have any suggestions about the config let >> me know for faster routing speed) >> >> cpu HAMMER >> ident GENERIC >> >> #makeoptions DEBUG=-g # Build kernel with gdb(1) >> debug symbols >> >> options SCHED_4BSD # 4BSD scheduler >> #options PREEMPTION # Enable kernel thread >> preemption >> options INET # InterNETworking >> options INET6 # IPv6 communications protocols >> options FFS # Berkeley Fast Filesystem >> options SOFTUPDATES # Enable FFS soft updates >> support >> options UFS_ACL # Support for access control >> lists >> options UFS_DIRHASH # Improve performance on big >> directories >> options UFS_GJOURNAL # Enable gjournal-based UFS >> journaling >> options MD_ROOT # MD is a potential root device >> options NTFS # NT File System >> options MSDOSFS # MSDOS Filesystem >> options CD9660 # ISO 9660 Filesystem >> options PROCFS # Process filesystem >> (requires PSEUDOFS) >> options PSEUDOFS # Pseudo-filesystem framework >> options GEOM_PART_GPT # GUID Partition Tables. >> options GEOM_LABEL # Provides labelization >> options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP >> THIS!] >> options COMPAT_IA32 # Compatible with i386 binaries >> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >> options SCSI_DELAY=5000 # Delay (in ms) before >> probing SCSI >> options KTRACE # ktrace(1) support >> options SYSVSHM # SYSV-style shared memory >> options SYSVMSG # SYSV-style message queues >> options SYSVSEM # SYSV-style semaphores >> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B >> real-time extensions >> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >> #options ADAPTIVE_GIANT # Giant mutex is adaptive. >> options NO_ADAPTIVE_MUTEXES ## >> options STOP_NMI # Stop CPUS using NMI instead >> of IPI >> options AUDIT # Security event auditing >> options IPSEC ## for tcp md5 >> options TCP_SIGNATURE ##include support for RFC 2385 >> device crypto ## for md5 >> device cryptodev ## for md5 >> >> # Make an SMP-capable kernel by default >> options SMP # Symmetric MultiProcessor >> Kernel >> >> # CPU frequency control >> device cpufreq >> >> # Bus support. >> device acpi >> device pci >> >> # Floppy drives >> device fdc >> >> # ATA and ATAPI devices >> device ata >> device atadisk # ATA disk drives >> device ataraid # ATA RAID drives >> device atapicd # ATAPI CDROM drives >> device atapifd # ATAPI floppy drives >> device atapist # ATAPI tape drives >> options ATA_STATIC_ID # Static device numbering >> >> # SCSI peripherals >> device scbus # SCSI bus (required for SCSI) >> device ch # SCSI media changers >> device da # Direct Access (disks) >> device sa # Sequential Access (tape etc) >> device cd # CD >> device pass # Passthrough device (direct SCSI >> access) >> device ses # SCSI Environmental Services (and >> SAF-TE) >> >> # RAID controllers >> device mfi # LSI MegaRAID SAS >> >> # atkbdc0 controls both the keyboard and the PS/2 mouse >> device atkbdc # AT keyboard controller >> device atkbd # AT keyboard >> device psm # PS/2 mouse >> >> device kbdmux # keyboard multiplexer >> >> device vga # VGA video card driver >> >> device splash # Splash screen and screen saver support >> >> # syscons is the default console driver, resembling an SCO console >> device sc >> >> device agp # support several AGP chipsets >> >> # Serial (COM) ports >> device sio # 8250, 16[45]50 based serial ports >> device uart # Generic UART driver >> >> # PCI Ethernet NICs. >> device em # Intel PRO/1000 adapter Gigabit >> Ethernet Card >> # PCI Ethernet NICs that use the common MII bus controller code. >> # NOTE: Be sure to keep the 'device miibus' line in order to use >> these NICs! >> device miibus >> device bce # Broadcom BCM5706/BCM5708 Gigabit >> Ethernet >> >> # Pseudo devices. >> device loop # Network loopback >> device random # Entropy device >> device ether # Ethernet support >> device sl # Kernel SLIP >> device ppp # Kernel PPP >> device tun # Packet tunnel. >> device pty # Pseudo-ttys (telnet etc) >> device md # Memory "disks" >> device gif # IPv6 and IPv4 tunneling >> device faith # IPv6-to-IPv4 relaying (translation) >> device firmware # firmware assist module >> >> # The `bpf' device enables the Berkeley Packet Filter. >> # Be aware of the administrative consequences of enabling this! >> # Note that 'bpf' is required for DHCP. >> device bpf # Berkeley packet filter >> >> # USB support >> device uhci # UHCI PCI->USB interface >> device ohci # OHCI PCI->USB interface >> device ehci # EHCI PCI->USB interface (USB 2.0) >> device usb # USB Bus (required) >> #device udbp # USB Double Bulk Pipe devices >> device ugen # Generic >> device uhid # "Human Interface Devices" >> device ukbd # Keyboard >> device umass # Disks/Mass storage - Requires scbus >> and da >> device ums # Mouse >> >> ### OPTIONS >> >> >> options MP_WATCHDOG >> options DEVICE_POLLING >> device pf >> device pflog >> device pfsync >> device carp >> device vlan >> device gre >> device if_bridge >> device tun >> device lagg >> device stf #6to4 IPv6 over IPv4 encapsulation >> >> options ALTQ >> options ALTQ_CBQ # Class Bases Queuing (CBQ) >> options ALTQ_RED # Random Early Detection (RED) >> options ALTQ_RIO # RED In/Out >> options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC) >> options ALTQ_CDNR >> options ALTQ_PRIQ # Priority Queuing (PRIQ) >> options ALTQ_NOPCC # Required for SMP build >> >> >> >> options NETGRAPH >> options NETGRAPH_CISCO >> options NETGRAPH_FEC >> options NETGRAPH_ETHER >> >> >> >> >> >> Copyright (c) 1992-2008 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT 2008 >> root@CR2.MTL3.Gtcomm.net:/usr/obj/usr/src/sys/ROUTER >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2329.26-MHz >> K8-class CPU) >> Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 >> >> Features=0xbfebfbff >> >> >> Features2=0x4e3bd >> >> AMD Features=0x20100800 >> AMD Features2=0x1 >> Cores per package: 2 >> usable memory = 4286042112 (4087 MB) >> avail memory = 4124753920 (3933 MB) >> ACPI APIC Table: >> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> cpu2 (AP): APIC ID: 6 >> cpu3 (AP): APIC ID: 7 >> ioapic0: Changing APIC ID to 8 >> ioapic1: Changing APIC ID to 9 >> ioapic0 irqs 0-23 on motherboard >> ioapic1 irqs 64-87 on motherboard >> kbd1 at kbdmux0 >> cryptosoft0: on motherboard >> acpi0: on motherboard >> acpi0: [ITHREAD] >> acpi0: Power Button (fixed) >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 >> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 >> acpi_hpet0: iomem 0xfed00000-0xfed003ff >> on acpi0 >> Timecounter "HPET" frequency 14318180 Hz quality 900 >> cpu0: on acpi0 >> est0: on cpu0 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 728072806000728 >> device_attach: est0 attach returned 6 >> p4tcc0: on cpu0 >> cpu1: on acpi0 >> est1: on cpu1 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 728072806000728 >> device_attach: est1 attach returned 6 >> p4tcc1: on cpu1 >> cpu2: on acpi0 >> est2: on cpu2 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 728072806000728 >> device_attach: est2 attach returned 6 >> p4tcc2: on cpu2 >> cpu3: on acpi0 >> est3: on cpu3 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 728072806000728 >> device_attach: est3 attach returned 6 >> p4tcc3: on cpu3 >> pcib0: port 0xcf8-0xcff on acpi0 >> pci0: on pcib0 >> pcib1: at device 2.0 on pci0 >> pci6: on pcib1 >> pcib2: at device 0.0 on pci6 >> pci7: on pcib2 >> pcib3: at device 0.0 on pci7 >> pci8: on pcib3 >> pcib4: at device 0.0 on pci8 >> pci9: on pcib4 >> bce0: mem >> 0xd6000000-0xd7ffffff irq 16 at device 0.0 on pci9 >> miibus0: on bce0 >> brgphy0: PHY 1 on miibus0 >> brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, >> 1000baseT-FDX, auto >> bce0: Ethernet address: 00:19:b9:cd:60:44 >> bce0: [ITHREAD] >> bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W >> (0x02090105); Flags( MFW MSI ) >> pcib5: at device 1.0 on pci7 >> pci10: on pcib5 >> pcib6: at device 0.0 on pci10 >> pci11: on pcib6 >> pcib7: at device 0.0 on pci11 >> pci12: on pcib7 >> em0: port 0xece0-0xecff >> mem 0xd5ee0000-0xd5efffff,0xd5ec0000-0xd5edffff irq 17 at device 0.0 >> on pci12 >> em0: Using MSI interrupt >> em0: [FILTER] >> em0: Ethernet address: 00:15:17:5d:2a:40 >> em1: port 0xecc0-0xecdf >> mem 0xd5ea0000-0xd5ebffff,0xd5e80000-0xd5e9ffff irq 18 at device 0.1 >> on pci12 >> em1: Using MSI interrupt >> em1: [FILTER] >> em1: Ethernet address: 00:15:17:5d:2a:41 >> pcib8: at device 1.0 on pci11 >> pci13: on pcib8 >> em2: port 0xdce0-0xdcff >> mem 0xd5ce0000-0xd5cfffff,0xd5cc0000-0xd5cdffff irq 18 at device 0. >> >> >> lagg0: flags=8843 metric 0 >> mtu 1500 >> options=19b >> ether 00:15:17:5d:2a:40 >> media: Ethernet autoselect >> status: active >> laggproto lacp >> laggport: em1 flags=1c >> laggport: em0 flags=1c >> >> >> lagg1: flags=8843 metric 0 >> mtu 1500 >> options=19b >> ether 00:15:17:5d:28:62 >> inet netmask 0xfffffffc broadcast >> media: Ethernet autoselect >> status: active >> laggproto lacp >> laggport: em7 flags=1c >> laggport: em6 flags=1c >> >> lagg2: flags=8843 metric 0 >> mtu 1500 >> options=19b >> ether 00:15:17:5d:28:60 >> media: Ethernet autoselect >> status: active >> laggproto lacp >> laggport: em5 flags=1c >> laggport: em4 flags=1c >> >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > -- > > Best Wishes, > Stefan Lambrev > ICQ# 24134177 >