Date: Mon, 12 May 2008 02:45:48 -0400 From: Paul <paul@gtcomm.net> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: Discrepancy on netstat -w x -I and what Cisco reports Message-ID: <4827E79C.8050608@gtcomm.net>
next in thread | raw e-mail | index | archive | help
This is very strange.. I can do:
netstat -w 10 -I lagg0
input (lagg0) output
packets errs bytes packets errs bytes colls
57806 0 41751685 232442 0 51062425 0
56459 0 38341591 225146 0 48865209 0
60687 0 43552946 227987 0 52008241 0
which is roughly 23,000 pps and the Cisco switch reports
30 second input rate 44544000 bits/sec, 16198 packets/sec
Another example:
netstat -w 10 -I lagg0
input (lagg0) output
packets errs bytes packets errs bytes colls
71111 0 52180947 89734 0 25304669 0
66847 0 49028588 81737 0 21614941 0
63530 0 43502418 83419 0 24599547 0
8,300 or so pps
Cisco:
30 second input rate 19230000 bits/sec, 4594 packets/sec
In some cases it's pretty close, cisco says 6500 and bsd says 7500..
but sometimes it is way off
I even checked the em interfaces directly to see if it was a problem
with the lagg interface code and they report weird issues, check below.
Kind of weird.. I'm not sure if this is a Cisco issue or Fbsd issue with
the counters.
It's two different Cisco switches and two fbsd machines that have the
same kernel, etc.
Maybe this is a side effect of setting my kernel HZ at 4000?
Gets even worse like this:
netstat -w 100 lagg0
input (Total) output
packets errs bytes packets errs bytes colls
9229322 0 3337789024 9424932 0 3510341158 0
100 seconds, 9.4 million packets? That's 94,000 pps and cisco reports
2 minute input rate 44130000 bits/sec, 14892 packets/sec
and even check this out:
netstat -w 1 lagg0
input (Total) output
packets errs bytes packets errs bytes colls
92481 0 31630795 94952 0 34193131 0
89078 0 32498082 91460 0 35094821 0
87540 0 34526292 89992 0 37159101 0
88987 0 32391984 91765 0 35394351 0
netstat -w 1 em0
input (Total) output
packets errs bytes packets errs bytes colls
96802 0 39474036 99897 0 42814245 0
93277 0 37018533 95943 0 39860879 0
95916 0 37559076 99032 0 40739640 0
netstat -w 1 em1
input (Total) output
packets errs bytes packets errs bytes colls
97102 0 38369949 99508 0 40841183 0
92321 0 35375169 94305 0 37384073 0
92225 0 33171455 94253 0 35209658 0
What in the world?? em0 + em1 is almost 200k pps but lagg0 says 100k and
i KNOW it's not doing anywhere near..
cisco report
2 minute output rate 32928000 bits/sec, 5823 packets/sec
Now all lagg interfaces are reporting
netstat -w 1 lagg1
input (Total) output
packets errs bytes packets errs bytes colls
89324 0 30824353 91518 0 32770482 0
85875 0 31924738 87813 0 33552137 0
84105 0 31176932 85666 0 32393051 0
83617 0 32175677 84871 0 33120271 0
90611 0 37313093 92403 0 38818721 0
lagg1 goes directly to another freebsd box and on the other freebsd box
I do:
netstat -w 1 -I lagg1
input (lagg1) output
packets errs bytes packets errs bytes colls
45 0 3078 2213 0 1890198 0
48 0 3245 1958 0 1545642 0
43 0 3186 1975 0 1628916 0
43 0 2905 2169 0 1918250 0
46 0 3464 1859 0 1729764 0
46 0 3134 1873 0 1739662 0
and the other one
netstat -w 1 lagg1
input (Total) output
packets errs bytes packets errs bytes colls
92149 0 31706183 93523 0 32673138 0
89737 0 28119643 91323 0 28958816 0
Doing all these reports now seems to have the interfaces stuck at
packets errs bytes packets errs bytes colls
96937 0 31749525 98551 0 32678863 0
85892 0 29411078 87233 0 30182355 0
90435 0 31628680 91620 0 32215244 0
87383 0 30616741 88278 0 31026608 0
every interface on the machine is reporting the same PPS and bytes.. lol :)
So something is extremely fishy about the counters.. I'm going to try
and update to -STABLE to see if it makes any difference. It's not just
the lagg interface either because all the em's are showing it as well.
This is using 4 port Intel Server PCI Express NIC
ifstat seems to report correct usage in Kbps and seems to report correct
packet count.
Maybe it's just a netstat problem?
I will see if stable fixes it. Also, feel free to make any comments on
my config file for routing.
FreeBSD 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #6: Thu Apr 17 18:11:49
EDT 2008 amd64
UPDATE.. Changed 1 router to stable:
FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT
2008 amd64
Still see:
2 minute input rate 10463000 bits/sec, 2481 packets/sec
2 minute output rate 40075000 bits/sec, 6847 packets/sec
and
input (lagg0) output
packets errs bytes packets errs bytes colls
6940 0 5172153 4841 0 1345660 0
5922 0 4252074 3963 0 1087205 0
6673 0 4982394 4116 0 1056933 0
6659 0 4467398 4140 0 1068919 0
7085 0 4692973 4777 0 1665109 0
7140 0 4654486 4713 0 1658303 0
7070 0 4558384 5078 0 1994666 0
6375 0 4575464 4037 0 1121385 0
6257 0 3932910 4321 0 1607862 0
6504 0 4345014 4370 0 1278819 0
Hmmmm..
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0xece0-0xecff mem 0xd5ee0000-0xd5efffff,0xd5ec0000-0xd5edffff irq 17 at
device 0.0 on pci12
em0: Using MSI interrupt
em0: Ethernet address: 00:15:17:5d:18:00
em0: [FILTER]
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0xecc0-0xecdf mem 0xd5ea0000-0xd5ebffff,0xd5e80000-0xd5e9ffff irq 18 at
device 0.1 on pci12
em1: Using MSI interrupt
em1: Ethernet address: 00:15:17:5d:18:01
em1: [FILTER]
.......etc.. to em7
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-RELEASE-p1 #6: Thu Apr 17 18:11:49 EDT 2008
root@CR1.MTL3.Gtcomm.net:/usr/obj/usr/src/sys/ROUTER
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2329.28-MHz
K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
usable memory = 4285833216 (4087 MB)
avail memory = 4124545024 (3933 MB)
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
Kernel config file: (if you have any suggestions about the config let me
know for faster routing speed)
cpu HAMMER
ident GENERIC
#makeoptions DEBUG=-g # Build kernel with gdb(1) debug
symbols
options SCHED_4BSD # 4BSD scheduler
#options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big
directories
options UFS_GJOURNAL # Enable gjournal-based UFS
journaling
options MD_ROOT # MD is a potential root device
options NTFS # NT File System
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires
PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_IA32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
#options ADAPTIVE_GIANT # Giant mutex is adaptive.
options NO_ADAPTIVE_MUTEXES ##
options STOP_NMI # Stop CPUS using NMI instead of IPI
options AUDIT # Security event auditing
options IPSEC ## for tcp md5
options TCP_SIGNATURE ##include support for RFC 2385
device crypto ## for md5
device cryptodev ## for md5
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel
# CPU frequency control
device cpufreq
# Bus support.
device acpi
device pci
# Floppy drives
device fdc
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID # Static device numbering
# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
device ch # SCSI media changers
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
device ses # SCSI Environmental Services (and SAF-TE)
# RAID controllers
device mfi # LSI MegaRAID SAS
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
device agp # support several AGP chipsets
# Serial (COM) ports
device sio # 8250, 16[45]50 based serial ports
device uart # Generic UART driver
# PCI Ethernet NICs.
device em # Intel PRO/1000 adapter Gigabit
Ethernet Card
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus
device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device sl # Kernel SLIP
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
device firmware # firmware assist module
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# USB support
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device usb # USB Bus (required)
#device udbp # USB Double Bulk Pipe devices
device ugen # Generic
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse
### OPTIONS
options MP_WATCHDOG
options DEVICE_POLLING
device pf
device pflog
device pfsync
device carp
device vlan
device gre
device if_bridge
device tun
device lagg
device stf #6to4 IPv6 over IPv4 encapsulation
options ALTQ
options ALTQ_CBQ # Class Bases Queuing (CBQ)
options ALTQ_RED # Random Early Detection (RED)
options ALTQ_RIO # RED In/Out
options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC)
options ALTQ_CDNR
options ALTQ_PRIQ # Priority Queuing (PRIQ)
options ALTQ_NOPCC # Required for SMP build
options NETGRAPH
options NETGRAPH_CISCO
options NETGRAPH_FEC
options NETGRAPH_ETHER
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT 2008
root@CR2.MTL3.Gtcomm.net:/usr/obj/usr/src/sys/ROUTER
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2329.26-MHz
K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
usable memory = 4286042112 (4087 MB)
avail memory = 4124753920 (3933 MB)
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 64-87 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <DELL PE_SC3> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 728072806000728
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 728072806000728
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 728072806000728
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 728072806000728
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci6: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci6
pci7: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci7
pci8: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci8
pci9: <PCI bus> on pcib4
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xd6000000-0xd7ffffff irq 16 at device 0.0 on pci9
miibus0: <MII bus> on bce0
brgphy0: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bce0: Ethernet address: 00:19:b9:cd:60:44
bce0: [ITHREAD]
bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W
(0x02090105); Flags( MFW MSI )
pcib5: <ACPI PCI-PCI bridge> at device 1.0 on pci7
pci10: <ACPI PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 0.0 on pci10
pci11: <PCI bus> on pcib6
pcib7: <PCI-PCI bridge> at device 0.0 on pci11
pci12: <PCI bus> on pcib7
em0: <Intel(R) PRO/1000 Network Connection 6.9.0> port 0xece0-0xecff mem
0xd5ee0000-0xd5efffff,0xd5ec0000-0xd5edffff irq 17 at device 0.0 on pci12
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:15:17:5d:2a:40
em1: <Intel(R) PRO/1000 Network Connection 6.9.0> port 0xecc0-0xecdf mem
0xd5ea0000-0xd5ebffff,0xd5e80000-0xd5e9ffff irq 18 at device 0.1 on pci12
em1: Using MSI interrupt
em1: [FILTER]
em1: Ethernet address: 00:15:17:5d:2a:41
pcib8: <PCI-PCI bridge> at device 1.0 on pci11
pci13: <PCI bus> on pcib8
em2: <Intel(R) PRO/1000 Network Connection 6.9.0> port 0xdce0-0xdcff mem
0xd5ce0000-0xd5cfffff,0xd5cc0000-0xd5cdffff irq 18 at device 0.
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
ether 00:15:17:5d:2a:40
media: Ethernet autoselect
status: active
laggproto lacp
laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
ether 00:15:17:5d:28:62
inet netmask 0xfffffffc broadcast
media: Ethernet autoselect
status: active
laggproto lacp
laggport: em7 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em6 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
lagg2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
ether 00:15:17:5d:28:60
media: Ethernet autoselect
status: active
laggproto lacp
laggport: em5 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em4 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4827E79C.8050608>