Date: Mon, 24 Oct 2011 15:52:43 -0400 From: Andrew Boyer <aboyer@averesystems.com> To: Sergey Saley <sergeysaley@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Too much interrupts on ixgbe Message-ID: <C2323025-E6EC-491F-9B03-80FFCBDFBA5E@averesystems.com> In-Reply-To: <1319449307149-4931883.post@n5.nabble.com> References: <1319449307149-4931883.post@n5.nabble.com>
next in thread | previous in thread | raw e-mail | index | archive | help
You could try this patch. It disables the interrupt while it's being = handled. (The driver already re-enables it at the end of the handler = and the task.) -Andrew Index: sys/dev/ixgbe/ixgbe.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sys/dev/ixgbe/ixgbe.c (revision 226698) +++ sys/dev/ixgbe/ixgbe.c (working copy) @@ -1362,6 +1362,7 @@ bool more_tx, more_rx; u32 newitr =3D 0; =20 + ixgbe_disable_queue(adapter, que->msix); ++que->irqs; =20 more_rx =3D ixgbe_rxeof(que, adapter->rx_process_limit); On Oct 24, 2011, at 5:41 AM, Sergey Saley wrote: > There is my FreeBSD box: >=20 > kernel > --------------- > # > # GENERIC -- Generic kernel configuration file for FreeBSD/i386 > # > # For more information on this file, please read the config(5) manual = page, > # and/or the handbook section on Kernel Configuration Files: > # > # =20 > = http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-con= fig.html > # > # The handbook is also available locally in /usr/share/doc/handbook > # if you've installed the doc distribution, otherwise always see the > # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the > # latest information. > # > # An exhaustive list of options and more detailed explanations of the > # device lines is also present in the ../../conf/NOTES and NOTES = files. > # If you are in doubt as to the purpose or necessity of a line, check = first > # in NOTES. > # > # $FreeBSD: head/sys/i386/conf/GENERIC 221743 2011-05-10 16:44:16Z = jkim $ >=20 > cpu I686_CPU > ident POINT07 >=20 >=20 > options SCHED_ULE # ULE scheduler > options PREEMPTION # Enable kernel thread = preemption > options INET # InterNETworking > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates = support > options UFS_DIRHASH # Improve performance on big = directories > options MD_ROOT # MD is a potential root device > options MSDOSFS # MSDOS Filesystem > options CD9660 # ISO 9660 Filesystem > options PROCFS # Process filesystem (requires = PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options KTRACE # ktrace(1) support > options STACK # stack(9) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions > options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) > #options KDTRACE_HOOKS # Kernel DTrace hooks > options INCLUDE_CONFIG_FILE # Include this file in kernel >=20 >=20 > # To make an SMP kernel, the next two lines are needed > options SMP # Symmetric MultiProcessor = Kernel > device apic # I/O APIC >=20 > # CPU frequency control > device cpufreq >=20 > # Bus support. > device acpi > device pci >=20 > # Floppy drives > device fdc >=20 > # ATA controllers > device ahci # AHCI-compatible SATA = controllers > device ata # Legacy ATA/SATA controllers > options ATA_CAM # Handle legacy controllers with CAM > options ATA_STATIC_ID # Static device numbering > device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA > device siis # SiliconImage = SiI3124/SiI3132/SiI3531 SATA >=20 > # ATA/SCSI peripherals > device scbus # SCSI bus (required for = ATA/SCSI) > device ch # SCSI media changers > device da # Direct Access (disks) > device sa # Sequential Access (tape etc) > device cd # CD > device pass # Passthrough device (direct = ATA/SCSI access) > device ses # SCSI Environmental Services = (and SAF-TE) >=20 >=20 > # atkbdc0 controls both the keyboard and the PS/2 mouse > device atkbdc # AT keyboard controller > device atkbd # AT keyboard > device psm # PS/2 mouse >=20 > device kbdmux # keyboard multiplexer >=20 > device vga # VGA video card driver >=20 >=20 > # syscons is the default console driver, resembling an SCO console > device sc > options SC_PIXEL_MODE # add support for the raster text mode >=20 > device agp # support several AGP chipsets >=20 > # Power management support (see NOTES for more options) > #device apm > # Add suspend/resume support for the i8254. > device pmtimer >=20 > # Serial (COM) ports > device uart # Generic UART driver >=20 > # Parallel port > device ppc > device ppbus # Parallel port bus (required) > device lpt # Printer > #device plip # TCP/IP over parallel > device ppi # Parallel port interface device > #device vpo # Requires scbus and da >=20 > # If you've got a "dumb" serial or parallel PCI card that is > # supported by the puc(4) glue driver, uncomment the following > # line to enable it (connects to sio, uart and/or ppc drivers): > #device puc >=20 > # PCI Ethernet NICs. > device em # Intel PRO/1000 Gigabit = Ethernet Family > device ixgbe # Intel PRO/10GbE Ethernet Card >=20 > # Pseudo devices. > device loop # Network loopback > device random # Entropy device > device ether # Ethernet support > device vlan # 802.1Q VLAN support > #device tun # Packet tunnel. > device pty # BSD-style compatibility pseudo = ttys > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device faith # IPv6-to-IPv4 relaying = (translation) > device firmware # firmware assist module >=20 > # The `bpf' device enables the Berkeley Packet Filter. > # Be aware of the administrative consequences of enabling this! > # Note that 'bpf' is required for DHCP. > device bpf # Berkeley packet filter >=20 >=20 > options IPFIREWALL > options IPFIREWALL_DEFAULT_TO_ACCEPT > options IPFIREWALL_FORWARD > options DUMMYNET > options IPI_PREEMPTION >=20 > # netgraph(4). Enable the base netgraph code with the NETGRAPH option. > # Individual node types can be enabled with the corresponding option > # listed below; however, this is not strictly necessary as netgraph > # will automatically load the corresponding KLD module if the node = type > # is not already compiled into the kernel. Each type below has a > # corresponding man page, e.g., ng_async(8). >=20 > options NETGRAPH # netgraph(4) system >=20 > options NETGRAPH_CAR > options NETGRAPH_IFACE > options NETGRAPH_MPPC_ENCRYPTION > options NETGRAPH_PPP > options NETGRAPH_PPPOE > options NETGRAPH_PPTPGRE > options NETGRAPH_SOCKET > options NETGRAPH_TCPMSS > options NETGRAPH_ETHER > options NETGRAPH_TEE > options NETGRAPH_VJC > options NETGRAPH_BPF > options NETGRAPH_KSOCKET > options NETGRAPH_IPFW >=20 > dmesg.boot > -------------------------- > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, = 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-BETA3 #1: Fri Oct 21 10:37:43 FET 2011 > root@point07.uch.net:/usr/obj/usr/src/sys/POINT07 amd64 > CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2394.05-MHz = K8-class > CPU) > Origin =3D "GenuineIntel" Id =3D 0x6fb Family =3D 6 Model =3D f = Stepping =3D 11 >=20 > = Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE= ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > = Features2=3D0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM= > > AMD Features=3D0x20100800<SYSCALL,NX,LM> > AMD Features2=3D0x1<LAHF> > TSC: P-state invariant, performance statistics > real memory =3D 2147483648 (2048 MB) > avail memory =3D 2056187904 (1960 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: <PTLTD APIC > > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 1 package(s) x 4 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > ioapic0 <Version 2.0> irqs 0-23 on motherboard > kbd1 at kbdmux0 > acpi0: <PTLTD XSDT> on motherboard > acpi0: Power Button (fixed) > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 > cpu0: <ACPI CPU> on acpi0 > cpu1: <ACPI CPU> on acpi0 > cpu2: <ACPI CPU> on acpi0 > cpu3: <ACPI CPU> on acpi0 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 > pci1: <ACPI PCI bus> on pcib1 > ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11> = port > 0x2000-0x201f mem 0xdc680000-0xdc6fffff,0xdc600000-0xdc603fff irq 16 = at > device 0.0 on pci1 > ix0: Using MSIX interrupts with 5 vectors > ix0: Ethernet address: 00:25:90:3c:42:6a > ix0: PCI Express Bus: Speed 2.5Gb/s Width x8 > ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11> = port > 0x2020-0x203f mem 0xdc700000-0xdc77ffff,0xdc604000-0xdc607fff irq 17 = at > device 0.1 on pci1 > ix1: Using MSIX interrupts with 5 vectors > ix1: RX Descriptors exceed system mbuf max, using default instead! > ix1: Ethernet address: 00:25:90:3c:42:6b > ix1: PCI Express Bus: Speed 2.5Gb/s Width x8 > pci0: <serial bus, USB> at device 26.0 (no driver attached) > pci0: <serial bus, USB> at device 26.1 (no driver attached) > pci0: <serial bus, USB> at device 26.2 (no driver attached) > pci0: <serial bus, USB> at device 26.7 (no driver attached) > pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 > pci5: <ACPI PCI bus> on pcib2 > pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 > pci13: <ACPI PCI bus> on pcib3 > em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x3000-0x301f = mem > 0xdc100000-0xdc11ffff irq 16 at device 0.0 on pci13 > em0: Using an MSI interrupt > em0: Ethernet address: 00:30:48:d2:12:ac > pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0 > pci15: <ACPI PCI bus> on pcib4 > em1: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x4000-0x401f = mem > 0xdc200000-0xdc21ffff irq 17 at device 0.0 on pci15 > em1: Using an MSI interrupt > em1: Ethernet address: 00:30:48:d2:12:ad > pci0: <serial bus, USB> at device 29.0 (no driver attached) > pci0: <serial bus, USB> at device 29.1 (no driver attached) > pci0: <serial bus, USB> at device 29.2 (no driver attached) > pci0: <serial bus, USB> at device 29.7 (no driver attached) > pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0 > pci17: <ACPI PCI bus> on pcib5 > vgapci0: <VGA-compatible display> port 0x5000-0x507f mem > 0xde000000-0xdfffffff,0xdc300000-0xdc33ffff at device 4.0 on pci17 > isab0: <PCI-ISA bridge> at device 31.0 on pci0 > isa0: <ISA bus> on isab0 > atapci0: <Intel ICH9 SATA300 controller> port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1c10-0x1c1f,0x1c00-0x1c0f at = device > 31.2 on pci0 > ata0: <ATA channel 0> on atapci0 > ata1: <ATA channel 1> on atapci0 > pci0: <serial bus, SMBus> at device 31.3 (no driver attached) > atapci1: <Intel ICH9 SATA300 controller> port > = 0x1c68-0x1c6f,0x1c5c-0x1c5f,0x1c60-0x1c67,0x1c58-0x1c5b,0x1c30-0x1c3f,0x1c= 20-0x1c2f > irq 18 at device 31.5 on pci0 > ata2: <ATA channel 0> on atapci1 > ata3: <ATA channel 1> on atapci1 > pci0: <dasp> at device 31.6 (no driver attached) > acpi_button0: <Power Button> on acpi0 > atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 > atkbd0: <AT Keyboard> irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on = acpi0 > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 > fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on = acpi0 > ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on = acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppbus0: <Parallel port bus> on ppc0 > lpt0: <Printer> on ppbus0 > lpt0: Interrupt-driven port > ppi0: <Parallel I/O> on ppbus0 > sc0: <System console> at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=3D0x300> > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 > est0: <Enhanced SpeedStep Frequency Control> on cpu0 > p4tcc0: <CPU Frequency Thermal Control> on cpu0 > est1: <Enhanced SpeedStep Frequency Control> on cpu1 > p4tcc1: <CPU Frequency Thermal Control> on cpu1 > est2: <Enhanced SpeedStep Frequency Control> on cpu2 > p4tcc2: <CPU Frequency Thermal Control> on cpu2 > est3: <Enhanced SpeedStep Frequency Control> on cpu3 > p4tcc3: <CPU Frequency Thermal Control> on cpu3 > Timecounters tick every 1.000 msec > ipfw2 initialized, divert loadable, nat loadable, rule-based = forwarding > enabled, default to accept, logging disabled > DUMMYNET 0 with IPv6 initialized (100409) > load_dn_sched dn_sched QFQ loaded > load_dn_sched dn_sched RR loaded > load_dn_sched dn_sched WF2Q+ loaded > load_dn_sched dn_sched FIFO loaded > load_dn_sched dn_sched PRIO loaded > ada0 at ata0 bus 0 scbus0 target 0 lun 0 > ada0: <SAMSUNG HD161HJ GF100-07> ATA-8 SATA 2.x device > ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) > ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) > ada0: Previously was known as ad0 > SMP: AP CPU #3 Launched! > SMP: AP CPU #1 Launched! > SMP: AP CPU #2 Launched! > Timecounter "TSC-low" frequency 9351750 Hz quality 1000 > Trying to mount root from ufs:/dev/ada0p2 [rw]... >=20 > sysctl.conf > -------------- > net.inet.ip.fastforwarding=3D1 > net.isr.direct=3D0 > net.isr.direct_force=3D0 > net.inet.ip.fw.one_pass=3D1 > hw.intr_storm_threshold=3D9000 > kern.ipc.nmbclusters=3D262144 > kern.ipc.nmbjumbop=3D262144 > dev.ix.0.rx_processing_limit=3D4096 > dev.ix.1.rx_processing_limit=3D4096 > net.inet.ip.intr_queue_maxlen=3D256 > dev.ix.0.flow_control=3D0 > dev.ix.1.flow_control=3D0 >=20 > vmstat -i > ------------- > # vmstat -i > interrupt total rate > irq1: atkbd0 1 0 > irq6: fdc0 1 0 > irq14: ata0 80855 9 > irq15: ata1 1 0 > cpu0:timer 9694291 1122 > irq256: ix0:que 0 54169187 6273 > irq257: ix0:que 1 19307922 2236 > irq258: ix0:que 2 24265573 2810 > irq259: ix0:que 3 25845865 2993 > irq260: ix0:link 2 0 > irq261: ix1:que 0 63210220 7321 > irq262: ix1:que 1 265895 30 > irq263: ix1:que 2 42114 4 > irq264: ix1:que 3 98953 11 > irq265: ix1:link 5 0 > cpu3:timer 7165968 829 > cpu1:timer 7386880 855 > cpu2:timer 7307920 846 > Total 218841653 25346 >=20 > netstat > ------------ > # netstat -I ix0 -w1 > input (ix0) output > packets errs idrops bytes packets errs bytes colls > 26592 0 0 30220498 20757 0 11490639 0 > 31232 0 0 34001150 24699 0 14554631 0 > 27785 0 0 29672129 23699 0 16154542 0 > 25525 0 0 27450183 22868 0 16180216 0 > ^C > # netstat -I ix1 -w1 > input (ix1) output > packets errs idrops bytes packets errs bytes colls > 22398 0 0 17232635 23781 0 24718205 0 > 23671 0 0 17608589 25859 0 28970777 0 > 24096 0 0 17278557 27188 0 30804019 0 > 22043 0 0 16271250 24116 0 26238423 0 > ^C >=20 >=20 > top -SCHP > ----------------- > last pid: 74628; load averages: 1.26, 1.01, 0.89 = = =20 > up 0+02:25:14 13:22:11 > 110 processes: 9 running, 79 sleeping, 22 waiting > CPU 0: 0.4% user, 0.0% nice, 3.1% system, 44.1% interrupt, 52.4% = idle > CPU 1: 0.0% user, 0.0% nice, 3.1% system, 13.4% interrupt, 83.5% = idle > CPU 2: 0.8% user, 0.0% nice, 2.4% system, 18.5% interrupt, 78.3% = idle > CPU 3: 0.4% user, 0.0% nice, 4.7% system, 13.0% interrupt, 81.9% = idle > Mem: 44M Active, 118M Inact, 326M Wired, 792K Cache, 213M Buf, 1487M = Free > Swap: 4096M Total, 4096M Free >=20 > PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND > 10 root 155 ki31 0K 64K RUN 2 130:32 94.68% = idle{idle: > cpu2} > 10 root 155 ki31 0K 64K RUN 1 130:49 93.07% = idle{idle: > cpu1} > 10 root 155 ki31 0K 64K CPU3 3 132:48 89.26% = idle{idle: > cpu3} > 10 root 155 ki31 0K 64K RUN 0 101:00 58.89% = idle{idle: > cpu0} > 11 root -92 - 0K 416K CPU0 0 30:52 36.77% = intr{irq261: > ix1:que } > 11 root -92 - 0K 416K CPU3 3 8:03 11.77% = intr{irq259: > ix0:que } > 11 root -92 - 0K 416K CPU1 1 5:37 9.86% = intr{irq257: > ix0:que } > 11 root -92 - 0K 416K WAIT 2 8:14 9.28% = intr{irq258: > ix0:que } > 11 root -92 - 0K 416K RUN 0 7:07 5.18% = intr{irq256: > ix0:que } > 0 root -92 0 0K 352K - 2 1:12 1.17% = kernel{ix1 > que} > 12 root -16 - 0K 64K sleep 0 1:02 1.07% > ng_queue{ng_queue1} > 12 root -16 - 0K 64K sleep 0 1:02 0.88% > ng_queue{ng_queue0} > 12 root -16 - 0K 64K sleep 2 1:02 0.78% > ng_queue{ng_queue3} > 12 root -16 - 0K 64K sleep 0 1:02 0.78% > ng_queue{ng_queue2} > 0 root -92 0 0K 352K - 3 0:58 0.39% = kernel{ix0 > que} > 11 root -60 - 0K 416K WAIT 3 5:21 0.00% = intr{swi4: > clock} > 0 root -16 0 0K 352K sched 1 0:54 0.00% > kernel{swapper} > 14320 root 20 0 78528K 32900K select 1 0:52 0.00% = mpd5{mpd5} > 0 root -92 0 0K 352K - 0 0:28 0.00% = kernel{ix0 > que} > 0 root -92 0 0K 352K - 2 0:25 0.00% > kernel{dummynet} > 2382 root 20 0 18832K 4060K select 3 0:21 0.00% zebra > 0 root -92 0 0K 352K - 2 0:17 0.00% = kernel{ix0 > que} > 14 root -16 - 0K 16K - 2 0:12 0.00% yarrow > 0 root -92 0 0K 352K - 3 0:07 0.00% = kernel{ix0 > que} > 2388 root 20 0 25632K 6900K select 3 0:06 0.00% ospfd > 19419 www 20 0 16332K 5440K kqread 1 0:05 0.00% thttpd > 3 root -16 - 0K 16K ccb_sc 0 0:04 0.00% xpt_thrd > 9 root 16 - 0K 16K syncer 2 0:02 0.00% syncer > 0 root -92 0 0K 352K - 1 0:02 0.00% = kernel{ix1 > linkq} > 3846 root 20 0 22340K 3452K select 1 0:02 0.00% ntpd > 4005 root 20 0 68024K 5976K select 1 0:01 0.00% sshd > 11 root -92 - 0K 416K WAIT 1 0:01 0.00% = intr{irq262: > ix1:que } > 11 root -68 - 0K 416K WAIT 2 0:01 0.00% = intr{swi2: > cambio} > 13 root -8 - 0K 48K - 3 0:01 0.00% = geom{g_up} > 57732 root 20 0 32136K 2920K uwait 1 0:01 0.00% > collectd{collectd} > 11 root -88 - 0K 416K WAIT 3 0:01 0.00% = intr{irq14: > ata0} > 3576 root 20 0 12192K 1716K select 2 0:01 0.00% syslogd > 81248 root 20 0 68024K 5708K select 1 0:01 0.00% sshd > 13 root -8 - 0K 48K - 0 0:01 0.00% = geom{g_down} > 11 root -92 - 0K 416K WAIT 3 0:01 0.00% = intr{irq264: > ix1:que } > 15 root -16 - 0K 16K sdflus 3 0:00 0.00% = softdepflush > 11 root -60 - 0K 416K WAIT 2 0:00 0.00% = intr{swi4: > clock} >=20 > pciconf -lvc > --------------- > ix0@pci0:1:0:0: class=3D0x020000 card=3D0x061115d9 chip=3D0x10fb8086 = rev=3D0x01 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D '82599EB 10-Gigabit SFI/SFP+ Network Connection' > class =3D network > subclass =3D ethernet > cap 01[40] =3D powerspec 3 supports D0 D3 current D0 > cap 05[50] =3D MSI supports 1 message, 64 bit, vector masks > cap 11[70] =3D MSI-X supports 64 messages in map 0x20 enabled > cap 10[a0] =3D PCI-Express 2 endpoint max data 128(512) link x8(x8) > cap 03[e0] =3D VPD > ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] =3D Serial 1 002590ffff3c426a > ecap 000e[150] =3D unknown 1 > ecap 0010[160] =3D unknown 1 > ix1@pci0:1:0:1: class=3D0x020000 card=3D0x061115d9 chip=3D0x10fb8086 = rev=3D0x01 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D '82599EB 10-Gigabit SFI/SFP+ Network Connection' > class =3D network > subclass =3D ethernet > cap 01[40] =3D powerspec 3 supports D0 D3 current D0 > cap 05[50] =3D MSI supports 1 message, 64 bit, vector masks > cap 11[70] =3D MSI-X supports 64 messages in map 0x20 enabled > cap 10[a0] =3D PCI-Express 2 endpoint max data 128(512) link x8(x8) > cap 03[e0] =3D VPD > ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] =3D Serial 1 002590ffff3c426a > ecap 000e[150] =3D unknown 1 > ecap 0010[160] =3D unknown 1 >=20 > The question is: > ----------------- > Why there are so much interrupts? > Why the card is using only one intr on second port (ix1)? >=20 > I've got another server with 82598EB 10 Gigabit AT CX4 Network = Connection > and the situation on it is dramatically different: >=20 > last pid: 42234; load averages: 2.14, 1.65, 1.55 = = =20 > up 119+05:58:58 13:26:01 > 98 processes: 8 running, 62 sleeping, 28 waiting > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 34.0% interrupt, 66.0% = idle > CPU 1: 0.0% user, 0.0% nice, 0.0% system, 43.4% interrupt, 56.6% = idle > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 39.6% interrupt, 60.4% = idle > CPU 3: 0.0% user, 0.0% nice, 0.0% system, 35.8% interrupt, 64.2% = idle > Mem: 279M Active, 294M Inact, 300M Wired, 132K Cache, 112M Buf, 2128M = Free > Swap: 4096M Total, 4096M Free >=20 > PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND > 10 root 171 ki31 0K 32K RUN 0 2159.3 70.46% {idle: = cpu0} > 10 root 171 ki31 0K 32K RUN 2 2166.8 69.48% {idle: = cpu2} > 10 root 171 ki31 0K 32K RUN 3 2176.3 68.90% {idle: = cpu3} > 10 root 171 ki31 0K 32K RUN 1 2163.1 68.55% {idle: = cpu1} > 11 root -68 - 0K 248K CPU3 3 335.0H 19.38% {irq259: > ix0:que } > 11 root -68 - 0K 248K WAIT 1 335.5H 17.97% {irq257: > ix0:que } > 11 root -68 - 0K 248K CPU2 2 335.3H 17.77% {irq258: > ix0:que } > 11 root -68 - 0K 248K CPU0 0 336.2H 17.58% {irq256: > ix0:que } > 11 root -68 - 0K 248K WAIT 3 323.4H 17.09% {irq264: > ix1:que } > 11 root -68 - 0K 248K WAIT 1 323.9H 16.55% {irq262: > ix1:que } > 11 root -68 - 0K 248K WAIT 2 325.6H 16.06% {irq263: > ix1:que } > 11 root -68 - 0K 248K WAIT 0 325.9H 15.77% {irq261: > ix1:que } >=20 > vmstat -i > ------------- > r# vmstat -i > interrupt total rate > irq19: atapci0 1667110 0 > cpu0: timer 3386721810 328 > irq256: ix0:que 0 3395843376 329 > irq257: ix0:que 1 2642665824 256 > irq258: ix0:que 2 2838302235 275 > irq259: ix0:que 3 2176207954 211 > irq260: ix0:link 18 0 > irq261: ix1:que 0 282359321 27 > irq262: ix1:que 1 3989170496 387 > irq263: ix1:que 2 375956573 36 > irq264: ix1:que 3 3966352151 384 > irq265: ix1:link 1 0 > irq266: em0:rx 0 10283114 0 > irq269: em1:rx 0 10283114 0 > cpu3: timer 3386697130 328 > cpu1: timer 3386709028 328 > cpu2: timer 3386700908 328 > Total 33235920163 3225 >=20 > netstat=20 > ------------ > # netstat -I ix0 -w1 > input (ix0) output > packets errs idrops bytes packets errs bytes colls > 264572 0 0 252383894 253290 0 178845781 0 > 266323 0 0 254825280 251277 0 173769218 0 > 274462 0 0 265247306 260819 0 181113304 0 > 271142 0 0 263263325 253941 0 171694438 0 > ^C > # netstat -I ix1 -w1 > input (ix1) output > packets errs idrops bytes packets errs bytes colls > 259427 0 0 183038665 275711 0 262429749 0 > 248177 0 0 171479966 264441 0 252993752 0 > 255271 0 0 185006000 266886 0 247494148 0 > 264543 0 0 190970875 275087 0 259555066 0 > ^C >=20 > Tuning on both systems are almost the same > As You can see, 82598EB card produce about 20 times less interrupts at = about > 10 times more pps. >=20 > Please help me to fix the problem... >=20 > -- > View this message in context: = http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp493188= 3p4931883.html > Sent from the freebsd-net mailing list archive at Nabble.com. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -------------------------------------------------- Andrew Boyer aboyer@averesystems.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C2323025-E6EC-491F-9B03-80FFCBDFBA5E>