Date: Mon, 12 Mar 2012 18:49:40 +0200 From: =?utf-8?B?0JrQvtC90YzQutC+0LIg0JXQstCz0LXQvdC40Lk=?= <kes-kes@yandex.ru> To: "Steven Hartland" <killing@multiplay.co.uk> Cc: freebsd-net@freebsd.org Subject: Re: ixgbe interface micro stalls / slow responses Message-ID: <853549959.20120312184940@yandex.ru> In-Reply-To: <18969B34B4EE402986389D5022DCCB35@multiplay.co.uk> References: <18969B34B4EE402986389D5022DCCB35@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Здравствуйте, Steven. Вы писали 12 марта 2012 г., 17:05:12: SH> We've got a machine where with an ix interface on 8.2-RELEASE SH> which is seeing intermittent slow responses. It shows as stalls SH> on the console and is visible as high pings on an mtr SH> from a local machine e.g. SH> Packets Pings SH> Host Loss% Snt Last Avg Best Wrst StDev SH> 1. X.X.X.X 0.0% 181 0.1 117.7 0.1 2665. 314.8 SH> We've tried updating from 2.3.10 release driver + alias fix SH> to 2.4.5 (the latest from 8.3) but still the behavour is the SH> same. SH> If we do a trace to an igb on the same machine everything is SH> clean. SH> Packets Pings SH> Host Loss% Snt Last Avg Best Wrst StDev SH> 1. 10.10.10.64 0.0% 136 0.1 0.2 0.1 12.5 1.1 SH> We are seeing "RX Descriptors exceed system mbuf max, using SH> default instead!" on boot with the latest driver but the fix SH> listed in the readme has no effect, as in sysctl.conf we have SH> kern.ipc.nmbclusters=524288 SH> kern.ipc.nmbjumbop=262144 SH> Nothing looks out of the ordinary by there's definitely a SH> problem there somewhere, any ideas? SH> Detailed info which may be use below. >>From dmeg:- SH> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5> port 0x2000-0x201f mem SH> 0xd8400000-0xd847ffff,0xd8480000-0xd8483fff irq 52 at device 0.0 on pci5 SH> ix0: Using MSIX interrupts with 9 vectors SH> ix0: RX Descriptors exceed system mbuf max, using default instead! SH> ix0: [ITHREAD] SH> ix0: [ITHREAD] SH> ix0: [ITHREAD] SH> ix0: [ITHREAD] SH> ix0: [ITHREAD] SH> ix0: [ITHREAD] SH> ix0: [ITHREAD] SH> ix0: [ITHREAD] SH> ix0: [ITHREAD] SH> ix0: Ethernet address: 00:1b:21:7e:2e:8c SH> ix0: PCI Express Bus: Speed 5.0Gb/s Width x8 SH> pciconf -v -l SH> ix0@pci0:5:0:0: class=0x020000 card=0x00068086 chip=0x10fb8086 rev=0x01 hdr=0x00 SH> vendor = 'Intel Corporation' SH> class = network SH> subclass = ethernet SH> sysctl dev.ix SH> dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5 SH> dev.ix.0.%driver: ix SH> dev.ix.0.%location: slot=0 function=0 SH> dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10fb subvendor=0x8086 subdevice=0x0006 class=0x020000 SH> dev.ix.0.%parent: pci5 SH> dev.ix.0.fc: 3 SH> dev.ix.0.advertise_gig: 0 SH> dev.ix.0.enable_aim: 1 SH> dev.ix.0.advertise_speed: 0 SH> dev.ix.0.rx_processing_limit: 128 SH> dev.ix.0.dropped: 0 SH> dev.ix.0.mbuf_defrag_failed: 0 SH> dev.ix.0.no_tx_dma_setup: 0 SH> dev.ix.0.watchdog_events: 0 SH> dev.ix.0.tso_tx: 174470 SH> dev.ix.0.link_irq: 3 SH> dev.ix.0.queue0.interrupt_rate: 1000000 SH> dev.ix.0.queue0.txd_head: 59 SH> dev.ix.0.queue0.txd_tail: 59 SH> dev.ix.0.queue0.no_desc_avail: 0 SH> dev.ix.0.queue0.tx_packets: 38913 SH> dev.ix.0.queue0.rxd_head: 384 SH> dev.ix.0.queue0.rxd_tail: 383 SH> dev.ix.0.queue0.rx_packets: 54982 SH> dev.ix.0.queue0.rx_bytes: 36197485 SH> dev.ix.0.queue0.lro_queued: 0 SH> dev.ix.0.queue0.lro_flushed: 0 SH> dev.ix.0.queue1.interrupt_rate: 1000000 SH> dev.ix.0.queue1.txd_head: 1417 SH> dev.ix.0.queue1.txd_tail: 1417 SH> dev.ix.0.queue1.no_desc_avail: 0 SH> dev.ix.0.queue1.tx_packets: 51196 SH> dev.ix.0.queue1.rxd_head: 445 SH> dev.ix.0.queue1.rxd_tail: 444 SH> dev.ix.0.queue1.rx_packets: 70841 SH> dev.ix.0.queue1.rx_bytes: 26319740 SH> dev.ix.0.queue1.lro_queued: 0 SH> dev.ix.0.queue1.lro_flushed: 0 SH> dev.ix.0.queue2.interrupt_rate: 20408 SH> dev.ix.0.queue2.txd_head: 194 SH> dev.ix.0.queue2.txd_tail: 194 SH> dev.ix.0.queue2.no_desc_avail: 0 SH> dev.ix.0.queue2.tx_packets: 45102 SH> dev.ix.0.queue2.rxd_head: 696 SH> dev.ix.0.queue2.rxd_tail: 695 SH> dev.ix.0.queue2.rx_packets: 65107 SH> dev.ix.0.queue2.rx_bytes: 49222403 SH> dev.ix.0.queue2.lro_queued: 0 SH> dev.ix.0.queue2.lro_flushed: 0 SH> dev.ix.0.queue3.interrupt_rate: 200000 SH> dev.ix.0.queue3.txd_head: 1605 SH> dev.ix.0.queue3.txd_tail: 1605 SH> dev.ix.0.queue3.no_desc_avail: 0 SH> dev.ix.0.queue3.tx_packets: 77375 SH> dev.ix.0.queue3.rxd_head: 79 SH> dev.ix.0.queue3.rxd_tail: 78 SH> dev.ix.0.queue3.rx_packets: 109498 SH> dev.ix.0.queue3.rx_bytes: 109951775 SH> dev.ix.0.queue3.lro_queued: 0 SH> dev.ix.0.queue3.lro_flushed: 0 SH> dev.ix.0.queue4.interrupt_rate: 10526 SH> dev.ix.0.queue4.txd_head: 1624 SH> dev.ix.0.queue4.txd_tail: 1624 SH> dev.ix.0.queue4.no_desc_avail: 0 SH> dev.ix.0.queue4.tx_packets: 39497 SH> dev.ix.0.queue4.rxd_head: 480 SH> dev.ix.0.queue4.rxd_tail: 479 SH> dev.ix.0.queue4.rx_packets: 51998 SH> dev.ix.0.queue4.rx_bytes: 21965859 SH> dev.ix.0.queue4.lro_queued: 0 SH> dev.ix.0.queue4.lro_flushed: 0 SH> dev.ix.0.queue5.interrupt_rate: 1000000 SH> dev.ix.0.queue5.txd_head: 1613 SH> dev.ix.0.queue5.txd_tail: 1613 SH> dev.ix.0.queue5.no_desc_avail: 0 SH> dev.ix.0.queue5.tx_packets: 69860 SH> dev.ix.0.queue5.rxd_head: 846 SH> dev.ix.0.queue5.rxd_tail: 845 SH> dev.ix.0.queue5.rx_packets: 81331 SH> dev.ix.0.queue5.rx_bytes: 32429926 SH> dev.ix.0.queue5.lro_queued: 0 SH> dev.ix.0.queue5.lro_flushed: 0 SH> dev.ix.0.queue6.interrupt_rate: 142857 SH> dev.ix.0.queue6.txd_head: 1482 SH> dev.ix.0.queue6.txd_tail: 1484 SH> dev.ix.0.queue6.no_desc_avail: 0 SH> dev.ix.0.queue6.tx_packets: 45878 SH> dev.ix.0.queue6.rxd_head: 355 SH> dev.ix.0.queue6.rxd_tail: 354 SH> dev.ix.0.queue6.rx_packets: 62211 SH> dev.ix.0.queue6.rx_bytes: 27653559 SH> dev.ix.0.queue6.lro_queued: 0 SH> dev.ix.0.queue6.lro_flushed: 0 SH> dev.ix.0.queue7.interrupt_rate: 5347 SH> dev.ix.0.queue7.txd_head: 603 SH> dev.ix.0.queue7.txd_tail: 603 SH> dev.ix.0.queue7.no_desc_avail: 0 SH> dev.ix.0.queue7.tx_packets: 61997 SH> dev.ix.0.queue7.rxd_head: 826 SH> dev.ix.0.queue7.rxd_tail: 825 SH> dev.ix.0.queue7.rx_packets: 83460 SH> dev.ix.0.queue7.rx_bytes: 50183116 SH> dev.ix.0.queue7.lro_queued: 0 SH> dev.ix.0.queue7.lro_flushed: 0 SH> dev.ix.0.mac_stats.crc_errs: 0 SH> dev.ix.0.mac_stats.ill_errs: 0 SH> dev.ix.0.mac_stats.byte_errs: 0 SH> dev.ix.0.mac_stats.short_discards: 0 SH> dev.ix.0.mac_stats.local_faults: 3 SH> dev.ix.0.mac_stats.remote_faults: 1 SH> dev.ix.0.mac_stats.rec_len_errs: 0 SH> dev.ix.0.mac_stats.link_xon_txd: 0 SH> dev.ix.0.mac_stats.link_xon_rcvd: 0 SH> dev.ix.0.mac_stats.link_xoff_txd: 0 SH> dev.ix.0.mac_stats.link_xoff_rcvd: 0 SH> dev.ix.0.mac_stats.total_octets_rcvd: 360072702 SH> dev.ix.0.mac_stats.good_octets_rcvd: 359999778 SH> dev.ix.0.mac_stats.total_pkts_rcvd: 637428 SH> dev.ix.0.mac_stats.good_pkts_rcvd: 636321 SH> dev.ix.0.mac_stats.mcast_pkts_rcvd: 35 SH> dev.ix.0.mac_stats.bcast_pkts_rcvd: 1411 SH> dev.ix.0.mac_stats.rx_frames_64: 222251 SH> dev.ix.0.mac_stats.rx_frames_65_127: 159044 SH> dev.ix.0.mac_stats.rx_frames_128_255: 15139 SH> dev.ix.0.mac_stats.rx_frames_256_511: 13885 SH> dev.ix.0.mac_stats.rx_frames_512_1023: 21283 SH> dev.ix.0.mac_stats.rx_frames_1024_1522: 204719 SH> dev.ix.0.mac_stats.recv_undersized: 0 SH> dev.ix.0.mac_stats.recv_fragmented: 0 SH> dev.ix.0.mac_stats.recv_oversized: 0 SH> dev.ix.0.mac_stats.recv_jabberd: 0 SH> dev.ix.0.mac_stats.management_pkts_rcvd: 0 SH> dev.ix.0.mac_stats.management_pkts_drpd: 0 SH> dev.ix.0.mac_stats.checksum_errs: 0 SH> dev.ix.0.mac_stats.good_octets_txd: 882467530 SH> dev.ix.0.mac_stats.total_pkts_txd: 816387 SH> dev.ix.0.mac_stats.good_pkts_txd: 816387 SH> dev.ix.0.mac_stats.bcast_pkts_txd: 36 SH> dev.ix.0.mac_stats.mcast_pkts_txd: 0 SH> dev.ix.0.mac_stats.management_pkts_txd: 0 SH> dev.ix.0.mac_stats.tx_frames_64: 21509 SH> dev.ix.0.mac_stats.tx_frames_65_127: 168051 SH> dev.ix.0.mac_stats.tx_frames_128_255: 19184 SH> dev.ix.0.mac_stats.tx_frames_256_511: 22775 SH> dev.ix.0.mac_stats.tx_frames_512_1023: 24222 SH> dev.ix.0.mac_stats.tx_frames_1024_1522: 560646 SH> dev.ix.0.mac_stats.fc_crc: 0 SH> dev.ix.0.mac_stats.fc_last: 0 SH> dev.ix.0.mac_stats.fc_drpd: 0 SH> dev.ix.0.mac_stats.fc_pkts_rcvd: 0 SH> dev.ix.0.mac_stats.fc_pkts_txd: 0 SH> dev.ix.0.mac_stats.fc_dword_rcvd: 0 SH> dev.ix.0.mac_stats.fc_dword_txd: 0 SH> vmstat -i SH> interrupt total rate SH> irq1: atkbd0 1 0 SH> irq6: fdc0 1 0 SH> irq14: ata0 35 0 SH> irq20: uhci0 1 0 SH> irq23: ehci0 41 0 SH> irq66: arcmsr0 66431 74 SH> cpu0: timer 1773313 1999 SH> irq256: ix0:que 0 99591 112 SH> irq257: ix0:que 1 109526 123 SH> irq258: ix0:que 2 97963 110 SH> irq259: ix0:que 3 220346 248 SH> irq260: ix0:que 4 85912 96 SH> irq261: ix0:que 5 155002 174 SH> irq262: ix0:que 6 99027 111 SH> irq263: ix0:que 7 124176 139 SH> irq264: ix0:link 3 0 SH> irq270: igb1:que 0 312 0 SH> irq271: igb1:que 1 2 0 SH> irq274: igb1:link 2 0 SH> cpu7: timer 1765259 1990 SH> cpu6: timer 1765259 1990 SH> cpu4: timer 1765260 1990 SH> cpu5: timer 1765260 1990 SH> cpu1: timer 1765259 1990 SH> cpu2: timer 1765259 1990 SH> cpu3: timer 1765260 1990 SH> Total 15188501 17123 SH> netstat -m SH> 13479/5091/18570 mbufs in use (current/cache/total) SH> 12327/4319/16646/524288 mbuf clusters in use (current/cache/total/max) SH> 12285/1667 mbuf+clusters out of packet secondary zone in use (current/cache) SH> 6/506/512/262144 4k (page size) jumbo clusters in use (current/cache/total/max) SH> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) SH> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) SH> 28047K/11934K/39982K bytes allocated to network (current/cache/total) SH> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) SH> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) SH> 0/0/0 sfbufs in use (current/peak/max) SH> 0 requests for sfbufs denied SH> 0 requests for sfbufs delayed SH> 1257 requests for I/O initiated by sendfile SH> 0 calls to protocol drain routines SH> netstat -i SH> Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll SH> ix0 1500 <Link#1> 00:1b:21:7e:2e:8c 1307940 0 0 1708291 0 0 SH> ix0 1500 X.X.X.0 ixhost 26195 - - 1601162 - - SH> igb0* 1500 <Link#2> 00:30:48:c5:31:02 0 0 0 0 0 0 SH> igb1 1500 <Link#3> 00:30:48:c5:31:03 741 0 0 721 0 0 SH> igb1 1500 10.10.10.0 10.10.10.64 679 - - 718 - - SH> lo0 16384 <Link#4> 6824 0 0 6824 0 0 SH> lo0 16384 fe80:4::1 fe80:4::1 0 - - 0 - - SH> lo0 16384 localhost ::1 0 - - 0 - - SH> lo0 16384 your-net localhost 26 - - 6824 - - SH> A ping from the cisco 6509 its connected to:- SH> Sending 1000, 100-byte ICMP Echos to 85.236.96.64, timeout is 2 seconds: SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SH> !!!!!!!!!!!!!!!!!!!! SH> Success rate is 99 percent (996/1000), round-trip min/avg/max = 1/1/208 ms SH> Config on the cisco end:- SH> TenGigabitEthernet9/2 is up, line protocol is up (connected) SH> Hardware is C6k 10000Mb 802.3, address is 001e.1323.f325 (bia 001e.1323.f325) SH> Description: ixhost (10Gbps) SH> MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, SH> reliability 255/255, txload 1/255, rxload 1/255 SH> Encapsulation ARPA, loopback not set SH> Keepalive set (10 sec) SH> Full-duplex, 10Gb/s SH> input flow-control is on, output flow-control is on SH> ARP type: ARPA, ARP Timeout 04:00:00 SH> Last input never, output 46w5d, output hang never SH> Last clearing of "show interface" counters never SH> Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 SH> Queueing strategy: fifo SH> Output queue: 0/40 (size/max) SH> 5 minute input rate 12204000 bits/sec, 1351 packets/sec SH> 5 minute output rate 4998000 bits/sec, 1007 packets/sec SH> 78180252111 packets input, 92996518599740 bytes, 0 no buffer SH> Received 314449 broadcasts (0 multicasts) SH> 0 runts, 0 giants, 0 throttles SH> 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored SH> 0 watchdog, 0 multicast, 0 pause input SH> 0 input packets with dribble condition detected SH> 51832915763 packets output, 24954526878125 bytes, 0 underruns SH> 0 output errors, 0 collisions, 3 interface resets SH> 0 babbles, 0 late collision, 0 deferred SH> 0 lost carrier, 0 no carrier, 0 PAUSE output SH> 0 output buffer failures, 0 output buffers swapped out SH> systat 1 :if SH> /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 SH> Load Average | SH> Interface Traffic Peak Total SH> lo0 in 0.953 KB/s 1.916 KB/s 2.669 MB SH> out 0.953 KB/s 1.916 KB/s 2.669 MB SH> igb1 in 0.063 KB/s 0.128 KB/s 85.663 KB SH> out 0.142 KB/s 0.269 KB/s 203.177 KB SH> ix0 in 215.019 KB/s 679.274 KB/s 1000.871 MB SH> out 755.770 KB/s 1.113 MB/s 2.447 GB SH> Regards SH> Steve can you show netstat -Q ? -- С уважением, Коньков mailto:kes-kes@yandex.ru
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?853549959.20120312184940>