From owner-freebsd-net@FreeBSD.ORG Mon Mar 12 15:46:42 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C15351065678 for ; Mon, 12 Mar 2012 15:46:42 +0000 (UTC) (envelope-from seyit.ozgur@istanbul.net) Received: from spamtrap1.istanbul.net (spamtrap1.istanbul.net [85.111.12.35]) by mx1.freebsd.org (Postfix) with ESMTP id EA4268FC19 for ; Mon, 12 Mar 2012 15:46:41 +0000 (UTC) X-ASG-Debug-ID: 1331567191-0426b062bc15b2e0001-QdxwpM Received: from GAMMA.magnetdigital.local (gamma.magnetdigital.local [192.168.131.244]) by spamtrap1.istanbul.net with ESMTP id F53VpxBttOQwBghl; Mon, 12 Mar 2012 17:46:31 +0200 (EET) X-Barracuda-Envelope-From: seyit.ozgur@istanbul.net X-Barracuda-RBL-Trusted-Forwarder: 192.168.131.244 Received: from YUHANNA.magnetdigital.local ([fe80::1058:3088:f9b1:1346]) by GAMMA.magnetdigital.local ([fe80::3cca:d6ef:febb:fafb%17]) with mapi id 14.01.0218.012; Mon, 12 Mar 2012 17:45:43 +0200 From: =?iso-8859-9?Q?Seyit_=D6zg=FCr?= X-Barracuda-Apparent-Source-IP: fe80::1058:3088:f9b1:1346 To: Steven Hartland , "freebsd-net@freebsd.org" Thread-Topic: ixgbe interface micro stalls / slow responses X-ASG-Orig-Subj: RE: ixgbe interface micro stalls / slow responses Thread-Index: AQHNAGGGkJ4cFbi5ykipjjnJogjL25ZmzPrg Date: Mon, 12 Mar 2012 15:45:42 +0000 Message-ID: <3807CE6F3BF4B04EB897F4EBF2D258CE5C055123@yuhanna.magnetdigital.local> References: <18969B34B4EE402986389D5022DCCB35@multiplay.co.uk> In-Reply-To: <18969B34B4EE402986389D5022DCCB35@multiplay.co.uk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [192.168.134.34] Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_00D6_01CD0077.F16AF800" MIME-Version: 1.0 X-Barracuda-Connect: gamma.magnetdigital.local[192.168.131.244] X-Barracuda-Start-Time: 1331567191 X-Barracuda-URL: http://10.10.140.223:8000/cgi-mod/mark.cgi X-ASG-Whitelist: Body =?UTF-8?B?Ll9ldmVudHM=?= X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: RE: ixgbe interface micro stalls / slow responses X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Mar 2012 15:46:42 -0000 ------=_NextPart_000_00D6_01CD0077.F16AF800 Content-Type: text/plain; charset="iso-8859-9" Content-Transfer-Encoding: quoted-printable If you put=20 kern.ipc.nmbclusters=3D524288 kern.ipc.nmbjumbop=3D262144=20 on /boot/loader.conf it won't be "ix0: RX Descriptors exceed system mbuf max, using default instead!" Seyit =D6zg=FCr Network Y=F6neticisi -----Original Message----- From: owner-freebsd-net@freebsd.org = [mailto:owner-freebsd-net@freebsd.org] On Behalf Of Steven Hartland Sent: Monday, March 12, 2012 5:05 PM To: freebsd-net@freebsd.org Subject: ixgbe interface micro stalls / slow responses We've got a machine where with an ix interface on 8.2-RELEASE which is seeing intermittent slow responses. It shows as stalls on the console and is visible as high pings on an mtr from a local machine e.g. Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. X.X.X.X 0.0% 181 0.1 117.7 0.1 2665. 314.8 We've tried updating from 2.3.10 release driver + alias fix to 2.4.5 (the latest from 8.3) but still the behavour is the same. If we do a trace to an igb on the same machine everything is clean. Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.10.10.64 0.0% 136 0.1 0.2 0.1 12.5 1.1 We are seeing "RX Descriptors exceed system mbuf max, using default instead!" on boot with the latest driver but the fix listed in the readme has no effect, as in sysctl.conf we have kern.ipc.nmbclusters=3D524288 kern.ipc.nmbjumbop=3D262144 Nothing looks out of the ordinary by there's definitely a problem there somewhere, any ideas? Detailed info which may be use below. >From dmeg:- ix0: = port 0x2000-0x201f mem=20 0xd8400000-0xd847ffff,0xd8480000-0xd8483fff irq 52 at device 0.0 on pci5 ix0: Using MSIX interrupts with 9 vectors ix0: RX Descriptors exceed system mbuf max, using default instead! ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD] ix0: Ethernet address: 00:1b:21:7e:2e:8c ix0: PCI Express Bus: Speed 5.0Gb/s Width x8 pciconf -v -l ix0@pci0:5:0:0: class=3D0x020000 card=3D0x00068086 chip=3D0x10fb8086 = rev=3D0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' class =3D network subclass =3D ethernet sysctl dev.ix dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5 dev.ix.0.%driver: ix dev.ix.0.%location: slot=3D0 function=3D0 dev.ix.0.%pnpinfo: vendor=3D0x8086 device=3D0x10fb subvendor=3D0x8086 subdevice=3D0x0006 class=3D0x020000 dev.ix.0.%parent: pci5 dev.ix.0.fc: 3 dev.ix.0.advertise_gig: 0 dev.ix.0.enable_aim: 1 dev.ix.0.advertise_speed: 0 dev.ix.0.rx_processing_limit: 128 dev.ix.0.dropped: 0 dev.ix.0.mbuf_defrag_failed: 0 dev.ix.0.no_tx_dma_setup: 0 dev.ix.0.watchdog_events: 0 dev.ix.0.tso_tx: 174470 dev.ix.0.link_irq: 3 dev.ix.0.queue0.interrupt_rate: 1000000 dev.ix.0.queue0.txd_head: 59 dev.ix.0.queue0.txd_tail: 59 dev.ix.0.queue0.no_desc_avail: 0 dev.ix.0.queue0.tx_packets: 38913 dev.ix.0.queue0.rxd_head: 384 dev.ix.0.queue0.rxd_tail: 383 dev.ix.0.queue0.rx_packets: 54982 dev.ix.0.queue0.rx_bytes: 36197485 dev.ix.0.queue0.lro_queued: 0 dev.ix.0.queue0.lro_flushed: 0 dev.ix.0.queue1.interrupt_rate: 1000000 dev.ix.0.queue1.txd_head: 1417 dev.ix.0.queue1.txd_tail: 1417 dev.ix.0.queue1.no_desc_avail: 0 dev.ix.0.queue1.tx_packets: 51196 dev.ix.0.queue1.rxd_head: 445 dev.ix.0.queue1.rxd_tail: 444 dev.ix.0.queue1.rx_packets: 70841 dev.ix.0.queue1.rx_bytes: 26319740 dev.ix.0.queue1.lro_queued: 0 dev.ix.0.queue1.lro_flushed: 0 dev.ix.0.queue2.interrupt_rate: 20408 dev.ix.0.queue2.txd_head: 194 dev.ix.0.queue2.txd_tail: 194 dev.ix.0.queue2.no_desc_avail: 0 dev.ix.0.queue2.tx_packets: 45102 dev.ix.0.queue2.rxd_head: 696 dev.ix.0.queue2.rxd_tail: 695 dev.ix.0.queue2.rx_packets: 65107 dev.ix.0.queue2.rx_bytes: 49222403 dev.ix.0.queue2.lro_queued: 0 dev.ix.0.queue2.lro_flushed: 0 dev.ix.0.queue3.interrupt_rate: 200000 dev.ix.0.queue3.txd_head: 1605 dev.ix.0.queue3.txd_tail: 1605 dev.ix.0.queue3.no_desc_avail: 0 dev.ix.0.queue3.tx_packets: 77375 dev.ix.0.queue3.rxd_head: 79 dev.ix.0.queue3.rxd_tail: 78 dev.ix.0.queue3.rx_packets: 109498 dev.ix.0.queue3.rx_bytes: 109951775 dev.ix.0.queue3.lro_queued: 0 dev.ix.0.queue3.lro_flushed: 0 dev.ix.0.queue4.interrupt_rate: 10526 dev.ix.0.queue4.txd_head: 1624 dev.ix.0.queue4.txd_tail: 1624 dev.ix.0.queue4.no_desc_avail: 0 dev.ix.0.queue4.tx_packets: 39497 dev.ix.0.queue4.rxd_head: 480 dev.ix.0.queue4.rxd_tail: 479 dev.ix.0.queue4.rx_packets: 51998 dev.ix.0.queue4.rx_bytes: 21965859 dev.ix.0.queue4.lro_queued: 0 dev.ix.0.queue4.lro_flushed: 0 dev.ix.0.queue5.interrupt_rate: 1000000 dev.ix.0.queue5.txd_head: 1613 dev.ix.0.queue5.txd_tail: 1613 dev.ix.0.queue5.no_desc_avail: 0 dev.ix.0.queue5.tx_packets: 69860 dev.ix.0.queue5.rxd_head: 846 dev.ix.0.queue5.rxd_tail: 845 dev.ix.0.queue5.rx_packets: 81331 dev.ix.0.queue5.rx_bytes: 32429926 dev.ix.0.queue5.lro_queued: 0 dev.ix.0.queue5.lro_flushed: 0 dev.ix.0.queue6.interrupt_rate: 142857 dev.ix.0.queue6.txd_head: 1482 dev.ix.0.queue6.txd_tail: 1484 dev.ix.0.queue6.no_desc_avail: 0 dev.ix.0.queue6.tx_packets: 45878 dev.ix.0.queue6.rxd_head: 355 dev.ix.0.queue6.rxd_tail: 354 dev.ix.0.queue6.rx_packets: 62211 dev.ix.0.queue6.rx_bytes: 27653559 dev.ix.0.queue6.lro_queued: 0 dev.ix.0.queue6.lro_flushed: 0 dev.ix.0.queue7.interrupt_rate: 5347 dev.ix.0.queue7.txd_head: 603 dev.ix.0.queue7.txd_tail: 603 dev.ix.0.queue7.no_desc_avail: 0 dev.ix.0.queue7.tx_packets: 61997 dev.ix.0.queue7.rxd_head: 826 dev.ix.0.queue7.rxd_tail: 825 dev.ix.0.queue7.rx_packets: 83460 dev.ix.0.queue7.rx_bytes: 50183116 dev.ix.0.queue7.lro_queued: 0 dev.ix.0.queue7.lro_flushed: 0 dev.ix.0.mac_stats.crc_errs: 0 dev.ix.0.mac_stats.ill_errs: 0 dev.ix.0.mac_stats.byte_errs: 0 dev.ix.0.mac_stats.short_discards: 0 dev.ix.0.mac_stats.local_faults: 3 dev.ix.0.mac_stats.remote_faults: 1 dev.ix.0.mac_stats.rec_len_errs: 0 dev.ix.0.mac_stats.link_xon_txd: 0 dev.ix.0.mac_stats.link_xon_rcvd: 0 dev.ix.0.mac_stats.link_xoff_txd: 0 dev.ix.0.mac_stats.link_xoff_rcvd: 0 dev.ix.0.mac_stats.total_octets_rcvd: 360072702 dev.ix.0.mac_stats.good_octets_rcvd: 359999778 dev.ix.0.mac_stats.total_pkts_rcvd: 637428 dev.ix.0.mac_stats.good_pkts_rcvd: 636321 dev.ix.0.mac_stats.mcast_pkts_rcvd: 35 dev.ix.0.mac_stats.bcast_pkts_rcvd: 1411 dev.ix.0.mac_stats.rx_frames_64: 222251 dev.ix.0.mac_stats.rx_frames_65_127: 159044 dev.ix.0.mac_stats.rx_frames_128_255: 15139 dev.ix.0.mac_stats.rx_frames_256_511: 13885 dev.ix.0.mac_stats.rx_frames_512_1023: 21283 dev.ix.0.mac_stats.rx_frames_1024_1522: 204719 dev.ix.0.mac_stats.recv_undersized: 0 dev.ix.0.mac_stats.recv_fragmented: 0 dev.ix.0.mac_stats.recv_oversized: 0 dev.ix.0.mac_stats.recv_jabberd: 0 dev.ix.0.mac_stats.management_pkts_rcvd: 0 dev.ix.0.mac_stats.management_pkts_drpd: 0 dev.ix.0.mac_stats.checksum_errs: 0 dev.ix.0.mac_stats.good_octets_txd: 882467530 dev.ix.0.mac_stats.total_pkts_txd: 816387 dev.ix.0.mac_stats.good_pkts_txd: 816387 dev.ix.0.mac_stats.bcast_pkts_txd: 36 dev.ix.0.mac_stats.mcast_pkts_txd: 0 dev.ix.0.mac_stats.management_pkts_txd: 0 dev.ix.0.mac_stats.tx_frames_64: 21509 dev.ix.0.mac_stats.tx_frames_65_127: 168051 dev.ix.0.mac_stats.tx_frames_128_255: 19184 dev.ix.0.mac_stats.tx_frames_256_511: 22775 dev.ix.0.mac_stats.tx_frames_512_1023: 24222 dev.ix.0.mac_stats.tx_frames_1024_1522: 560646 dev.ix.0.mac_stats.fc_crc: 0 dev.ix.0.mac_stats.fc_last: 0 dev.ix.0.mac_stats.fc_drpd: 0 dev.ix.0.mac_stats.fc_pkts_rcvd: 0 dev.ix.0.mac_stats.fc_pkts_txd: 0 dev.ix.0.mac_stats.fc_dword_rcvd: 0 dev.ix.0.mac_stats.fc_dword_txd: 0 vmstat -i interrupt total rate irq1: atkbd0 1 0 irq6: fdc0 1 0 irq14: ata0 35 0 irq20: uhci0 1 0 irq23: ehci0 41 0 irq66: arcmsr0 66431 74 cpu0: timer 1773313 1999 irq256: ix0:que 0 99591 112 irq257: ix0:que 1 109526 123 irq258: ix0:que 2 97963 110 irq259: ix0:que 3 220346 248 irq260: ix0:que 4 85912 96 irq261: ix0:que 5 155002 174 irq262: ix0:que 6 99027 111 irq263: ix0:que 7 124176 139 irq264: ix0:link 3 0 irq270: igb1:que 0 312 0 irq271: igb1:que 1 2 0 irq274: igb1:link 2 0 cpu7: timer 1765259 1990 cpu6: timer 1765259 1990 cpu4: timer 1765260 1990 cpu5: timer 1765260 1990 cpu1: timer 1765259 1990 cpu2: timer 1765259 1990 cpu3: timer 1765260 1990 Total 15188501 17123 netstat -m 13479/5091/18570 mbufs in use (current/cache/total) 12327/4319/16646/524288 mbuf clusters in use (current/cache/total/max) 12285/1667 mbuf+clusters out of packet secondary zone in use = (current/cache) 6/506/512/262144 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 28047K/11934K/39982K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 1257 requests for I/O initiated by sendfile 0 calls to protocol drain routines netstat -i Name Mtu Network Address Ipkts Ierrs Idrop = Opkts Oerrs Coll ix0 1500 00:1b:21:7e:2e:8c 1307940 0 0 = 1708291 0 0 ix0 1500 X.X.X.0 ixhost 26195 - - = 1601162 - - igb0* 1500 00:30:48:c5:31:02 0 0 0 = 0 0 0 igb1 1500 00:30:48:c5:31:03 741 0 0 = 721 0 0 igb1 1500 10.10.10.0 10.10.10.64 679 - - = 718 - - lo0 16384 6824 0 0 = 6824 0 0 lo0 16384 fe80:4::1 fe80:4::1 0 - - = 0 - - lo0 16384 localhost ::1 0 - - = 0 - - lo0 16384 your-net localhost 26 - - = 6824 - - A ping from the cisco 6509 its connected to:- Sending 1000, 100-byte ICMP Echos to 85.236.96.64, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!! Success rate is 99 percent (996/1000), round-trip min/avg/max =3D = 1/1/208 ms Config on the cisco end:- TenGigabitEthernet9/2 is up, line protocol is up (connected) Hardware is C6k 10000Mb 802.3, address is 001e.1323.f325 (bia 001e.1323.f325) Description: ixhost (10Gbps) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 10Gb/s input flow-control is on, output flow-control is on ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 46w5d, output hang never Last clearing of "show interface" counters never Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: = 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 12204000 bits/sec, 1351 packets/sec 5 minute output rate 4998000 bits/sec, 1007 packets/sec 78180252111 packets input, 92996518599740 bytes, 0 no buffer Received 314449 broadcasts (0 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected 51832915763 packets output, 24954526878125 bytes, 0 underruns 0 output errors, 0 collisions, 3 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out systat 1 :if /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 = /10 Load Average | Interface Traffic Peak = Total lo0 in 0.953 KB/s 1.916 KB/s 2.669 = MB out 0.953 KB/s 1.916 KB/s 2.669 = MB igb1 in 0.063 KB/s 0.128 KB/s 85.663 = KB out 0.142 KB/s 0.269 KB/s 203.177 = KB ix0 in 215.019 KB/s 679.274 KB/s 1000.871 = MB out 755.770 KB/s 1.113 MB/s 2.447 = GB Regards Steve=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e.mail is private and confidential between Multiplay (UK) Ltd. and = the person or entity to whom it is addressed. In the event of misdirection, = the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.=20 In the event of misdirection, illegible or incomplete transmission = please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" ------=_NextPart_000_00D6_01CD0077.F16AF800--