Date: Mon, 12 Mar 2012 15:05:12 -0000 From: "Steven Hartland" <killing@multiplay.co.uk> To: <freebsd-net@freebsd.org> Subject: ixgbe interface micro stalls / slow responses Message-ID: <18969B34B4EE402986389D5022DCCB35@multiplay.co.uk>
next in thread | raw e-mail | index | archive | help
We've got a machine where with an ix interface on 8.2-RELEASE
which is seeing intermittent slow responses. It shows as stalls
on the console and is visible as high pings on an mtr
from a local machine e.g.
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. X.X.X.X 0.0% 181 0.1 117.7 0.1 2665. 314.8
We've tried updating from 2.3.10 release driver + alias fix
to 2.4.5 (the latest from 8.3) but still the behavour is the
same.
If we do a trace to an igb on the same machine everything is
clean.
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.10.10.64 0.0% 136 0.1 0.2 0.1 12.5 1.1
We are seeing "RX Descriptors exceed system mbuf max, using
default instead!" on boot with the latest driver but the fix
listed in the readme has no effect, as in sysctl.conf we have
kern.ipc.nmbclusters=524288
kern.ipc.nmbjumbop=262144
Nothing looks out of the ordinary by there's definitely a
problem there somewhere, any ideas?
Detailed info which may be use below.
>From dmeg:-
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5> port 0x2000-0x201f mem
0xd8400000-0xd847ffff,0xd8480000-0xd8483fff irq 52 at device 0.0 on pci5
ix0: Using MSIX interrupts with 9 vectors
ix0: RX Descriptors exceed system mbuf max, using default instead!
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: Ethernet address: 00:1b:21:7e:2e:8c
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
pciconf -v -l
ix0@pci0:5:0:0: class=0x020000 card=0x00068086 chip=0x10fb8086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
class = network
subclass = ethernet
sysctl dev.ix
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=0 function=0
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10fb subvendor=0x8086 subdevice=0x0006 class=0x020000
dev.ix.0.%parent: pci5
dev.ix.0.fc: 3
dev.ix.0.advertise_gig: 0
dev.ix.0.enable_aim: 1
dev.ix.0.advertise_speed: 0
dev.ix.0.rx_processing_limit: 128
dev.ix.0.dropped: 0
dev.ix.0.mbuf_defrag_failed: 0
dev.ix.0.no_tx_dma_setup: 0
dev.ix.0.watchdog_events: 0
dev.ix.0.tso_tx: 174470
dev.ix.0.link_irq: 3
dev.ix.0.queue0.interrupt_rate: 1000000
dev.ix.0.queue0.txd_head: 59
dev.ix.0.queue0.txd_tail: 59
dev.ix.0.queue0.no_desc_avail: 0
dev.ix.0.queue0.tx_packets: 38913
dev.ix.0.queue0.rxd_head: 384
dev.ix.0.queue0.rxd_tail: 383
dev.ix.0.queue0.rx_packets: 54982
dev.ix.0.queue0.rx_bytes: 36197485
dev.ix.0.queue0.lro_queued: 0
dev.ix.0.queue0.lro_flushed: 0
dev.ix.0.queue1.interrupt_rate: 1000000
dev.ix.0.queue1.txd_head: 1417
dev.ix.0.queue1.txd_tail: 1417
dev.ix.0.queue1.no_desc_avail: 0
dev.ix.0.queue1.tx_packets: 51196
dev.ix.0.queue1.rxd_head: 445
dev.ix.0.queue1.rxd_tail: 444
dev.ix.0.queue1.rx_packets: 70841
dev.ix.0.queue1.rx_bytes: 26319740
dev.ix.0.queue1.lro_queued: 0
dev.ix.0.queue1.lro_flushed: 0
dev.ix.0.queue2.interrupt_rate: 20408
dev.ix.0.queue2.txd_head: 194
dev.ix.0.queue2.txd_tail: 194
dev.ix.0.queue2.no_desc_avail: 0
dev.ix.0.queue2.tx_packets: 45102
dev.ix.0.queue2.rxd_head: 696
dev.ix.0.queue2.rxd_tail: 695
dev.ix.0.queue2.rx_packets: 65107
dev.ix.0.queue2.rx_bytes: 49222403
dev.ix.0.queue2.lro_queued: 0
dev.ix.0.queue2.lro_flushed: 0
dev.ix.0.queue3.interrupt_rate: 200000
dev.ix.0.queue3.txd_head: 1605
dev.ix.0.queue3.txd_tail: 1605
dev.ix.0.queue3.no_desc_avail: 0
dev.ix.0.queue3.tx_packets: 77375
dev.ix.0.queue3.rxd_head: 79
dev.ix.0.queue3.rxd_tail: 78
dev.ix.0.queue3.rx_packets: 109498
dev.ix.0.queue3.rx_bytes: 109951775
dev.ix.0.queue3.lro_queued: 0
dev.ix.0.queue3.lro_flushed: 0
dev.ix.0.queue4.interrupt_rate: 10526
dev.ix.0.queue4.txd_head: 1624
dev.ix.0.queue4.txd_tail: 1624
dev.ix.0.queue4.no_desc_avail: 0
dev.ix.0.queue4.tx_packets: 39497
dev.ix.0.queue4.rxd_head: 480
dev.ix.0.queue4.rxd_tail: 479
dev.ix.0.queue4.rx_packets: 51998
dev.ix.0.queue4.rx_bytes: 21965859
dev.ix.0.queue4.lro_queued: 0
dev.ix.0.queue4.lro_flushed: 0
dev.ix.0.queue5.interrupt_rate: 1000000
dev.ix.0.queue5.txd_head: 1613
dev.ix.0.queue5.txd_tail: 1613
dev.ix.0.queue5.no_desc_avail: 0
dev.ix.0.queue5.tx_packets: 69860
dev.ix.0.queue5.rxd_head: 846
dev.ix.0.queue5.rxd_tail: 845
dev.ix.0.queue5.rx_packets: 81331
dev.ix.0.queue5.rx_bytes: 32429926
dev.ix.0.queue5.lro_queued: 0
dev.ix.0.queue5.lro_flushed: 0
dev.ix.0.queue6.interrupt_rate: 142857
dev.ix.0.queue6.txd_head: 1482
dev.ix.0.queue6.txd_tail: 1484
dev.ix.0.queue6.no_desc_avail: 0
dev.ix.0.queue6.tx_packets: 45878
dev.ix.0.queue6.rxd_head: 355
dev.ix.0.queue6.rxd_tail: 354
dev.ix.0.queue6.rx_packets: 62211
dev.ix.0.queue6.rx_bytes: 27653559
dev.ix.0.queue6.lro_queued: 0
dev.ix.0.queue6.lro_flushed: 0
dev.ix.0.queue7.interrupt_rate: 5347
dev.ix.0.queue7.txd_head: 603
dev.ix.0.queue7.txd_tail: 603
dev.ix.0.queue7.no_desc_avail: 0
dev.ix.0.queue7.tx_packets: 61997
dev.ix.0.queue7.rxd_head: 826
dev.ix.0.queue7.rxd_tail: 825
dev.ix.0.queue7.rx_packets: 83460
dev.ix.0.queue7.rx_bytes: 50183116
dev.ix.0.queue7.lro_queued: 0
dev.ix.0.queue7.lro_flushed: 0
dev.ix.0.mac_stats.crc_errs: 0
dev.ix.0.mac_stats.ill_errs: 0
dev.ix.0.mac_stats.byte_errs: 0
dev.ix.0.mac_stats.short_discards: 0
dev.ix.0.mac_stats.local_faults: 3
dev.ix.0.mac_stats.remote_faults: 1
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.link_xon_txd: 0
dev.ix.0.mac_stats.link_xon_rcvd: 0
dev.ix.0.mac_stats.link_xoff_txd: 0
dev.ix.0.mac_stats.link_xoff_rcvd: 0
dev.ix.0.mac_stats.total_octets_rcvd: 360072702
dev.ix.0.mac_stats.good_octets_rcvd: 359999778
dev.ix.0.mac_stats.total_pkts_rcvd: 637428
dev.ix.0.mac_stats.good_pkts_rcvd: 636321
dev.ix.0.mac_stats.mcast_pkts_rcvd: 35
dev.ix.0.mac_stats.bcast_pkts_rcvd: 1411
dev.ix.0.mac_stats.rx_frames_64: 222251
dev.ix.0.mac_stats.rx_frames_65_127: 159044
dev.ix.0.mac_stats.rx_frames_128_255: 15139
dev.ix.0.mac_stats.rx_frames_256_511: 13885
dev.ix.0.mac_stats.rx_frames_512_1023: 21283
dev.ix.0.mac_stats.rx_frames_1024_1522: 204719
dev.ix.0.mac_stats.recv_undersized: 0
dev.ix.0.mac_stats.recv_fragmented: 0
dev.ix.0.mac_stats.recv_oversized: 0
dev.ix.0.mac_stats.recv_jabberd: 0
dev.ix.0.mac_stats.management_pkts_rcvd: 0
dev.ix.0.mac_stats.management_pkts_drpd: 0
dev.ix.0.mac_stats.checksum_errs: 0
dev.ix.0.mac_stats.good_octets_txd: 882467530
dev.ix.0.mac_stats.total_pkts_txd: 816387
dev.ix.0.mac_stats.good_pkts_txd: 816387
dev.ix.0.mac_stats.bcast_pkts_txd: 36
dev.ix.0.mac_stats.mcast_pkts_txd: 0
dev.ix.0.mac_stats.management_pkts_txd: 0
dev.ix.0.mac_stats.tx_frames_64: 21509
dev.ix.0.mac_stats.tx_frames_65_127: 168051
dev.ix.0.mac_stats.tx_frames_128_255: 19184
dev.ix.0.mac_stats.tx_frames_256_511: 22775
dev.ix.0.mac_stats.tx_frames_512_1023: 24222
dev.ix.0.mac_stats.tx_frames_1024_1522: 560646
dev.ix.0.mac_stats.fc_crc: 0
dev.ix.0.mac_stats.fc_last: 0
dev.ix.0.mac_stats.fc_drpd: 0
dev.ix.0.mac_stats.fc_pkts_rcvd: 0
dev.ix.0.mac_stats.fc_pkts_txd: 0
dev.ix.0.mac_stats.fc_dword_rcvd: 0
dev.ix.0.mac_stats.fc_dword_txd: 0
vmstat -i
interrupt total rate
irq1: atkbd0 1 0
irq6: fdc0 1 0
irq14: ata0 35 0
irq20: uhci0 1 0
irq23: ehci0 41 0
irq66: arcmsr0 66431 74
cpu0: timer 1773313 1999
irq256: ix0:que 0 99591 112
irq257: ix0:que 1 109526 123
irq258: ix0:que 2 97963 110
irq259: ix0:que 3 220346 248
irq260: ix0:que 4 85912 96
irq261: ix0:que 5 155002 174
irq262: ix0:que 6 99027 111
irq263: ix0:que 7 124176 139
irq264: ix0:link 3 0
irq270: igb1:que 0 312 0
irq271: igb1:que 1 2 0
irq274: igb1:link 2 0
cpu7: timer 1765259 1990
cpu6: timer 1765259 1990
cpu4: timer 1765260 1990
cpu5: timer 1765260 1990
cpu1: timer 1765259 1990
cpu2: timer 1765259 1990
cpu3: timer 1765260 1990
Total 15188501 17123
netstat -m
13479/5091/18570 mbufs in use (current/cache/total)
12327/4319/16646/524288 mbuf clusters in use (current/cache/total/max)
12285/1667 mbuf+clusters out of packet secondary zone in use (current/cache)
6/506/512/262144 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
28047K/11934K/39982K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
1257 requests for I/O initiated by sendfile
0 calls to protocol drain routines
netstat -i
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
ix0 1500 <Link#1> 00:1b:21:7e:2e:8c 1307940 0 0 1708291 0 0
ix0 1500 X.X.X.0 ixhost 26195 - - 1601162 - -
igb0* 1500 <Link#2> 00:30:48:c5:31:02 0 0 0 0 0 0
igb1 1500 <Link#3> 00:30:48:c5:31:03 741 0 0 721 0 0
igb1 1500 10.10.10.0 10.10.10.64 679 - - 718 - -
lo0 16384 <Link#4> 6824 0 0 6824 0 0
lo0 16384 fe80:4::1 fe80:4::1 0 - - 0 - -
lo0 16384 localhost ::1 0 - - 0 - -
lo0 16384 your-net localhost 26 - - 6824 - -
A ping from the cisco 6509 its connected to:-
Sending 1000, 100-byte ICMP Echos to 85.236.96.64, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (996/1000), round-trip min/avg/max = 1/1/208 ms
Config on the cisco end:-
TenGigabitEthernet9/2 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 001e.1323.f325 (bia 001e.1323.f325)
Description: ixhost (10Gbps)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 46w5d, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 12204000 bits/sec, 1351 packets/sec
5 minute output rate 4998000 bits/sec, 1007 packets/sec
78180252111 packets input, 92996518599740 bytes, 0 no buffer
Received 314449 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
51832915763 packets output, 24954526878125 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
systat 1 :if
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |
Interface Traffic Peak Total
lo0 in 0.953 KB/s 1.916 KB/s 2.669 MB
out 0.953 KB/s 1.916 KB/s 2.669 MB
igb1 in 0.063 KB/s 0.128 KB/s 85.663 KB
out 0.142 KB/s 0.269 KB/s 203.177 KB
ix0 in 215.019 KB/s 679.274 KB/s 1000.871 MB
out 755.770 KB/s 1.113 MB/s 2.447 GB
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?18969B34B4EE402986389D5022DCCB35>