Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 12 Mar 2012 15:45:42 +0000
From:      =?iso-8859-9?Q?Seyit_=D6zg=FCr?= <seyit.ozgur@istanbul.net>
To:        Steven Hartland <killing@multiplay.co.uk>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>
Subject:   RE: ixgbe interface micro stalls / slow responses
Message-ID:  <3807CE6F3BF4B04EB897F4EBF2D258CE5C055123@yuhanna.magnetdigital.local>
In-Reply-To: <18969B34B4EE402986389D5022DCCB35@multiplay.co.uk>
References:  <18969B34B4EE402986389D5022DCCB35@multiplay.co.uk>

next in thread | previous in thread | raw e-mail | index | archive | help
------=_NextPart_000_00D6_01CD0077.F16AF800
Content-Type: text/plain;
	charset="iso-8859-9"
Content-Transfer-Encoding: quoted-printable

If you put=20

kern.ipc.nmbclusters=3D524288
kern.ipc.nmbjumbop=3D262144=20
on  /boot/loader.conf

it won't be "ix0: RX Descriptors exceed system mbuf max, using default
instead!"


Seyit =D6zg=FCr
Network Y=F6neticisi


-----Original Message-----
From: owner-freebsd-net@freebsd.org =
[mailto:owner-freebsd-net@freebsd.org]
On Behalf Of Steven Hartland
Sent: Monday, March 12, 2012 5:05 PM
To: freebsd-net@freebsd.org
Subject: ixgbe interface micro stalls / slow responses

We've got a machine where with an ix interface on 8.2-RELEASE
which is seeing intermittent slow responses. It shows as stalls
on the console and is visible as high pings on an mtr
from a local machine e.g.

                      Packets               Pings
 Host              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. X.X.X.X         0.0%   181    0.1 117.7   0.1 2665. 314.8

We've tried updating from 2.3.10 release driver + alias fix
to 2.4.5 (the latest from 8.3) but still the behavour is the
same.

If we do a trace to an igb on the same machine everything is
clean.
                      Packets               Pings
 Host              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.10.10.64     0.0%   136    0.1   0.2   0.1  12.5   1.1

We are seeing "RX Descriptors exceed system mbuf max, using
default instead!" on boot with the latest driver but the fix
listed in the readme has no effect, as in sysctl.conf we have
kern.ipc.nmbclusters=3D524288
kern.ipc.nmbjumbop=3D262144

Nothing looks out of the ordinary by there's definitely a
problem there somewhere, any ideas?

Detailed info which may be use below.

>From dmeg:-
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5> =
port
0x2000-0x201f mem=20
0xd8400000-0xd847ffff,0xd8480000-0xd8483fff irq 52 at device 0.0 on pci5
ix0: Using MSIX interrupts with 9 vectors
ix0: RX Descriptors exceed system mbuf max, using default instead!
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: Ethernet address: 00:1b:21:7e:2e:8c
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8

pciconf -v -l
ix0@pci0:5:0:0: class=3D0x020000 card=3D0x00068086 chip=3D0x10fb8086 =
rev=3D0x01
hdr=3D0x00
    vendor     =3D 'Intel Corporation'
    class      =3D network
    subclass   =3D ethernet


sysctl dev.ix
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version -
2.4.5
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=3D0 function=3D0
dev.ix.0.%pnpinfo: vendor=3D0x8086 device=3D0x10fb subvendor=3D0x8086
subdevice=3D0x0006 class=3D0x020000
dev.ix.0.%parent: pci5
dev.ix.0.fc: 3
dev.ix.0.advertise_gig: 0
dev.ix.0.enable_aim: 1
dev.ix.0.advertise_speed: 0
dev.ix.0.rx_processing_limit: 128
dev.ix.0.dropped: 0
dev.ix.0.mbuf_defrag_failed: 0
dev.ix.0.no_tx_dma_setup: 0
dev.ix.0.watchdog_events: 0
dev.ix.0.tso_tx: 174470
dev.ix.0.link_irq: 3
dev.ix.0.queue0.interrupt_rate: 1000000
dev.ix.0.queue0.txd_head: 59
dev.ix.0.queue0.txd_tail: 59
dev.ix.0.queue0.no_desc_avail: 0
dev.ix.0.queue0.tx_packets: 38913
dev.ix.0.queue0.rxd_head: 384
dev.ix.0.queue0.rxd_tail: 383
dev.ix.0.queue0.rx_packets: 54982
dev.ix.0.queue0.rx_bytes: 36197485
dev.ix.0.queue0.lro_queued: 0
dev.ix.0.queue0.lro_flushed: 0
dev.ix.0.queue1.interrupt_rate: 1000000
dev.ix.0.queue1.txd_head: 1417
dev.ix.0.queue1.txd_tail: 1417
dev.ix.0.queue1.no_desc_avail: 0
dev.ix.0.queue1.tx_packets: 51196
dev.ix.0.queue1.rxd_head: 445
dev.ix.0.queue1.rxd_tail: 444
dev.ix.0.queue1.rx_packets: 70841
dev.ix.0.queue1.rx_bytes: 26319740
dev.ix.0.queue1.lro_queued: 0
dev.ix.0.queue1.lro_flushed: 0
dev.ix.0.queue2.interrupt_rate: 20408
dev.ix.0.queue2.txd_head: 194
dev.ix.0.queue2.txd_tail: 194
dev.ix.0.queue2.no_desc_avail: 0
dev.ix.0.queue2.tx_packets: 45102
dev.ix.0.queue2.rxd_head: 696
dev.ix.0.queue2.rxd_tail: 695
dev.ix.0.queue2.rx_packets: 65107
dev.ix.0.queue2.rx_bytes: 49222403
dev.ix.0.queue2.lro_queued: 0
dev.ix.0.queue2.lro_flushed: 0
dev.ix.0.queue3.interrupt_rate: 200000
dev.ix.0.queue3.txd_head: 1605
dev.ix.0.queue3.txd_tail: 1605
dev.ix.0.queue3.no_desc_avail: 0
dev.ix.0.queue3.tx_packets: 77375
dev.ix.0.queue3.rxd_head: 79
dev.ix.0.queue3.rxd_tail: 78
dev.ix.0.queue3.rx_packets: 109498
dev.ix.0.queue3.rx_bytes: 109951775
dev.ix.0.queue3.lro_queued: 0
dev.ix.0.queue3.lro_flushed: 0
dev.ix.0.queue4.interrupt_rate: 10526
dev.ix.0.queue4.txd_head: 1624
dev.ix.0.queue4.txd_tail: 1624
dev.ix.0.queue4.no_desc_avail: 0
dev.ix.0.queue4.tx_packets: 39497
dev.ix.0.queue4.rxd_head: 480
dev.ix.0.queue4.rxd_tail: 479
dev.ix.0.queue4.rx_packets: 51998
dev.ix.0.queue4.rx_bytes: 21965859
dev.ix.0.queue4.lro_queued: 0
dev.ix.0.queue4.lro_flushed: 0
dev.ix.0.queue5.interrupt_rate: 1000000
dev.ix.0.queue5.txd_head: 1613
dev.ix.0.queue5.txd_tail: 1613
dev.ix.0.queue5.no_desc_avail: 0
dev.ix.0.queue5.tx_packets: 69860
dev.ix.0.queue5.rxd_head: 846
dev.ix.0.queue5.rxd_tail: 845
dev.ix.0.queue5.rx_packets: 81331
dev.ix.0.queue5.rx_bytes: 32429926
dev.ix.0.queue5.lro_queued: 0
dev.ix.0.queue5.lro_flushed: 0
dev.ix.0.queue6.interrupt_rate: 142857
dev.ix.0.queue6.txd_head: 1482
dev.ix.0.queue6.txd_tail: 1484
dev.ix.0.queue6.no_desc_avail: 0
dev.ix.0.queue6.tx_packets: 45878
dev.ix.0.queue6.rxd_head: 355
dev.ix.0.queue6.rxd_tail: 354
dev.ix.0.queue6.rx_packets: 62211
dev.ix.0.queue6.rx_bytes: 27653559
dev.ix.0.queue6.lro_queued: 0
dev.ix.0.queue6.lro_flushed: 0
dev.ix.0.queue7.interrupt_rate: 5347
dev.ix.0.queue7.txd_head: 603
dev.ix.0.queue7.txd_tail: 603
dev.ix.0.queue7.no_desc_avail: 0
dev.ix.0.queue7.tx_packets: 61997
dev.ix.0.queue7.rxd_head: 826
dev.ix.0.queue7.rxd_tail: 825
dev.ix.0.queue7.rx_packets: 83460
dev.ix.0.queue7.rx_bytes: 50183116
dev.ix.0.queue7.lro_queued: 0
dev.ix.0.queue7.lro_flushed: 0
dev.ix.0.mac_stats.crc_errs: 0
dev.ix.0.mac_stats.ill_errs: 0
dev.ix.0.mac_stats.byte_errs: 0
dev.ix.0.mac_stats.short_discards: 0
dev.ix.0.mac_stats.local_faults: 3
dev.ix.0.mac_stats.remote_faults: 1
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.link_xon_txd: 0
dev.ix.0.mac_stats.link_xon_rcvd: 0
dev.ix.0.mac_stats.link_xoff_txd: 0
dev.ix.0.mac_stats.link_xoff_rcvd: 0
dev.ix.0.mac_stats.total_octets_rcvd: 360072702
dev.ix.0.mac_stats.good_octets_rcvd: 359999778
dev.ix.0.mac_stats.total_pkts_rcvd: 637428
dev.ix.0.mac_stats.good_pkts_rcvd: 636321
dev.ix.0.mac_stats.mcast_pkts_rcvd: 35
dev.ix.0.mac_stats.bcast_pkts_rcvd: 1411
dev.ix.0.mac_stats.rx_frames_64: 222251
dev.ix.0.mac_stats.rx_frames_65_127: 159044
dev.ix.0.mac_stats.rx_frames_128_255: 15139
dev.ix.0.mac_stats.rx_frames_256_511: 13885
dev.ix.0.mac_stats.rx_frames_512_1023: 21283
dev.ix.0.mac_stats.rx_frames_1024_1522: 204719
dev.ix.0.mac_stats.recv_undersized: 0
dev.ix.0.mac_stats.recv_fragmented: 0
dev.ix.0.mac_stats.recv_oversized: 0
dev.ix.0.mac_stats.recv_jabberd: 0
dev.ix.0.mac_stats.management_pkts_rcvd: 0
dev.ix.0.mac_stats.management_pkts_drpd: 0
dev.ix.0.mac_stats.checksum_errs: 0
dev.ix.0.mac_stats.good_octets_txd: 882467530
dev.ix.0.mac_stats.total_pkts_txd: 816387
dev.ix.0.mac_stats.good_pkts_txd: 816387
dev.ix.0.mac_stats.bcast_pkts_txd: 36
dev.ix.0.mac_stats.mcast_pkts_txd: 0
dev.ix.0.mac_stats.management_pkts_txd: 0
dev.ix.0.mac_stats.tx_frames_64: 21509
dev.ix.0.mac_stats.tx_frames_65_127: 168051
dev.ix.0.mac_stats.tx_frames_128_255: 19184
dev.ix.0.mac_stats.tx_frames_256_511: 22775
dev.ix.0.mac_stats.tx_frames_512_1023: 24222
dev.ix.0.mac_stats.tx_frames_1024_1522: 560646
dev.ix.0.mac_stats.fc_crc: 0
dev.ix.0.mac_stats.fc_last: 0
dev.ix.0.mac_stats.fc_drpd: 0
dev.ix.0.mac_stats.fc_pkts_rcvd: 0
dev.ix.0.mac_stats.fc_pkts_txd: 0
dev.ix.0.mac_stats.fc_dword_rcvd: 0
dev.ix.0.mac_stats.fc_dword_txd: 0

vmstat -i
interrupt                          total       rate
irq1: atkbd0                           1          0
irq6: fdc0                             1          0
irq14: ata0                           35          0
irq20: uhci0                           1          0
irq23: ehci0                          41          0
irq66: arcmsr0                     66431         74
cpu0: timer                      1773313       1999
irq256: ix0:que 0                  99591        112
irq257: ix0:que 1                 109526        123
irq258: ix0:que 2                  97963        110
irq259: ix0:que 3                 220346        248
irq260: ix0:que 4                  85912         96
irq261: ix0:que 5                 155002        174
irq262: ix0:que 6                  99027        111
irq263: ix0:que 7                 124176        139
irq264: ix0:link                       3          0
irq270: igb1:que 0                   312          0
irq271: igb1:que 1                     2          0
irq274: igb1:link                      2          0
cpu7: timer                      1765259       1990
cpu6: timer                      1765259       1990
cpu4: timer                      1765260       1990
cpu5: timer                      1765260       1990
cpu1: timer                      1765259       1990
cpu2: timer                      1765259       1990
cpu3: timer                      1765260       1990
Total                           15188501      17123

netstat -m
13479/5091/18570 mbufs in use (current/cache/total)
12327/4319/16646/524288 mbuf clusters in use (current/cache/total/max)
12285/1667 mbuf+clusters out of packet secondary zone in use =
(current/cache)
6/506/512/262144 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
28047K/11934K/39982K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
1257 requests for I/O initiated by sendfile
0 calls to protocol drain routines

netstat -i
Name    Mtu Network       Address              Ipkts Ierrs Idrop    =
Opkts
Oerrs  Coll
ix0    1500 <Link#1>      00:1b:21:7e:2e:8c  1307940     0     0  =
1708291
0     0
ix0    1500 X.X.X.0       ixhost               26195     -     -  =
1601162
-     -
igb0*  1500 <Link#2>      00:30:48:c5:31:02        0     0     0        =
0
0     0
igb1   1500 <Link#3>      00:30:48:c5:31:03      741     0     0      =
721
0     0
igb1   1500 10.10.10.0    10.10.10.64            679     -     -      =
718
-     -
lo0   16384 <Link#4>                            6824     0     0     =
6824
0     0
lo0   16384 fe80:4::1     fe80:4::1                0     -     -        =
0
-     -
lo0   16384 localhost     ::1                      0     -     -        =
0
-     -
lo0   16384 your-net      localhost               26     -     -     =
6824
-     -

A ping from the cisco 6509 its connected to:-
Sending 1000, 100-byte ICMP Echos to 85.236.96.64, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (996/1000), round-trip min/avg/max =3D =
1/1/208 ms

Config on the cisco end:-
TenGigabitEthernet9/2 is up, line protocol is up (connected)
  Hardware is C6k 10000Mb 802.3, address is 001e.1323.f325 (bia
001e.1323.f325)
  Description: ixhost (10Gbps)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s
  input flow-control is on, output flow-control is on
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 46w5d, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: =
0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 12204000 bits/sec, 1351 packets/sec
  5 minute output rate 4998000 bits/sec, 1007 packets/sec
     78180252111 packets input, 92996518599740 bytes, 0 no buffer
     Received 314449 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     51832915763 packets output, 24954526878125 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out


systat 1 :if
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   =
/10
     Load Average   |

      Interface           Traffic               Peak                =
Total
            lo0  in      0.953 KB/s          1.916 KB/s            2.669 =
MB
                 out     0.953 KB/s          1.916 KB/s            2.669 =
MB

           igb1  in      0.063 KB/s          0.128 KB/s           85.663 =
KB
                 out     0.142 KB/s          0.269 KB/s          203.177 =
KB

            ix0  in    215.019 KB/s        679.274 KB/s         1000.871 =
MB
                 out   755.770 KB/s          1.113 MB/s            2.447 =
GB

    Regards
    Steve=20


=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This e.mail is private and confidential between Multiplay (UK) Ltd. and =
the
person or entity to whom it is addressed. In the event of misdirection, =
the
recipient is prohibited from using, copying, printing or otherwise
disseminating it or any information contained in it.=20

In the event of misdirection, illegible or incomplete transmission =
please
telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.

_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"

------=_NextPart_000_00D6_01CD0077.F16AF800--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3807CE6F3BF4B04EB897F4EBF2D258CE5C055123>