Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 26 May 2015 11:37:33 -0300
From:      Marcelo Gondim <gondim@bsdinfo.com.br>
To:        freebsd-net@freebsd.org
Subject:   Re: FreeBSD 9.3 - Intel X520-SR2 stops passing packets
Message-ID:  <5564852D.8040008@bsdinfo.com.br>
In-Reply-To: <5560C395.8020807@farrokhi.net>
References:  <C8B78CF4-184C-4BFE-B848-37A91A3B696B@gmail.com> <5560C395.8020807@farrokhi.net>

next in thread | previous in thread | raw e-mail | index | archive | help
On 23-05-2015 15:14, Babak Farrokhi wrote:
> Look at the interrupts per queue. 500,000 is the maximum and it is the
> reason your interface is not accepting new packets.
>
>> Guy Helmer <mailto:guy.helmer@gmail.com>
>> May 21, 2015 at 6:03 PM
>> I’ve noticed that there have been reports of problems with Intel
>> X520-SR2 network interfaces stopping working. I think I’m seeing a
>> similar issue where the 10Gb interfaces stop receiving traffic
>> (they’re being used in promiscuous mode to sniff traffic from a tap).
>> ifconfig shows the interfaces are still active and the links are OK.
>> ifconfig down/up restores activity. I’ve changed
>> hw.intr_storm_threshold=8000 but I couldn’t tell if the interrupt
>> storm threshold had been triggered at the time the interfaces stopped
>> passing traffic.
>>
>> Output from sysctl:
>>
>> # sysctl dev.ix.0
>> dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version
>> - 2.5.15
>> dev.ix.0.%driver: ix
>> dev.ix.0.%location: slot=0 function=0
>> dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10fb subvendor=0x8086
>> subdevice=0x0003 class=0x020000
>> dev.ix.0.%parent: pci4
>> dev.ix.0.fc: 0
>> dev.ix.0.enable_aim: 1
>> dev.ix.0.advertise_speed: 0
>> dev.ix.0.dropped: 0
>> dev.ix.0.mbuf_defrag_failed: 0
>> dev.ix.0.watchdog_events: 0
>> dev.ix.0.link_irq: 3
>> dev.ix.0.queue0.interrupt_rate: 500000
>> dev.ix.0.queue0.irqs: 454449470
>> dev.ix.0.queue0.txd_head: 0
>> dev.ix.0.queue0.txd_tail: 0
>> dev.ix.0.queue0.tso_tx: 0
>> dev.ix.0.queue0.no_tx_dma_setup: 0
>> dev.ix.0.queue0.no_desc_avail: 0
>> dev.ix.0.queue0.tx_packets: 0
>> dev.ix.0.queue0.rxd_head: 1437
>> dev.ix.0.queue0.rxd_tail: 1436
>> dev.ix.0.queue0.rx_packets: 547499168
>> dev.ix.0.queue0.rx_bytes: 87201112584
>> dev.ix.0.queue0.rx_copies: 7934870
>> dev.ix.0.queue0.lro_queued: 0
>> dev.ix.0.queue0.lro_flushed: 0
>> dev.ix.0.queue1.interrupt_rate: 500000
>> dev.ix.0.queue1.irqs: 466235043
>> dev.ix.0.queue1.txd_head: 0
>> dev.ix.0.queue1.txd_tail: 0
>> dev.ix.0.queue1.tso_tx: 0
>> dev.ix.0.queue1.no_tx_dma_setup: 0
>> dev.ix.0.queue1.no_desc_avail: 0
>> dev.ix.0.queue1.tx_packets: 0
>> dev.ix.0.queue1.rxd_head: 277
>> dev.ix.0.queue1.rxd_tail: 276
>> dev.ix.0.queue1.rx_packets: 547668680
>> dev.ix.0.queue1.rx_bytes: 86205679601
>> dev.ix.0.queue1.rx_copies: 7846653
>> dev.ix.0.queue1.lro_queued: 0
>> dev.ix.0.queue1.lro_flushed: 0
>> dev.ix.0.queue2.interrupt_rate: 500000
>> dev.ix.0.queue2.irqs: 473958473
>> dev.ix.0.queue2.txd_head: 0
>> dev.ix.0.queue2.txd_tail: 0
>> dev.ix.0.queue2.tso_tx: 0
>> dev.ix.0.queue2.no_tx_dma_setup: 0
>> dev.ix.0.queue2.no_desc_avail: 0
>> dev.ix.0.queue2.tx_packets: 0
>> dev.ix.0.queue2.rxd_head: 576
>> dev.ix.0.queue2.rxd_tail: 575
>> dev.ix.0.queue2.rx_packets: 555704840
>> dev.ix.0.queue2.rx_bytes: 87294164455
>> dev.ix.0.queue2.rx_copies: 8297211
>> dev.ix.0.queue2.lro_queued: 0
>> dev.ix.0.queue2.lro_flushed: 0
>> dev.ix.0.queue3.interrupt_rate: 500000
>> dev.ix.0.queue3.irqs: 477587504
>> dev.ix.0.queue3.txd_head: 0
>> dev.ix.0.queue3.txd_tail: 0
>> dev.ix.0.queue3.tso_tx: 0
>> dev.ix.0.queue3.no_tx_dma_setup: 0
>> dev.ix.0.queue3.no_desc_avail: 0
>> dev.ix.0.queue3.tx_packets: 0
>> dev.ix.0.queue3.rxd_head: 267
>> dev.ix.0.queue3.rxd_tail: 266
>> dev.ix.0.queue3.rx_packets: 559921557
>> dev.ix.0.queue3.rx_bytes: 86832161258
>> dev.ix.0.queue3.rx_copies: 7918011
>> dev.ix.0.queue3.lro_queued: 0
>> dev.ix.0.queue3.lro_flushed: 0
>> dev.ix.0.queue4.interrupt_rate: 500000
>> dev.ix.0.queue4.irqs: 558339677
>> dev.ix.0.queue4.txd_head: 0
>> dev.ix.0.queue4.txd_tail: 0
>> dev.ix.0.queue4.tso_tx: 0
>> dev.ix.0.queue4.no_tx_dma_setup: 0
>> dev.ix.0.queue4.no_desc_avail: 0
>> dev.ix.0.queue4.tx_packets: 0
>> dev.ix.0.queue4.rxd_head: 1240
>> dev.ix.0.queue4.rxd_tail: 1239
>> dev.ix.0.queue4.rx_packets: 646909190
>> dev.ix.0.queue4.rx_bytes: 87117307815
>> dev.ix.0.queue4.rx_copies: 7944848
>> dev.ix.0.queue4.lro_queued: 0
>> dev.ix.0.queue4.lro_flushed: 0
>> dev.ix.0.queue5.interrupt_rate: 500000
>> dev.ix.0.queue5.irqs: 467836647
>> dev.ix.0.queue5.txd_head: 0
>> dev.ix.0.queue5.txd_tail: 0
>> dev.ix.0.queue5.tso_tx: 0
>> dev.ix.0.queue5.no_tx_dma_setup: 0
>> dev.ix.0.queue5.no_desc_avail: 0
>> dev.ix.0.queue5.tx_packets: 0
>> dev.ix.0.queue5.rxd_head: 1411
>> dev.ix.0.queue5.rxd_tail: 1410
>> dev.ix.0.queue5.rx_packets: 549666835
>> dev.ix.0.queue5.rx_bytes: 84671540121
>> dev.ix.0.queue5.rx_copies: 8258025
>> dev.ix.0.queue5.lro_queued: 0
>> dev.ix.0.queue5.lro_flushed: 0
>> dev.ix.0.queue6.interrupt_rate: 500000
>> dev.ix.0.queue6.irqs: 490798561
>> dev.ix.0.queue6.txd_head: 0
>> dev.ix.0.queue6.txd_tail: 0
>> dev.ix.0.queue6.tso_tx: 0
>> dev.ix.0.queue6.no_tx_dma_setup: 0
>> dev.ix.0.queue6.no_desc_avail: 0
>> dev.ix.0.queue6.tx_packets: 0
>> dev.ix.0.queue6.rxd_head: 160
>> dev.ix.0.queue6.rxd_tail: 159
>> dev.ix.0.queue6.rx_packets: 590187606
>> dev.ix.0.queue6.rx_bytes: 92115960421
>> dev.ix.0.queue6.rx_copies: 8262802
>> dev.ix.0.queue6.lro_queued: 0
>> dev.ix.0.queue6.lro_flushed: 0
>> dev.ix.0.queue7.interrupt_rate: 500000
>> dev.ix.0.queue7.irqs: 471051540
>> dev.ix.0.queue7.txd_head: 0
>> dev.ix.0.queue7.txd_tail: 0
>> dev.ix.0.queue7.tso_tx: 0
>> dev.ix.0.queue7.no_tx_dma_setup: 0
>> dev.ix.0.queue7.no_desc_avail: 0
>> dev.ix.0.queue7.tx_packets: 0
>> dev.ix.0.queue7.rxd_head: 640
>> dev.ix.0.queue7.rxd_tail: 639
>> dev.ix.0.queue7.rx_packets: 553362982
>> dev.ix.0.queue7.rx_bytes: 84470102891
>> dev.ix.0.queue7.rx_copies: 7954102
>> dev.ix.0.queue7.lro_queued: 0
>> dev.ix.0.queue7.lro_flushed: 0
>> dev.ix.0.mac_stats.crc_errs: 2091
>> dev.ix.0.mac_stats.ill_errs: 26
>> dev.ix.0.mac_stats.byte_errs: 140
>> dev.ix.0.mac_stats.short_discards: 0
>> dev.ix.0.mac_stats.local_faults: 0
>> dev.ix.0.mac_stats.remote_faults: 0
>> dev.ix.0.mac_stats.rec_len_errs: 0
>> dev.ix.0.mac_stats.xon_txd: 0
>> dev.ix.0.mac_stats.xon_recvd: 0
>> dev.ix.0.mac_stats.xoff_txd: 0
>> dev.ix.0.mac_stats.xoff_recvd: 0
>> dev.ix.0.mac_stats.total_octets_rcvd: 17956217280225
>> dev.ix.0.mac_stats.good_octets_rcvd: 17945313085409
>> dev.ix.0.mac_stats.total_pkts_rcvd: 15243381335
>> dev.ix.0.mac_stats.good_pkts_rcvd: 4550864350
>> dev.ix.0.mac_stats.mcast_pkts_rcvd: 0
>> dev.ix.0.mac_stats.bcast_pkts_rcvd: 0
>> dev.ix.0.mac_stats.rx_frames_64: 721599526
>> dev.ix.0.mac_stats.rx_frames_65_127: 950131946
>> dev.ix.0.mac_stats.rx_frames_128_255: 918463009
>> dev.ix.0.mac_stats.rx_frames_256_511: 291858186
>> dev.ix.0.mac_stats.rx_frames_512_1023: 237479208
>> dev.ix.0.mac_stats.rx_frames_1024_1522: 12114578925
>> dev.ix.0.mac_stats.recv_undersized: 0
>> dev.ix.0.mac_stats.recv_fragmented: 0
>> dev.ix.0.mac_stats.recv_oversized: 0
>> dev.ix.0.mac_stats.recv_jabberd: 5
>> dev.ix.0.mac_stats.management_pkts_rcvd: 0
>> dev.ix.0.mac_stats.management_pkts_drpd: 0
>> dev.ix.0.mac_stats.checksum_errs: 4379061
>> dev.ix.0.mac_stats.good_octets_txd: 0
>> dev.ix.0.mac_stats.total_pkts_txd: 0
>> dev.ix.0.mac_stats.good_pkts_txd: 0
>> dev.ix.0.mac_stats.bcast_pkts_txd: 0
>> dev.ix.0.mac_stats.mcast_pkts_txd: 0
>> dev.ix.0.mac_stats.management_pkts_txd: 0
>> dev.ix.0.mac_stats.tx_frames_64: 0
>> dev.ix.0.mac_stats.tx_frames_65_127: 0
>> dev.ix.0.mac_stats.tx_frames_128_255: 0
>> dev.ix.0.mac_stats.tx_frames_256_511: 0
>> dev.ix.0.mac_stats.tx_frames_512_1023: 0
>> dev.ix.0.mac_stats.tx_frames_1024_1522: 0
>>
>> # sysctl dev.ix.1
>> dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version
>> - 2.5.15
>> dev.ix.1.%driver: ix
>> dev.ix.1.%location: slot=0 function=1
>> dev.ix.1.%pnpinfo: vendor=0x8086 device=0x10fb subvendor=0x8086
>> subdevice=0x0003 class=0x020000
>> dev.ix.1.%parent: pci4
>> dev.ix.1.fc: 0
>> dev.ix.1.enable_aim: 1
>> dev.ix.1.advertise_speed: 0
>> dev.ix.1.dropped: 0
>> dev.ix.1.mbuf_defrag_failed: 0
>> dev.ix.1.watchdog_events: 0
>> dev.ix.1.link_irq: 3
>> dev.ix.1.queue0.interrupt_rate: 500000
>> dev.ix.1.queue0.irqs: 537134504
>> dev.ix.1.queue0.txd_head: 0
>> dev.ix.1.queue0.txd_tail: 0
>> dev.ix.1.queue0.tso_tx: 0
>> dev.ix.1.queue0.no_tx_dma_setup: 0
>> dev.ix.1.queue0.no_desc_avail: 0
>> dev.ix.1.queue0.tx_packets: 0
>> dev.ix.1.queue0.rxd_head: 1757
>> dev.ix.1.queue0.rxd_tail: 1756
>> dev.ix.1.queue0.rx_packets: 565486932
>> dev.ix.1.queue0.rx_bytes: 7763122874
>> dev.ix.1.queue0.rx_copies: 40953968
>> dev.ix.1.queue0.lro_queued: 0
>> dev.ix.1.queue0.lro_flushed: 0
>> dev.ix.1.queue1.interrupt_rate: 500000
>> dev.ix.1.queue1.irqs: 561383741
>> dev.ix.1.queue1.txd_head: 0
>> dev.ix.1.queue1.txd_tail: 0
>> dev.ix.1.queue1.tso_tx: 0
>> dev.ix.1.queue1.no_tx_dma_setup: 0
>> dev.ix.1.queue1.no_desc_avail: 0
>> dev.ix.1.queue1.tx_packets: 0
>> dev.ix.1.queue1.rxd_head: 138
>> dev.ix.1.queue1.rxd_tail: 137
>> dev.ix.1.queue1.rx_packets: 577262064
>> dev.ix.1.queue1.rx_bytes: 8709306631
>> dev.ix.1.queue1.rx_copies: 40844466
>> dev.ix.1.queue1.lro_queued: 0
>> dev.ix.1.queue1.lro_flushed: 0
>> dev.ix.1.queue2.interrupt_rate: 500000
>> dev.ix.1.queue2.irqs: 547852317
>> dev.ix.1.queue2.txd_head: 0
>> dev.ix.1.queue2.txd_tail: 0
>> dev.ix.1.queue2.tso_tx: 0
>> dev.ix.1.queue2.no_tx_dma_setup: 0
>> dev.ix.1.queue2.no_desc_avail: 0
>> dev.ix.1.queue2.tx_packets: 0
>> dev.ix.1.queue2.rxd_head: 386
>> dev.ix.1.queue2.rxd_tail: 385
>> dev.ix.1.queue2.rx_packets: 562301518
>> dev.ix.1.queue2.rx_bytes: 6698895889
>> dev.ix.1.queue2.rx_copies: 40867897
>> dev.ix.1.queue2.lro_queued: 0
>> dev.ix.1.queue2.lro_flushed: 0
>> dev.ix.1.queue3.interrupt_rate: 500000
>> dev.ix.1.queue3.irqs: 551254360
>> dev.ix.1.queue3.txd_head: 0
>> dev.ix.1.queue3.txd_tail: 0
>> dev.ix.1.queue3.tso_tx: 0
>> dev.ix.1.queue3.no_tx_dma_setup: 0
>> dev.ix.1.queue3.no_desc_avail: 0
>> dev.ix.1.queue3.tx_packets: 0
>> dev.ix.1.queue3.rxd_head: 1446
>> dev.ix.1.queue3.rxd_tail: 1445
>> dev.ix.1.queue3.rx_packets: 566052657
>> dev.ix.1.queue3.rx_bytes: 8010009389
>> dev.ix.1.queue3.rx_copies: 41116971
>> dev.ix.1.queue3.lro_queued: 0
>> dev.ix.1.queue3.lro_flushed: 0
>> dev.ix.1.queue4.interrupt_rate: 500000
>> dev.ix.1.queue4.irqs: 546581703
>> dev.ix.1.queue4.txd_head: 0
>> dev.ix.1.queue4.txd_tail: 0
>> dev.ix.1.queue4.tso_tx: 0
>> dev.ix.1.queue4.no_tx_dma_setup: 0
>> dev.ix.1.queue4.no_desc_avail: 0
>> dev.ix.1.queue4.tx_packets: 0
>> dev.ix.1.queue4.rxd_head: 965
>> dev.ix.1.queue4.rxd_tail: 964
>> dev.ix.1.queue4.rx_packets: 561519824
>> dev.ix.1.queue4.rx_bytes: 7656671816
>> dev.ix.1.queue4.rx_copies: 41183608
>> dev.ix.1.queue4.lro_queued: 0
>> dev.ix.1.queue4.lro_flushed: 0
>> dev.ix.1.queue5.interrupt_rate: 500000
>> dev.ix.1.queue5.irqs: 557099892
>> dev.ix.1.queue5.txd_head: 0
>> dev.ix.1.queue5.txd_tail: 0
>> dev.ix.1.queue5.tso_tx: 0
>> dev.ix.1.queue5.no_tx_dma_setup: 0
>> dev.ix.1.queue5.no_desc_avail: 0
>> dev.ix.1.queue5.tx_packets: 0
>> dev.ix.1.queue5.rxd_head: 1788
>> dev.ix.1.queue5.rxd_tail: 1787
>> dev.ix.1.queue5.rx_packets: 572588639
>> dev.ix.1.queue5.rx_bytes: 7259699024
>> dev.ix.1.queue5.rx_copies: 43207640
>> dev.ix.1.queue5.lro_queued: 0
>> dev.ix.1.queue5.lro_flushed: 0
>> dev.ix.1.queue6.interrupt_rate: 500000
>> dev.ix.1.queue6.irqs: 574139280
>> dev.ix.1.queue6.txd_head: 0
>> dev.ix.1.queue6.txd_tail: 0
>> dev.ix.1.queue6.tso_tx: 0
>> dev.ix.1.queue6.no_tx_dma_setup: 0
>> dev.ix.1.queue6.no_desc_avail: 0
>> dev.ix.1.queue6.tx_packets: 0
>> dev.ix.1.queue6.rxd_head: 45
>> dev.ix.1.queue6.rxd_tail: 44
>> dev.ix.1.queue6.rx_packets: 589160795
>> dev.ix.1.queue6.rx_bytes: 7475849844
>> dev.ix.1.queue6.rx_copies: 40589940
>> dev.ix.1.queue6.lro_queued: 0
>> dev.ix.1.queue6.lro_flushed: 0
>> dev.ix.1.queue7.interrupt_rate: 500000
>> dev.ix.1.queue7.irqs: 552769977
>> dev.ix.1.queue7.txd_head: 0
>> dev.ix.1.queue7.txd_tail: 0
>> dev.ix.1.queue7.tso_tx: 0
>> dev.ix.1.queue7.no_tx_dma_setup: 0
>> dev.ix.1.queue7.no_desc_avail: 0
>> dev.ix.1.queue7.tx_packets: 0
>> dev.ix.1.queue7.rxd_head: 1050
>> dev.ix.1.queue7.rxd_tail: 1049
>> dev.ix.1.queue7.rx_packets: 567580543
>> dev.ix.1.queue7.rx_bytes: 7210216689
>> dev.ix.1.queue7.rx_copies: 41856967
>> dev.ix.1.queue7.lro_queued: 0
>> dev.ix.1.queue7.lro_flushed: 0
>> dev.ix.1.mac_stats.crc_errs: 40044743
>> dev.ix.1.mac_stats.ill_errs: 4347098
>> dev.ix.1.mac_stats.byte_errs: 7192103
>> dev.ix.1.mac_stats.short_discards: 49169
>> dev.ix.1.mac_stats.local_faults: 0
>> dev.ix.1.mac_stats.remote_faults: 0
>> dev.ix.1.mac_stats.rec_len_errs: 41772
>> dev.ix.1.mac_stats.xon_txd: 0
>> dev.ix.1.mac_stats.xon_recvd: 0
>> dev.ix.1.mac_stats.xoff_txd: 0
>> dev.ix.1.mac_stats.xoff_recvd: 0
>> dev.ix.1.mac_stats.total_octets_rcvd: 1741353301146
>> dev.ix.1.mac_stats.good_octets_rcvd: 1704100700961
>> dev.ix.1.mac_stats.total_pkts_rcvd: 9354020520
>> dev.ix.1.mac_stats.good_pkts_rcvd: 4561867527
>> dev.ix.1.mac_stats.mcast_pkts_rcvd: 139746
>> dev.ix.1.mac_stats.bcast_pkts_rcvd: 0
>> dev.ix.1.mac_stats.rx_frames_64: 3314959123
>> dev.ix.1.mac_stats.rx_frames_65_127: 4610233544
>> dev.ix.1.mac_stats.rx_frames_128_255: 256517169
>> dev.ix.1.mac_stats.rx_frames_256_511: 304326606
>> dev.ix.1.mac_stats.rx_frames_512_1023: 223999237
>> dev.ix.1.mac_stats.rx_frames_1024_1522: 591102680
>> dev.ix.1.mac_stats.recv_undersized: 0
>> dev.ix.1.mac_stats.recv_fragmented: 0
>> dev.ix.1.mac_stats.recv_oversized: 0
>> dev.ix.1.mac_stats.recv_jabberd: 71008
>> dev.ix.1.mac_stats.management_pkts_rcvd: 0
>> dev.ix.1.mac_stats.management_pkts_drpd: 0
>> dev.ix.1.mac_stats.checksum_errs: 3901883
>> dev.ix.1.mac_stats.good_octets_txd: 0
>> dev.ix.1.mac_stats.total_pkts_txd: 0
>> dev.ix.1.mac_stats.good_pkts_txd: 0
>> dev.ix.1.mac_stats.bcast_pkts_txd: 0
>> dev.ix.1.mac_stats.mcast_pkts_txd: 0
>> dev.ix.1.mac_stats.management_pkts_txd: 0
>> dev.ix.1.mac_stats.tx_frames_64: 0
>> dev.ix.1.mac_stats.tx_frames_65_127: 0
>> dev.ix.1.mac_stats.tx_frames_128_255: 0
>> dev.ix.1.mac_stats.tx_frames_256_511: 0
>> dev.ix.1.mac_stats.tx_frames_512_1023: 0
>> dev.ix.1.mac_stats.tx_frames_1024_1522: 0
>> _______________________________________________
>> freebsd-net@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
Hi,

I had this problem and one day updated the system 10.1- RELEASE to 10.1- 
STABLE and the problem stopped. I was one years with this problem and a 
script running and testing the interface when the interface stopped 
working I was doing exactly what you did. Today I no longer have that 
problem anymore.

I'm using 10.1-STABLE r281235

Gondim



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5564852D.8040008>