Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 1 Mar 2012 12:31:29 -0700
From:      Jason Wolfe <nitroboost@gmail.com>
To:        freebsd-net@freebsd.org
Subject:   Intel 82574L interface wedging - em7.3.2/8.2-STABLE
Message-ID:  <CAAAm0r3Qj%2B2rf8cx54bcyAXGQezcE8J=xXYPq4W-jDy75r8qew@mail.gmail.com>

next in thread | raw e-mail | index | archive | help
So since the 7.3.0/7.3.2 code released out of the "Intel 82574L
interface wedging on em 7.1.9/7.2.3 when MSIX enabled" thread I've
been having some good results in 8.2-STABLE, and 'wedges' are much
less common.  I am however still seeing them rarely, using some fuzzy
math based on uptime on the new code and number of boxes, about once
every 250 days.  MUCH better than prior, but wondering if there is
something else still lingering?  It appears to have the same symptoms
as before with a full buffer, where dropped packets start climbing and
packets out stall.  These servers have MSI-X enabled.

I bounced em0 because dropped packets incremented 1036224 to 1036874
and the interface is not incrementing packets out.

 9:15PM  up 18 days,  5:07, 0 users, load averages: 1.75, 2.25, 2.35

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
	ether 00:25:90:2c:c3:a5
	inet6 fe80::225:90ff:fe2c:c3a4%em0 prefixlen 64 scopeid 0x1
	nd6 options=1<PERFORMNUD>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
	ether 00:25:90:2c:c3:a5
	inet6 fe80::225:90ff:fe2c:c3a5%em1 prefixlen 64 scopeid 0x2
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet 127.0.0.1 netmask 0xff000000
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
	inet 69.28.143.32 netmask 0xffffffff
	inet 69.28.143.33 netmask 0xffffffff
	inet 69.28.143.58 netmask 0xffffffff
	inet 69.28.143.59 netmask 0xffffffff
	inet 69.28.143.60 netmask 0xffffffff
	inet 69.28.143.61 netmask 0xffffffff
	inet 68.142.118.4 netmask 0xffffffff
	inet 68.142.118.254 netmask 0xffffffff
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
	ether 00:25:90:2c:c3:a5
	inet 69.164.10.184 netmask 0xffffff80 broadcast 69.164.10.255
	inet6 fe80::225:90ff:fe2c:c3a5%lagg0 prefixlen 64 scopeid 0x5
	inet6 2607:f4e8:320:14:225:90ff:fe2c:c3a5 prefixlen 64 autoconf
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
	media: Ethernet autoselect
	status: active
	laggproto loadbalance
	laggport: em0 flags=4<ACTIVE>
	laggport: em1 flags=4<ACTIVE>

interrupt                          total       rate
irq3: uart1                         9150          0
cpu0: timer                   3147281076       2000
irq256: em0:rx 0                92406843         58
irq257: em0:tx 0              7707454123       4897
irq258: em0:link                       2          0
irq259: em1:rx 0              9059996943       5757
irq260: em1:tx 0              7697538938       4891
irq261: em1:link                   10325          0
irq262: mps0                  1109270744        704
cpu2: timer                   3147265253       2000
cpu3: timer                   3147265245       2000
cpu1: timer                   3147265253       2000
Total                        38255763895      24310

65436/10254/75690 mbufs in use (current/cache/total)
9092/4636/13728/5956826 mbuf clusters in use (current/cache/total/max)
9092/980 mbuf+clusters out of packet secondary zone in use (current/cache)
49150/877/50027/2978413 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
231143K/15343K/246486K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
195213 requests for I/O initiated by sendfile
0 calls to protocol drain routines

Name    Mtu Network       Address              Ipkts Ierrs Idrop
Opkts Oerrs  Coll Drop
em0    1500 <Link#1>      00:25:90:2c:c3:a5 91137540     0     0
49557533207     0     0 1039464
em0    1500 fe80::225:90f fe80::225:90ff:fe        0     -     -
 5     -     -    -
em1    1500 <Link#2>      00:25:90:2c:c3:a5 46770487879 52357     0
49003644601     0     0 132545
em1    1500 fe80::225:90f fe80::225:90ff:fe        0     -     -
 1     -     -    -
lagg0  1500 <Link#5>      00:25:90:2c:c3:a5 46861296630     0     0
98539892148 1172007     0    0
lagg0  1500 69.164.10.128 69.164.10.184     44434261867     -     -
98536592195     -     -    -
lagg0  1500 fe80::225:90f fe80::225:90ff:fe     2737     -     -
2750     -     -    -
lagg0  1500 2607:f4e8:320 2607:f4e8:320:14: 22188979     -     -
22152621     -     -    -

kern.msgbuf:

Feb 27 21:15:07 cds447 kernel: Interface is RUNNING and ACTIVE
Feb 27 21:15:07 cds447 kernel: em0: hw tdh = 1385, hw tdt = 1385
Feb 27 21:15:07 cds447 kernel: em0: hw rdh = 452, hw rdt = 451
Feb 27 21:15:07 cds447 kernel: em0: Tx Queue Status = 0
Feb 27 21:15:07 cds447 kernel: em0: TX descriptors avail = 2048
Feb 27 21:15:07 cds447 kernel: em0: Tx Descriptors avail failure = 0
Feb 27 21:15:07 cds447 kernel: em0: RX discarded packets = 0
Feb 27 21:15:07 cds447 kernel: em0: RX Next to Check = 452
Feb 27 21:15:07 cds447 kernel: em0: RX Next to Refresh = 451
Feb 27 21:15:07 cds447 kernel: Interface is RUNNING and ACTIVE
Feb 27 21:15:07 cds447 kernel: em1: hw tdh = 221, hw tdt = 342
Feb 27 21:15:07 cds447 kernel: em1: hw rdh = 335, hw rdt = 233
Feb 27 21:15:07 cds447 kernel: em1: Tx Queue Status = 0
Feb 27 21:15:07 cds447 kernel: em1: TX descriptors avail = 2048
Feb 27 21:15:07 cds447 kernel: em1: Tx Descriptors avail failure = 0
Feb 27 21:15:07 cds447 kernel: em1: RX discarded packets = 0
Feb 27 21:15:07 cds447 kernel: em1: RX Next to Check = 832
Feb 27 21:15:07 cds447 kernel: em1: RX Next to Refresh = 838

net.inet.ip.intr_queue_maxlen: 512
net.inet.ip.intr_queue_drops: 0
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.3.2
dev.em.0.%driver: em
dev.em.0.%location: slot=0 function=0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9
subdevice=0x10d3 class=0x020000
dev.em.0.%parent: pci1
dev.em.0.nvm: -1
dev.em.0.debug: -1
dev.em.0.fc: 3
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_processing_limit: 100
dev.em.0.eee_control: 0
dev.em.0.link_irq: 2
dev.em.0.mbuf_alloc_fail: 0
dev.em.0.cluster_alloc_fail: 0
dev.em.0.dropped: 0
dev.em.0.tx_dma_fail: 0
dev.em.0.rx_overruns: 0
dev.em.0.watchdog_timeouts: 0
dev.em.0.device_control: 1074790984
dev.em.0.rx_control: 67141634
dev.em.0.fc_high_water: 18432
dev.em.0.fc_low_water: 16932
dev.em.0.queue0.txd_head: 1385
dev.em.0.queue0.txd_tail: 1385
dev.em.0.queue0.tx_irq: 7707451554
dev.em.0.queue0.no_desc_avail: 0
dev.em.0.queue0.rxd_head: 659
dev.em.0.queue0.rxd_tail: 658
dev.em.0.queue0.rx_irq: 92406848
dev.em.0.mac_stats.excess_coll: 0
dev.em.0.mac_stats.single_coll: 0
dev.em.0.mac_stats.multiple_coll: 0
dev.em.0.mac_stats.late_coll: 0
dev.em.0.mac_stats.collision_count: 0
dev.em.0.mac_stats.symbol_errors: 0
dev.em.0.mac_stats.sequence_errors: 0
dev.em.0.mac_stats.defer_count: 0
dev.em.0.mac_stats.missed_packets: 0
dev.em.0.mac_stats.recv_no_buff: 0
dev.em.0.mac_stats.recv_undersize: 0
dev.em.0.mac_stats.recv_fragmented: 0
dev.em.0.mac_stats.recv_oversize: 0
dev.em.0.mac_stats.recv_jabber: 0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_stats.crc_errs: 0
dev.em.0.mac_stats.alignment_errs: 0
dev.em.0.mac_stats.coll_ext_errs: 0
dev.em.0.mac_stats.xon_recvd: 0
dev.em.0.mac_stats.xon_txd: 0
dev.em.0.mac_stats.xoff_recvd: 0
dev.em.0.mac_stats.xoff_txd: 0
dev.em.0.mac_stats.total_pkts_recvd: 91202036
dev.em.0.mac_stats.good_pkts_recvd: 91202036
dev.em.0.mac_stats.bcast_pkts_recvd: 91083791
dev.em.0.mac_stats.mcast_pkts_recvd: 53524
dev.em.0.mac_stats.rx_frames_64: 91112331
dev.em.0.mac_stats.rx_frames_65_127: 26634
dev.em.0.mac_stats.rx_frames_128_255: 52891
dev.em.0.mac_stats.rx_frames_256_511: 6099
dev.em.0.mac_stats.rx_frames_512_1023: 4081
dev.em.0.mac_stats.rx_frames_1024_1522: 0
dev.em.0.mac_stats.good_octets_recvd: 5845701200
dev.em.0.mac_stats.good_octets_txd: 66047819213524
dev.em.0.mac_stats.total_pkts_txd: 49557613335
dev.em.0.mac_stats.good_pkts_txd: 49557613335
dev.em.0.mac_stats.bcast_pkts_txd: 35
dev.em.0.mac_stats.mcast_pkts_txd: 10502
dev.em.0.mac_stats.tx_frames_64: 191214392
dev.em.0.mac_stats.tx_frames_65_127: 5048362310
dev.em.0.mac_stats.tx_frames_128_255: 37903087
dev.em.0.mac_stats.tx_frames_256_511: 73859562
dev.em.0.mac_stats.tx_frames_512_1023: 491425065
dev.em.0.mac_stats.tx_frames_1024_1522: 43714848919
dev.em.0.mac_stats.tso_txd: 0
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.interrupts.asserts: 3
dev.em.0.interrupts.rx_pkt_timer: 0
dev.em.0.interrupts.rx_abs_timer: 0
dev.em.0.interrupts.tx_pkt_timer: 0
dev.em.0.interrupts.tx_abs_timer: 0
dev.em.0.interrupts.tx_queue_empty: 0
dev.em.0.interrupts.tx_queue_min_thresh: 0
dev.em.0.interrupts.rx_desc_min_thresh: 0
dev.em.0.interrupts.rx_overrun: 0
dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.3.2
dev.em.1.%driver: em
dev.em.1.%location: slot=0 function=0
dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9
subdevice=0x10d3 class=0x020000
dev.em.1.%parent: pci2
dev.em.1.nvm: -1
dev.em.1.debug: -1
dev.em.1.fc: 3
dev.em.1.rx_int_delay: 0
dev.em.1.tx_int_delay: 66
dev.em.1.rx_abs_int_delay: 66
dev.em.1.tx_abs_int_delay: 66
dev.em.1.rx_processing_limit: 100
dev.em.1.eee_control: 0
dev.em.1.link_irq: 9701
dev.em.1.mbuf_alloc_fail: 0
dev.em.1.cluster_alloc_fail: 0
dev.em.1.dropped: 0
dev.em.1.tx_dma_fail: 0
dev.em.1.rx_overruns: 0
dev.em.1.watchdog_timeouts: 0
dev.em.1.device_control: 1074790984
dev.em.1.rx_control: 67141634
dev.em.1.fc_high_water: 18432
dev.em.1.fc_low_water: 16932
dev.em.1.queue0.txd_head: 428
dev.em.1.queue0.txd_tail: 592
dev.em.1.queue0.tx_irq: 7697553110
dev.em.1.queue0.no_desc_avail: 0
dev.em.1.queue0.rxd_head: 190
dev.em.1.queue0.rxd_tail: 189
dev.em.1.queue0.rx_irq: 8990697150
dev.em.1.mac_stats.excess_coll: 0
dev.em.1.mac_stats.single_coll: 0
dev.em.1.mac_stats.multiple_coll: 0
dev.em.1.mac_stats.late_coll: 0
dev.em.1.mac_stats.collision_count: 0
dev.em.1.mac_stats.symbol_errors: 0
dev.em.1.mac_stats.sequence_errors: 0
dev.em.1.mac_stats.defer_count: 0
dev.em.1.mac_stats.missed_packets: 52357
dev.em.1.mac_stats.recv_no_buff: 12234
dev.em.1.mac_stats.recv_undersize: 0
dev.em.1.mac_stats.recv_fragmented: 0
dev.em.1.mac_stats.recv_oversize: 0
dev.em.1.mac_stats.recv_jabber: 0
dev.em.1.mac_stats.recv_errs: 0
dev.em.1.mac_stats.crc_errs: 0
dev.em.1.mac_stats.alignment_errs: 0
dev.em.1.mac_stats.coll_ext_errs: 0
dev.em.1.mac_stats.xon_recvd: 0
dev.em.1.mac_stats.xon_txd: 0
dev.em.1.mac_stats.xoff_recvd: 0
dev.em.1.mac_stats.xoff_txd: 0
dev.em.1.mac_stats.total_pkts_recvd: 46770604210
dev.em.1.mac_stats.good_pkts_recvd: 46770551853
dev.em.1.mac_stats.bcast_pkts_recvd: 91073541
dev.em.1.mac_stats.mcast_pkts_recvd: 53519
dev.em.1.mac_stats.rx_frames_64: 14339105496
dev.em.1.mac_stats.rx_frames_65_127: 23186724273
dev.em.1.mac_stats.rx_frames_128_255: 38625485
dev.em.1.mac_stats.rx_frames_256_511: 116531779
dev.em.1.mac_stats.rx_frames_512_1023: 165521693
dev.em.1.mac_stats.rx_frames_1024_1522: 8924043127
dev.em.1.mac_stats.good_octets_recvd: 16290987983606
dev.em.1.mac_stats.good_octets_txd: 65599822022242
dev.em.1.mac_stats.total_pkts_txd: 49003789456
dev.em.1.mac_stats.good_pkts_txd: 49003789452
dev.em.1.mac_stats.bcast_pkts_txd: 10085
dev.em.1.mac_stats.mcast_pkts_txd: 11
dev.em.1.mac_stats.tx_frames_64: 170250212
dev.em.1.mac_stats.tx_frames_65_127: 4806984601
dev.em.1.mac_stats.tx_frames_128_255: 36525028
dev.em.1.mac_stats.tx_frames_256_511: 71876326
dev.em.1.mac_stats.tx_frames_512_1023: 482360156
dev.em.1.mac_stats.tx_frames_1024_1522: 43435793133
dev.em.1.mac_stats.tso_txd: 0
dev.em.1.mac_stats.tso_ctx_fail: 0
dev.em.1.interrupts.asserts: 7943
dev.em.1.interrupts.rx_pkt_timer: 1
dev.em.1.interrupts.rx_abs_timer: 0
dev.em.1.interrupts.tx_pkt_timer: 0
dev.em.1.interrupts.tx_abs_timer: 1
dev.em.1.interrupts.tx_queue_empty: 0
dev.em.1.interrupts.tx_queue_min_thresh: 0
dev.em.1.interrupts.rx_desc_min_thresh: 0
dev.em.1.interrupts.rx_overrun: 6
hw.em.eee_setting: 0
hw.em.rx_process_limit: 100
hw.em.enable_msix: 1
hw.em.sbp: 0
hw.em.smart_pwr_down: 0
hw.em.txd: 2048
hw.em.rxd: 2048
hw.em.rx_abs_int_delay: 66
hw.em.tx_abs_int_delay: 66
hw.em.rx_int_delay: 0
hw.em.tx_int_delay: 66

Jason



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAAm0r3Qj%2B2rf8cx54bcyAXGQezcE8J=xXYPq4W-jDy75r8qew>