Date: Wed, 26 Oct 2011 14:15:12 +0330 From: Hooman Fazaeli <hoomanfazaeli@gmail.com> To: Jason Wolfe <nitroboost@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled Message-ID: <4EA7E4B8.9050003@gmail.com> In-Reply-To: <CAAAm0r1iS3z-7CBJ=xYDf%2BJOA1Q2nU0O54Twbyb7FjvgWHjKVw@mail.gmail.com> References: <CAAAm0r0RXEJo4UiKS=Ui0e5OQTg6sg-xcYf3mYB5%2Bvk8i8557w@mail.gmail.com> <4E8F157A.40702@sentex.net> <CAAAm0r2JH43Rct7UxQK2duH1p43Nepnj5mpb6bXo==DPayhJLg@mail.gmail.com> <4E8F51D4.1060509@sentex.net> <CACqU3MVwLaepFymZJkaVk6p=SpykGhqs=VYFjLh9fP9S=AxDhg@mail.gmail.com> <CAAAm0r1DKvoL9=Ket9up=4%2B5xiCzTTZJK99FhF9jcCA28B0M%2BA@mail.gmail.com> <CAAAm0r3XdsMHZh%2BP_NF-txZasdExzwZ8ymmGQgGhJQds0fOiBQ@mail.gmail.com> <CAAAm0r1iS3z-7CBJ=xYDf%2BJOA1Q2nU0O54Twbyb7FjvgWHjKVw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Jason Have you tried: hw.em.fc_setting="0" (in loader.conf) ifconfig emX -tso -lro -rxcsum -txcsum -vlanhwtag -wol with MSIX and no multiqueue. Advanced features has always been a source of problem. It is worth a try and help to narrow down possibilities. It would also be helpful if you provide 'ifconfig' output when the problem happens. And a question: Does interface RX also hangs or it is just TX? On 10/26/2011 12:25 AM, Jason Wolfe wrote: > On Fri, Oct 7, 2011 at 2:14 PM, Jason Wolfe<nitroboost@gmail.com> wrote: > >> Bumping rx/tx descriptors to 2048 was actually for performance reasons and >> not to try to get around the issue. I did some fairly in depth testing and >> found under heavy load it performed the best with those settings. >> >> As mentioned on the other thread I'll re enable MSI-X on a few servers here >> and collect uptime and the kernel msgbuf in addition. I'll bump the >> descriptors down to 512 to try and increase our chances and compile the >> driver with EM_MULTIQUEUE also. >> >> Jason >> > Hi there, > > So I have a small pool of server running EM_MULTIQUEUE with lower > descriptors as promised and just received an alert of an event. I have a > fairly large pool of servers on the same hardware running the same OS/driver > sans MSI-X and multiqueue with not a single 'wedge' event in about 2 months > now, and it seems multiqueue has not changed the commonality of the issue. > Here is my loader.conf followed by everything collected: > > net.inet.tcp.tcbhashsize="4096" > net.inet.tcp.syncache.hashsize="1024" > net.inet.tcp.syncache.bucketlimit="512" > net.inet.tcp.syncache.cachelimit="65536" > net.inet.tcp.hostcache.hashsize="1024" > net.inet.tcp.hostcache.bucketlimit="512" > net.inet.tcp.hostcache.cachelimit="65536" > hw.em.rxd="512" > hw.em.txd="512" > cc_cubic_load="YES" > > I bounced em1 because dropped packets incremented 1386169 to 1386355 and the > interface is not incrementing packets out. > > 1:30PM up 4 days, 6:19, 0 users, load averages: 0.18, 0.38, 0.42 > > interrupt total rate > irq3: uart1 5816 0 > cpu0: timer 736655476 2000 > irq256: em0:rx 0 38122306 103 > irq257: em0:tx 0 1605535054 4359 > irq258: em0:link 1 0 > irq259: em1:rx 0 2192460862 5952 > irq260: em1:tx 0 1599049303 4341 > irq261: em1:link 4172 0 > irq262: mps0 212448927 576 > cpu2: timer 736647277 2000 > cpu3: timer 736647302 2000 > cpu1: timer 736647302 2000 > Total 8594223798 23333 > > 27653/6022/33675 mbufs in use (current/cache/total) > 3054/3196/6250/5700670 mbuf clusters in use (current/cache/total/max) > 3054/1041 mbuf+clusters out of packet secondary zone in use (current/cache) > 23266/1642/24908/2850335 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 106085K/14465K/120550K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 22 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop > em0 1500<Link#1> 00:25:90:1f:f5:7d 38575296 0 0 6300959828 0 0 706638 > em0 1500 fe80:1::225:9 fe80:1::225:90ff: 0 - - 3 - - - > em1 1500<Link#2> 00:25:90:1f:f5:7d 6091053202 22415 0 6327642657 0 0 > 1386797 > em1 1500 fe80:2::225:9 fe80:2::225:90ff: 0 - - 1 - - - > lagg0 1500<Link#5> 00:25:90:1f:f5:7d 6129556798 0 0 12627493094 2093435 0 0 > > lagg0 1500 69.164.38.0/2 69.164.38.93 5429109508 - - 12630422599 - - - > lagg0 1500 fe80:5::225:9 fe80:5::225:90ff: 12 - - 17 - - - > lagg0 1500 2607:f4e8:310 2607:f4e8:310:12: 13655 - - 13663 - - - > > kern.msgbuf: > > Oct 25 13:30:04 cds1043 kernel: Interface is RUNNING and INACTIVE > Oct 25 13:30:04 cds1043 kernel: em0: hw tdh = 105, hw tdt = 158 > Oct 25 13:30:04 cds1043 kernel: em0: hw rdh = 191, hw rdt = 190 > Oct 25 13:30:04 cds1043 kernel: em0: Tx Queue Status = 0 > Oct 25 13:30:04 cds1043 kernel: em0: TX descriptors avail = 422 > Oct 25 13:30:04 cds1043 kernel: em0: Tx Descriptors avail failure = 0 > Oct 25 13:30:04 cds1043 kernel: em0: RX discarded packets = 0 > Oct 25 13:30:04 cds1043 kernel: em0: RX Next to Check = 192 > Oct 25 13:30:04 cds1043 kernel: em0: RX Next to Refresh = 191 > Oct 25 13:30:04 cds1043 kernel: Interface is RUNNING and INACTIVE > Oct 25 13:30:04 cds1043 kernel: em1: hw tdh = 159, hw tdt = 159 > Oct 25 13:30:04 cds1043 kernel: em1: hw rdh = 193, hw rdt = 191 > Oct 25 13:30:04 cds1043 kernel: em1: Tx Queue Status = 0 > Oct 25 13:30:04 cds1043 kernel: em1: TX descriptors avail = 512 > Oct 25 13:30:04 cds1043 kernel: em1: Tx Descriptors avail failure = 0 > Oct 25 13:30:04 cds1043 kernel: em1: RX discarded packets = 0 > Oct 25 13:30:04 cds1043 kernel: em1: RX Next to Check = 407 > Oct 25 13:30:04 cds1043 kernel: em1: RX Next to Refresh = 436 > > net.inet.ip.intr_queue_maxlen: 512 > net.inet.ip.intr_queue_drops: 0 > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=0 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 > subdevice=0x10d3 class=0x020000 > dev.em.0.%parent: pci1 > dev.em.0.nvm: -1 > dev.em.0.debug: -1 > dev.em.0.rx_int_delay: 0 > dev.em.0.tx_int_delay: 66 > dev.em.0.rx_abs_int_delay: 66 > dev.em.0.tx_abs_int_delay: 66 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 3 > dev.em.0.eee_control: 0 > dev.em.0.link_irq: 1 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 0 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1074790984 > dev.em.0.rx_control: 67141634 > dev.em.0.fc_high_water: 18432 > dev.em.0.fc_low_water: 16932 > dev.em.0.queue0.txd_head: 208 > dev.em.0.queue0.txd_tail: 208 > dev.em.0.queue0.tx_irq: 1605545961 > dev.em.0.queue0.no_desc_avail: 0 > dev.em.0.queue0.rxd_head: 348 > dev.em.0.queue0.rxd_tail: 347 > dev.em.0.queue0.rx_irq: 38122461 > dev.em.0.mac_stats.excess_coll: 0 > dev.em.0.mac_stats.single_coll: 0 > dev.em.0.mac_stats.multiple_coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_count: 0 > dev.em.0.mac_stats.symbol_errors: 0 > dev.em.0.mac_stats.sequence_errors: 0 > dev.em.0.mac_stats.defer_count: 0 > dev.em.0.mac_stats.missed_packets: 0 > dev.em.0.mac_stats.recv_no_buff: 0 > dev.em.0.mac_stats.recv_undersize: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_oversize: 0 > dev.em.0.mac_stats.recv_jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_errs: 0 > dev.em.0.mac_stats.coll_ext_errs: 0 > dev.em.0.mac_stats.xon_recvd: 0 > dev.em.0.mac_stats.xon_txd: 0 > dev.em.0.mac_stats.xoff_recvd: 0 > dev.em.0.mac_stats.xoff_txd: 0 > dev.em.0.mac_stats.total_pkts_recvd: 38588018 > dev.em.0.mac_stats.good_pkts_recvd: 38588018 > dev.em.0.mac_stats.bcast_pkts_recvd: 38572739 > dev.em.0.mac_stats.mcast_pkts_recvd: 1868 > dev.em.0.mac_stats.rx_frames_64: 38572811 > dev.em.0.mac_stats.rx_frames_65_127: 2762 > dev.em.0.mac_stats.rx_frames_128_255: 12186 > dev.em.0.mac_stats.rx_frames_256_511: 236 > dev.em.0.mac_stats.rx_frames_512_1023: 23 > dev.em.0.mac_stats.rx_frames_1024_1522: 0 > dev.em.0.mac_stats.good_octets_recvd: 2470666426 > dev.em.0.mac_stats.good_octets_txd: 8361514233625 > dev.em.0.mac_stats.total_pkts_txd: 6301004697 > dev.em.0.mac_stats.good_pkts_txd: 6301004695 > dev.em.0.mac_stats.bcast_pkts_txd: 80 > dev.em.0.mac_stats.mcast_pkts_txd: 2425 > dev.em.0.mac_stats.tx_frames_64: 36413025 > dev.em.0.mac_stats.tx_frames_65_127: 648631416 > dev.em.0.mac_stats.tx_frames_128_255: 7701802 > dev.em.0.mac_stats.tx_frames_256_511: 11499983 > dev.em.0.mac_stats.tx_frames_512_1023: 56954995 > dev.em.0.mac_stats.tx_frames_1024_1522: 5539803478 > dev.em.0.mac_stats.tso_txd: 0 > dev.em.0.mac_stats.tso_ctx_fail: 0 > dev.em.0.interrupts.asserts: 3 > dev.em.0.interrupts.rx_pkt_timer: 0 > dev.em.0.interrupts.rx_abs_timer: 0 > dev.em.0.interrupts.tx_pkt_timer: 0 > dev.em.0.interrupts.tx_abs_timer: 0 > dev.em.0.interrupts.tx_queue_empty: 0 > dev.em.0.interrupts.tx_queue_min_thresh: 0 > dev.em.0.interrupts.rx_desc_min_thresh: 0 > dev.em.0.interrupts.rx_overrun: 0 > dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.1.%driver: em > dev.em.1.%location: slot=0 function=0 > dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 > subdevice=0x10d3 class=0x020000 > dev.em.1.%parent: pci2 > dev.em.1.nvm: -1 > dev.em.1.debug: -1 > dev.em.1.rx_int_delay: 0 > dev.em.1.tx_int_delay: 66 > dev.em.1.rx_abs_int_delay: 66 > dev.em.1.tx_abs_int_delay: 66 > dev.em.1.rx_processing_limit: 100 > dev.em.1.flow_control: 3 > dev.em.1.eee_control: 0 > dev.em.1.link_irq: 4172 > dev.em.1.mbuf_alloc_fail: 0 > dev.em.1.cluster_alloc_fail: 0 > dev.em.1.dropped: 0 > dev.em.1.tx_dma_fail: 0 > dev.em.1.rx_overruns: 0 > dev.em.1.watchdog_timeouts: 0 > dev.em.1.device_control: 1074790984 > dev.em.1.rx_control: 67141634 > dev.em.1.fc_high_water: 18432 > dev.em.1.fc_low_water: 16932 > dev.em.1.queue0.txd_head: 159 > dev.em.1.queue0.txd_tail: 159 > dev.em.1.queue0.tx_irq: 1599049292 > dev.em.1.queue0.no_desc_avail: 0 > dev.em.1.queue0.rxd_head: 122 > dev.em.1.queue0.rxd_tail: 121 > dev.em.1.queue0.rx_irq: 2190215040 > dev.em.1.mac_stats.excess_coll: 0 > dev.em.1.mac_stats.single_coll: 0 > dev.em.1.mac_stats.multiple_coll: 0 > dev.em.1.mac_stats.late_coll: 0 > dev.em.1.mac_stats.collision_count: 0 > dev.em.1.mac_stats.symbol_errors: 0 > dev.em.1.mac_stats.sequence_errors: 0 > dev.em.1.mac_stats.defer_count: 0 > dev.em.1.mac_stats.missed_packets: 22415 > dev.em.1.mac_stats.recv_no_buff: 11223 > dev.em.1.mac_stats.recv_undersize: 0 > dev.em.1.mac_stats.recv_fragmented: 0 > dev.em.1.mac_stats.recv_oversize: 0 > dev.em.1.mac_stats.recv_jabber: 0 > dev.em.1.mac_stats.recv_errs: 0 > dev.em.1.mac_stats.crc_errs: 0 > dev.em.1.mac_stats.alignment_errs: 0 > dev.em.1.mac_stats.coll_ext_errs: 0 > dev.em.1.mac_stats.xon_recvd: 0 > dev.em.1.mac_stats.xon_txd: 0 > dev.em.1.mac_stats.xoff_recvd: 0 > dev.em.1.mac_stats.xoff_txd: 0 > dev.em.1.mac_stats.total_pkts_recvd: 6091088292 > dev.em.1.mac_stats.good_pkts_recvd: 6091065877 > dev.em.1.mac_stats.bcast_pkts_recvd: 38569835 > dev.em.1.mac_stats.mcast_pkts_recvd: 1860 > dev.em.1.mac_stats.rx_frames_64: 2329378133 > dev.em.1.mac_stats.rx_frames_65_127: 2592499514 > dev.em.1.mac_stats.rx_frames_128_255: 7898056 > dev.em.1.mac_stats.rx_frames_256_511: 15815777 > dev.em.1.mac_stats.rx_frames_512_1023: 44494959 > dev.em.1.mac_stats.rx_frames_1024_1522: 1100979438 > dev.em.1.mac_stats.good_octets_recvd: 2043621877185 > dev.em.1.mac_stats.good_octets_txd: 8381779145365 > dev.em.1.mac_stats.total_pkts_txd: 6327642657 > dev.em.1.mac_stats.good_pkts_txd: 6327642657 > dev.em.1.mac_stats.bcast_pkts_txd: 2149 > dev.em.1.mac_stats.mcast_pkts_txd: 11 > dev.em.1.mac_stats.tx_frames_64: 36904932 > dev.em.1.mac_stats.tx_frames_65_127: 662019693 > dev.em.1.mac_stats.tx_frames_128_255: 7256854 > dev.em.1.mac_stats.tx_frames_256_511: 11840333 > dev.em.1.mac_stats.tx_frames_512_1023: 57343575 > dev.em.1.mac_stats.tx_frames_1024_1522: 5552277270 > dev.em.1.mac_stats.tso_txd: 0 > dev.em.1.mac_stats.tso_ctx_fail: 0 > dev.em.1.interrupts.asserts: 3934 > dev.em.1.interrupts.rx_pkt_timer: 3 > dev.em.1.interrupts.rx_abs_timer: 0 > dev.em.1.interrupts.tx_pkt_timer: 0 > dev.em.1.interrupts.tx_abs_timer: 0 > dev.em.1.interrupts.tx_queue_empty: 0 > dev.em.1.interrupts.tx_queue_min_thresh: 0 > dev.em.1.interrupts.rx_desc_min_thresh: 0 > dev.em.1.interrupts.rx_overrun: 1 > > Jason > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EA7E4B8.9050003>