Date: Mon, 7 Feb 2011 15:58:12 -0800 From: Pyun YongHyeon <pyunyh@gmail.com> To: Karim Fodil-Lemelin <fodillemlinkarim@gmail.com> Cc: jfv@FreeBSD.org, freebsd-net@freebsd.org Subject: Re: Fwd: igb driver tx hangs when out of mbuf clusters Message-ID: <20110207235811.GA1306@michelle.cdnetworks.com> In-Reply-To: <AANLkTi=AT6tVHbG-jz01275-8pMzVFfHiU9Hz5BzF4yZ@mail.gmail.com> References: <AANLkTim=OYB5cC1H86N_-tDW1w_ipR5-gZjZnT6k%2BMv5@mail.gmail.com> <10510673199.20110207203507@serebryakov.spb.ru> <AANLkTing_L5eLe09xzwgmtF4cp4qO8n8mdRsp4d4ZAxY@mail.gmail.com> <AANLkTikPEK_69if-gZ0nygdLBOTtrUZmPNTnyhHtJr6K@mail.gmail.com> <AANLkTinuLMVif-tNE3s%2B%2BtJccm%2B-_3qvM=Z4BtTF0%2B1q@mail.gmail.com> <AANLkTi=AT6tVHbG-jz01275-8pMzVFfHiU9Hz5BzF4yZ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin wrote: > Subject: Re: igb driver tx hangs when out of mbuf clusters > > > To: Lev Serebryakov <lev@serebryakov.spb.ru> > > Cc: freebsd-net@freebsd.org > > > > > > 2011/2/7 Lev Serebryakov <lev@serebryakov.spb.ru> > > > > Hello, Karim. > >> You wrote 7 февраля 2011 г., 19:58:04: > >> > >> > >> > The issue is with the igb driver from 7.4 RC3 r218406. If the driver > >> runs > >> > out of mbuf clusters it simply stops receiving even after the clusters > >> have > >> > been freed. > >> It looks like my problems with em0 (see thread "em0 hangs without > >> any messages like "Watchdog timeout", only down/up reset it.")... > >> Codebase for em and igb is somewhat common... > >> > >> -- > >> // Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru> > >> > >> I agree. > > > > Do you get missed packets in mac_stats (sysctl dev.em | grep missed)? > > > > I might not have mentioned but I can also 'fix' the problem by doing > > ifconfig igb0 down/up. > > > > I will try using POLLING to 'automatize' the reset as you mentioned in your > > thread. > > > > Karim. > > > > > Follow up on tests with POLLING: The problem is still occurring although it > takes more time ... Outputs of sysctl dev.igb0 and netstat -m will follow: > > 9219/99426/108645 mbufs in use (current/cache/total) > 9217/90783/100000/100000 mbuf clusters in use (current/cache/total/max) Do you see network processes are stuck in keglim state? If you see that I think that's not trivial to solve. You wouldn't even kill that process if it is under keglim state unless some more mbuf clusters are freed from other places. I think both igb(4) and em(4) pass received frame to upper stack before allocating new RX buffer. If driver fails to allocate new RX buffer driver will try to refill RX buffers in next run. Under extreme resource shortage case, this situation can produce no more RX buffers in RX descriptor ring and this will take the box out of network. Other drivers avoid that situation by allocating new RX buffer before passing received frame to upper stack. If RX buffer allocation fails driver will just reuse old RX buffer without passing received frame to upper stack. That does not completely solve the keglim issue though. I think you should have enough mbuf cluters to avoid keglim. However the output above indicates you have enough free mbuf clusters. So I guess igb(4) encountered zero available RX buffer situation in past but failed to refill the RX buffer again. I guess driver may be able to periodically check available RX buffers. Jack may have better idea if this was the case.(CCed) > 0/640 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/12800/12800/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 20738K/257622K/278361K bytes allocated to network (current/cache/total) > 0/291/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/5/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086 > subdevice=0x0000 class=0x020000 > dev.igb.0.%parent: pci7 > dev.igb.0.nvm: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.rx_processing_limit: 100 > dev.igb.0.link_irq: 4 > dev.igb.0.dropped: 0 > dev.igb.0.tx_dma_fail: 0 > dev.igb.0.rx_overruns: 464 > dev.igb.0.watchdog_timeouts: 0 > dev.igb.0.device_control: 1490027073 > dev.igb.0.rx_control: 67141658 > dev.igb.0.interrupt_mask: 0 > dev.igb.0.extended_int_mask: 0 > dev.igb.0.tx_buf_alloc: 14 > dev.igb.0.rx_buf_alloc: 34 > dev.igb.0.fc_high_water: 29488 > dev.igb.0.fc_low_water: 29480 > dev.igb.0.queue0.interrupt_rate: 111111 > dev.igb.0.queue0.txd_head: 877 > dev.igb.0.queue0.txd_tail: 877 > dev.igb.0.queue0.no_desc_avail: 0 > dev.igb.0.queue0.tx_packets: 92013 > dev.igb.0.queue0.rxd_head: 570 > dev.igb.0.queue0.rxd_tail: 570 > dev.igb.0.queue0.rx_packets: 163386 > dev.igb.0.queue0.rx_bytes: 240260310 > dev.igb.0.queue0.lro_queued: 0 > dev.igb.0.queue0.lro_flushed: 0 > dev.igb.0.mac_stats.excess_coll: 0 > dev.igb.0.mac_stats.single_coll: 0 > dev.igb.0.mac_stats.multiple_coll: 0 > dev.igb.0.mac_stats.late_coll: 0 > dev.igb.0.mac_stats.collision_count: 0 > dev.igb.0.mac_stats.symbol_errors: 0 > dev.igb.0.mac_stats.sequence_errors: 0 > dev.igb.0.mac_stats.defer_count: 0 > dev.igb.0.mac_stats.missed_packets: 3104 > dev.igb.0.mac_stats.recv_no_buff: 4016 > dev.igb.0.mac_stats.recv_undersize: 0 > dev.igb.0.mac_stats.recv_fragmented: 0 > dev.igb.0.mac_stats.recv_oversize: 0 > dev.igb.0.mac_stats.recv_jabber: 0 > dev.igb.0.mac_stats.recv_errs: 0 > dev.igb.0.mac_stats.crc_errs: 0 > dev.igb.0.mac_stats.alignment_errs: 0 > dev.igb.0.mac_stats.coll_ext_errs: 0 > dev.igb.0.mac_stats.xon_recvd: 0 > dev.igb.0.mac_stats.xon_txd: 346 > dev.igb.0.mac_stats.xoff_recvd: 0 > dev.igb.0.mac_stats.xoff_txd: 3450 > dev.igb.0.mac_stats.total_pkts_recvd: 166515 > dev.igb.0.mac_stats.good_pkts_recvd: 163411 > dev.igb.0.mac_stats.bcast_pkts_recvd: 0 > dev.igb.0.mac_stats.mcast_pkts_recvd: 51 > dev.igb.0.mac_stats.rx_frames_64: 10 > dev.igb.0.mac_stats.rx_frames_65_127: 1601 > dev.igb.0.mac_stats.rx_frames_128_255: 53 > dev.igb.0.mac_stats.rx_frames_256_511: 42 > dev.igb.0.mac_stats.rx_frames_512_1023: 18 > dev.igb.0.mac_stats.rx_frames_1024_1522: 161687 > dev.igb.0.mac_stats.good_octets_recvd: 240948229 > dev.igb.0.mac_stats.good_octets_txd: 5947150 > dev.igb.0.mac_stats.total_pkts_txd: 95809 > dev.igb.0.mac_stats.good_pkts_txd: 92013 > dev.igb.0.mac_stats.bcast_pkts_txd: 1516 > dev.igb.0.mac_stats.mcast_pkts_txd: 1817 > dev.igb.0.mac_stats.tx_frames_64: 90302 > dev.igb.0.mac_stats.tx_frames_65_127: 1711 > dev.igb.0.mac_stats.tx_frames_128_255: 0 > dev.igb.0.mac_stats.tx_frames_256_511: 0 > dev.igb.0.mac_stats.tx_frames_512_1023: 0 > dev.igb.0.mac_stats.tx_frames_1024_1522: 0 > dev.igb.0.mac_stats.tso_txd: 0 > dev.igb.0.mac_stats.tso_ctx_fail: 0 > dev.igb.0.interrupts.asserts: 5584 > dev.igb.0.interrupts.rx_pkt_timer: 163411 > dev.igb.0.interrupts.rx_abs_timer: 163386 > dev.igb.0.interrupts.tx_pkt_timer: 92013 > dev.igb.0.interrupts.tx_abs_timer: 0 > dev.igb.0.interrupts.tx_queue_empty: 92013 > dev.igb.0.interrupts.tx_queue_min_thresh: 0 > dev.igb.0.interrupts.rx_desc_min_thresh: 19 > dev.igb.0.interrupts.rx_overrun: 0 > dev.igb.0.host.breaker_tx_pkt: 0 > dev.igb.0.host.host_tx_pkt_discard: 0 > dev.igb.0.host.rx_pkt: 0 > dev.igb.0.host.breaker_rx_pkts: 0 > dev.igb.0.host.breaker_rx_pkt_drop: 0 > dev.igb.0.host.tx_good_pkt: 0 > dev.igb.0.host.breaker_tx_pkt_drop: 0 > dev.igb.0.host.rx_good_bytes: 240948229 > dev.igb.0.host.tx_good_bytes: 5947150 > dev.igb.0.host.length_errors: 0 > dev.igb.0.host.serdes_violation_pkt: 0 > dev.igb.0.host.header_redir_missed: 0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110207235811.GA1306>