From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 23:58:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CE52106564A; Mon, 7 Feb 2011 23:58:10 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id ABD8F8FC08; Mon, 7 Feb 2011 23:58:09 +0000 (UTC) Received: by ywl2 with SMTP id 2so391744ywl.13 for ; Mon, 07 Feb 2011 15:58:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=Z0esuCNU3IkUjJVQ9b1JoP5We2a/PEiUqWhE2TaMonE=; b=u8vwy/NIiHrX5kr63xUgrYBvgjzoAXsKLo8xD3H7+p/HPuPDq8OcrvTxZ9cH151l8m maqAbsnaABBGkW8tsIECP5cjYBJ+L8I+VyY4GYL6grVLOeRGB7z3Zlf7WPEnplnDZW4m 7tXUqWa19xIA+ed49W0ZUL0xt8ayP8Fh62qhg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent; b=SUeMsOhx9Gj4lXy7om/kPxEA7UfT7xhlWxBv6+uD/Vtv/Zuv9BZ2i9xValfp5ej0mF DVZtjT06QPjyiULc/D17+Kp0ZR7Sp0cBXIDQk8mlxeil9FnkOicOG6m+85i1TOyG7YYD ZJVfiHAnsc30biMfGHU+C1eAo5W+0BH5qYTeg= Received: by 10.100.165.20 with SMTP id n20mr9987419ane.114.1297123088626; Mon, 07 Feb 2011 15:58:08 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 17sm6255207anx.33.2011.02.07.15.58.06 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 07 Feb 2011 15:58:07 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 7 Feb 2011 15:58:12 -0800 From: Pyun YongHyeon Date: Mon, 7 Feb 2011 15:58:12 -0800 To: Karim Fodil-Lemelin Message-ID: <20110207235811.GA1306@michelle.cdnetworks.com> References: <10510673199.20110207203507@serebryakov.spb.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: jfv@FreeBSD.org, freebsd-net@freebsd.org Subject: Re: Fwd: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 23:58:10 -0000 On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin wrote: > Subject: Re: igb driver tx hangs when out of mbuf clusters > > > To: Lev Serebryakov > > Cc: freebsd-net@freebsd.org > > > > > > 2011/2/7 Lev Serebryakov > > > > Hello, Karim. > >> You wrote 7 февраля 2011 г., 19:58:04: > >> > >> > >> > The issue is with the igb driver from 7.4 RC3 r218406. If the driver > >> runs > >> > out of mbuf clusters it simply stops receiving even after the clusters > >> have > >> > been freed. > >> It looks like my problems with em0 (see thread "em0 hangs without > >> any messages like "Watchdog timeout", only down/up reset it.")... > >> Codebase for em and igb is somewhat common... > >> > >> -- > >> // Black Lion AKA Lev Serebryakov > >> > >> I agree. > > > > Do you get missed packets in mac_stats (sysctl dev.em | grep missed)? > > > > I might not have mentioned but I can also 'fix' the problem by doing > > ifconfig igb0 down/up. > > > > I will try using POLLING to 'automatize' the reset as you mentioned in your > > thread. > > > > Karim. > > > > > Follow up on tests with POLLING: The problem is still occurring although it > takes more time ... Outputs of sysctl dev.igb0 and netstat -m will follow: > > 9219/99426/108645 mbufs in use (current/cache/total) > 9217/90783/100000/100000 mbuf clusters in use (current/cache/total/max) Do you see network processes are stuck in keglim state? If you see that I think that's not trivial to solve. You wouldn't even kill that process if it is under keglim state unless some more mbuf clusters are freed from other places. I think both igb(4) and em(4) pass received frame to upper stack before allocating new RX buffer. If driver fails to allocate new RX buffer driver will try to refill RX buffers in next run. Under extreme resource shortage case, this situation can produce no more RX buffers in RX descriptor ring and this will take the box out of network. Other drivers avoid that situation by allocating new RX buffer before passing received frame to upper stack. If RX buffer allocation fails driver will just reuse old RX buffer without passing received frame to upper stack. That does not completely solve the keglim issue though. I think you should have enough mbuf cluters to avoid keglim. However the output above indicates you have enough free mbuf clusters. So I guess igb(4) encountered zero available RX buffer situation in past but failed to refill the RX buffer again. I guess driver may be able to periodically check available RX buffers. Jack may have better idea if this was the case.(CCed) > 0/640 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/12800/12800/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 20738K/257622K/278361K bytes allocated to network (current/cache/total) > 0/291/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/5/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086 > subdevice=0x0000 class=0x020000 > dev.igb.0.%parent: pci7 > dev.igb.0.nvm: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.rx_processing_limit: 100 > dev.igb.0.link_irq: 4 > dev.igb.0.dropped: 0 > dev.igb.0.tx_dma_fail: 0 > dev.igb.0.rx_overruns: 464 > dev.igb.0.watchdog_timeouts: 0 > dev.igb.0.device_control: 1490027073 > dev.igb.0.rx_control: 67141658 > dev.igb.0.interrupt_mask: 0 > dev.igb.0.extended_int_mask: 0 > dev.igb.0.tx_buf_alloc: 14 > dev.igb.0.rx_buf_alloc: 34 > dev.igb.0.fc_high_water: 29488 > dev.igb.0.fc_low_water: 29480 > dev.igb.0.queue0.interrupt_rate: 111111 > dev.igb.0.queue0.txd_head: 877 > dev.igb.0.queue0.txd_tail: 877 > dev.igb.0.queue0.no_desc_avail: 0 > dev.igb.0.queue0.tx_packets: 92013 > dev.igb.0.queue0.rxd_head: 570 > dev.igb.0.queue0.rxd_tail: 570 > dev.igb.0.queue0.rx_packets: 163386 > dev.igb.0.queue0.rx_bytes: 240260310 > dev.igb.0.queue0.lro_queued: 0 > dev.igb.0.queue0.lro_flushed: 0 > dev.igb.0.mac_stats.excess_coll: 0 > dev.igb.0.mac_stats.single_coll: 0 > dev.igb.0.mac_stats.multiple_coll: 0 > dev.igb.0.mac_stats.late_coll: 0 > dev.igb.0.mac_stats.collision_count: 0 > dev.igb.0.mac_stats.symbol_errors: 0 > dev.igb.0.mac_stats.sequence_errors: 0 > dev.igb.0.mac_stats.defer_count: 0 > dev.igb.0.mac_stats.missed_packets: 3104 > dev.igb.0.mac_stats.recv_no_buff: 4016 > dev.igb.0.mac_stats.recv_undersize: 0 > dev.igb.0.mac_stats.recv_fragmented: 0 > dev.igb.0.mac_stats.recv_oversize: 0 > dev.igb.0.mac_stats.recv_jabber: 0 > dev.igb.0.mac_stats.recv_errs: 0 > dev.igb.0.mac_stats.crc_errs: 0 > dev.igb.0.mac_stats.alignment_errs: 0 > dev.igb.0.mac_stats.coll_ext_errs: 0 > dev.igb.0.mac_stats.xon_recvd: 0 > dev.igb.0.mac_stats.xon_txd: 346 > dev.igb.0.mac_stats.xoff_recvd: 0 > dev.igb.0.mac_stats.xoff_txd: 3450 > dev.igb.0.mac_stats.total_pkts_recvd: 166515 > dev.igb.0.mac_stats.good_pkts_recvd: 163411 > dev.igb.0.mac_stats.bcast_pkts_recvd: 0 > dev.igb.0.mac_stats.mcast_pkts_recvd: 51 > dev.igb.0.mac_stats.rx_frames_64: 10 > dev.igb.0.mac_stats.rx_frames_65_127: 1601 > dev.igb.0.mac_stats.rx_frames_128_255: 53 > dev.igb.0.mac_stats.rx_frames_256_511: 42 > dev.igb.0.mac_stats.rx_frames_512_1023: 18 > dev.igb.0.mac_stats.rx_frames_1024_1522: 161687 > dev.igb.0.mac_stats.good_octets_recvd: 240948229 > dev.igb.0.mac_stats.good_octets_txd: 5947150 > dev.igb.0.mac_stats.total_pkts_txd: 95809 > dev.igb.0.mac_stats.good_pkts_txd: 92013 > dev.igb.0.mac_stats.bcast_pkts_txd: 1516 > dev.igb.0.mac_stats.mcast_pkts_txd: 1817 > dev.igb.0.mac_stats.tx_frames_64: 90302 > dev.igb.0.mac_stats.tx_frames_65_127: 1711 > dev.igb.0.mac_stats.tx_frames_128_255: 0 > dev.igb.0.mac_stats.tx_frames_256_511: 0 > dev.igb.0.mac_stats.tx_frames_512_1023: 0 > dev.igb.0.mac_stats.tx_frames_1024_1522: 0 > dev.igb.0.mac_stats.tso_txd: 0 > dev.igb.0.mac_stats.tso_ctx_fail: 0 > dev.igb.0.interrupts.asserts: 5584 > dev.igb.0.interrupts.rx_pkt_timer: 163411 > dev.igb.0.interrupts.rx_abs_timer: 163386 > dev.igb.0.interrupts.tx_pkt_timer: 92013 > dev.igb.0.interrupts.tx_abs_timer: 0 > dev.igb.0.interrupts.tx_queue_empty: 92013 > dev.igb.0.interrupts.tx_queue_min_thresh: 0 > dev.igb.0.interrupts.rx_desc_min_thresh: 19 > dev.igb.0.interrupts.rx_overrun: 0 > dev.igb.0.host.breaker_tx_pkt: 0 > dev.igb.0.host.host_tx_pkt_discard: 0 > dev.igb.0.host.rx_pkt: 0 > dev.igb.0.host.breaker_rx_pkts: 0 > dev.igb.0.host.breaker_rx_pkt_drop: 0 > dev.igb.0.host.tx_good_pkt: 0 > dev.igb.0.host.breaker_tx_pkt_drop: 0 > dev.igb.0.host.rx_good_bytes: 240948229 > dev.igb.0.host.tx_good_bytes: 5947150 > dev.igb.0.host.length_errors: 0 > dev.igb.0.host.serdes_violation_pkt: 0 > dev.igb.0.host.header_redir_missed: 0