From owner-freebsd-net@FreeBSD.ORG Thu Mar 20 19:56:45 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C7CE03F4 for ; Thu, 20 Mar 2014 19:56:45 +0000 (UTC) Received: from mail-qa0-x235.google.com (mail-qa0-x235.google.com [IPv6:2607:f8b0:400d:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 84DBFA50 for ; Thu, 20 Mar 2014 19:56:45 +0000 (UTC) Received: by mail-qa0-f53.google.com with SMTP id w8so1412989qac.26 for ; Thu, 20 Mar 2014 12:56:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=XenrVhIzH2jsKUafcLGxRI8MCslkxLkBTwu6okaXZzA=; b=K6LkhDXhRr/LXhf8nH8/BGE/cUry7yVfLMa1SY2mDnn336mg0N2bPUu5eglgmw8rmR OaHz3w7Cj3jia9VJrEdtaxNKEQ+YUugyIUdlO/CEVK4SZn5Z9Q+GDU7Jc+4MEoiu9Li5 wpvxF9Tj2SHu7M2Ei5Pwz1TUVS2H7ELdOsjFzcL8fTox/xF/irXur+lbY6hkyESko5WP z9OB/pUIuQqy0RCMdcFt4YH6ptoOijaIoFBWwKQHG/jeV5pO20t2mId537hETq62q+cF bD2v9gYK5okp5F9QomK89JlpVRtigLRfAJ/zL7MGVDmoQjzoXT0tyiGKQFPP7TxAzDoT +lUA== MIME-Version: 1.0 X-Received: by 10.140.107.10 with SMTP id g10mr50519430qgf.63.1395345404745; Thu, 20 Mar 2014 12:56:44 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Thu, 20 Mar 2014 12:56:44 -0700 (PDT) In-Reply-To: <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca> References: <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca> Date: Thu, 20 Mar 2014 16:56:44 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2014 19:56:45 -0000 BTW, When I have the problem, this is what I see from netstat -m 4080/2956/7036/6127254 mbuf clusters in use (current/cache/total/max) 4080/2636 mbuf+clusters out of packet secondary zone in use (current/cache) 0/50/50/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 32768/155/32923/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 312541K/9182K/321724K bytes allocated to network (current/cache/total) 34481/2600/4091 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 50/27433/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile It doesn't look that bad to me, other than all of the denied counts - But I can't see sysctl buffer numbers that look too low... For those who are interested, here is a dump of hw.ix and dev.ix.0 (I have ix.1 off) hw.ix.enable_aim: 1 hw.ix.max_interrupt_rate: 31250 hw.ix.rx_process_limit: 256 hw.ix.tx_process_limit: 256 hw.ix.enable_msix: 1 hw.ix.num_queues: 8 hw.ix.txd: 2048 hw.ix.rxd: 2048 2014-03-20 16:29:05.291 - INFO - dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15 dev.ix.0.%driver: ix dev.ix.0.%location: slot=0 function=0 dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10f8 subvendor=0x8086 subdevice=0x000c class=0x020000 dev.ix.0.%parent: pci5 dev.ix.0.fc: 3 dev.ix.0.enable_aim: 1 dev.ix.0.advertise_speed: 0 dev.ix.0.dropped: 0 dev.ix.0.mbuf_defrag_failed: 0 dev.ix.0.watchdog_events: 0 dev.ix.0.link_irq: 5 dev.ix.0.queue0.interrupt_rate: 500000 dev.ix.0.queue0.irqs: 452969 dev.ix.0.queue0.txd_head: 319 dev.ix.0.queue0.txd_tail: 319 dev.ix.0.queue0.tso_tx: 61107 dev.ix.0.queue0.no_tx_dma_setup: 0 dev.ix.0.queue0.no_desc_avail: 0 dev.ix.0.queue0.tx_packets: 257636 dev.ix.0.queue0.rxd_head: 531 dev.ix.0.queue0.rxd_tail: 530 dev.ix.0.queue0.rx_packets: 522771 dev.ix.0.queue0.rx_bytes: 1318022421 dev.ix.0.queue0.rx_copies: 224837 dev.ix.0.queue0.lro_queued: 424583 dev.ix.0.queue0.lro_flushed: 181580 dev.ix.0.queue1.interrupt_rate: 125000 dev.ix.0.queue1.irqs: 22756 dev.ix.0.queue1.txd_head: 1169 dev.ix.0.queue1.txd_tail: 1169 dev.ix.0.queue1.tso_tx: 0 dev.ix.0.queue1.no_tx_dma_setup: 0 dev.ix.0.queue1.no_desc_avail: 0 dev.ix.0.queue1.tx_packets: 23202 dev.ix.0.queue1.rxd_head: 337 dev.ix.0.queue1.rxd_tail: 336 dev.ix.0.queue1.rx_packets: 337 dev.ix.0.queue1.rx_bytes: 32988 dev.ix.0.queue1.rx_copies: 225 dev.ix.0.queue1.lro_queued: 335 dev.ix.0.queue1.lro_flushed: 320 dev.ix.0.queue2.interrupt_rate: 500000 dev.ix.0.queue2.irqs: 20256 dev.ix.0.queue2.txd_head: 1201 dev.ix.0.queue2.txd_tail: 1201 dev.ix.0.queue2.tso_tx: 0 dev.ix.0.queue2.no_tx_dma_setup: 0 dev.ix.0.queue2.no_desc_avail: 0 dev.ix.0.queue2.tx_packets: 20962 dev.ix.0.queue2.rxd_head: 1021 dev.ix.0.queue2.rxd_tail: 1020 dev.ix.0.queue2.rx_packets: 1021 dev.ix.0.queue2.rx_bytes: 99126 dev.ix.0.queue2.rx_copies: 891 dev.ix.0.queue2.lro_queued: 396 dev.ix.0.queue2.lro_flushed: 391 dev.ix.0.queue3.interrupt_rate: 71428 dev.ix.0.queue3.irqs: 25072 dev.ix.0.queue3.txd_head: 1465 dev.ix.0.queue3.txd_tail: 1465 dev.ix.0.queue3.tso_tx: 0 dev.ix.0.queue3.no_tx_dma_setup: 0 dev.ix.0.queue3.no_desc_avail: 0 dev.ix.0.queue3.tx_packets: 25726 dev.ix.0.queue3.rxd_head: 310 dev.ix.0.queue3.rxd_tail: 309 dev.ix.0.queue3.rx_packets: 310 dev.ix.0.queue3.rx_bytes: 36886 dev.ix.0.queue3.rx_copies: 150 dev.ix.0.queue3.lro_queued: 309 dev.ix.0.queue3.lro_flushed: 286 dev.ix.0.queue4.interrupt_rate: 500000 dev.ix.0.queue4.irqs: 21251 dev.ix.0.queue4.txd_head: 308 dev.ix.0.queue4.txd_tail: 308 dev.ix.0.queue4.tso_tx: 0 dev.ix.0.queue4.no_tx_dma_setup: 0 dev.ix.0.queue4.no_desc_avail: 0 dev.ix.0.queue4.tx_packets: 22090 dev.ix.0.queue4.rxd_head: 589 dev.ix.0.queue4.rxd_tail: 588 dev.ix.0.queue4.rx_packets: 589 dev.ix.0.queue4.rx_bytes: 57938 dev.ix.0.queue4.rx_copies: 558 dev.ix.0.queue4.lro_queued: 585 dev.ix.0.queue4.lro_flushed: 585 dev.ix.0.queue5.interrupt_rate: 41666 dev.ix.0.queue5.irqs: 20123 dev.ix.0.queue5.txd_head: 314 dev.ix.0.queue5.txd_tail: 314 dev.ix.0.queue5.tso_tx: 0 dev.ix.0.queue5.no_tx_dma_setup: 0 dev.ix.0.queue5.no_desc_avail: 0 dev.ix.0.queue5.tx_packets: 20618 dev.ix.0.queue5.rxd_head: 112 dev.ix.0.queue5.rxd_tail: 111 dev.ix.0.queue5.rx_packets: 112 dev.ix.0.queue5.rx_bytes: 10224 dev.ix.0.queue5.rx_copies: 84 dev.ix.0.queue5.lro_queued: 109 dev.ix.0.queue5.lro_flushed: 109 dev.ix.0.queue6.interrupt_rate: 71428 dev.ix.0.queue6.irqs: 18418 dev.ix.0.queue6.txd_head: 732 dev.ix.0.queue6.txd_tail: 732 dev.ix.0.queue6.tso_tx: 45 dev.ix.0.queue6.no_tx_dma_setup: 0 dev.ix.0.queue6.no_desc_avail: 0 dev.ix.0.queue6.tx_packets: 19137 dev.ix.0.queue6.rxd_head: 824 dev.ix.0.queue6.rxd_tail: 823 dev.ix.0.queue6.rx_packets: 824 dev.ix.0.queue6.rx_bytes: 92838 dev.ix.0.queue6.rx_copies: 583 dev.ix.0.queue6.lro_queued: 818 dev.ix.0.queue6.lro_flushed: 716 dev.ix.0.queue7.interrupt_rate: 62500 dev.ix.0.queue7.irqs: 17681 dev.ix.0.queue7.txd_head: 721 dev.ix.0.queue7.txd_tail: 721 dev.ix.0.queue7.tso_tx: 0 dev.ix.0.queue7.no_tx_dma_setup: 0 dev.ix.0.queue7.no_desc_avail: 0 dev.ix.0.queue7.tx_packets: 18067 dev.ix.0.queue7.rxd_head: 1407 dev.ix.0.queue7.rxd_tail: 1406 dev.ix.0.queue7.rx_packets: 1407 dev.ix.0.queue7.rx_bytes: 252631 dev.ix.0.queue7.rx_copies: 884 dev.ix.0.queue7.lro_queued: 1400 dev.ix.0.queue7.lro_flushed: 1390 dev.ix.0.mac_stats.crc_errs: 0 dev.ix.0.mac_stats.ill_errs: 0 dev.ix.0.mac_stats.byte_errs: 0 dev.ix.0.mac_stats.short_discards: 0 dev.ix.0.mac_stats.local_faults: 2 dev.ix.0.mac_stats.remote_faults: 3 dev.ix.0.mac_stats.rec_len_errs: 0 dev.ix.0.mac_stats.xon_txd: 0 dev.ix.0.mac_stats.xon_recvd: 0 dev.ix.0.mac_stats.xoff_txd: 0 dev.ix.0.mac_stats.xoff_recvd: 0 dev.ix.0.mac_stats.total_octets_rcvd: 1320732697 dev.ix.0.mac_stats.good_octets_rcvd: 1320713370 dev.ix.0.mac_stats.total_pkts_rcvd: 527648 dev.ix.0.mac_stats.good_pkts_rcvd: 527365 dev.ix.0.mac_stats.mcast_pkts_rcvd: 25 dev.ix.0.mac_stats.bcast_pkts_rcvd: 75 dev.ix.0.mac_stats.rx_frames_64: 128032 dev.ix.0.mac_stats.rx_frames_65_127: 100057 dev.ix.0.mac_stats.rx_frames_128_255: 115733 dev.ix.0.mac_stats.rx_frames_256_511: 1210 dev.ix.0.mac_stats.rx_frames_512_1023: 3075 dev.ix.0.mac_stats.rx_frames_1024_1522: 179258 dev.ix.0.mac_stats.recv_undersized: 0 dev.ix.0.mac_stats.recv_fragmented: 0 dev.ix.0.mac_stats.recv_oversized: 0 dev.ix.0.mac_stats.recv_jabberd: 0 dev.ix.0.mac_stats.management_pkts_rcvd: 0 dev.ix.0.mac_stats.management_pkts_drpd: 0 dev.ix.0.mac_stats.checksum_errs: 0 dev.ix.0.mac_stats.good_octets_txd: 2815129453 dev.ix.0.mac_stats.total_pkts_txd: 640355 dev.ix.0.mac_stats.good_pkts_txd: 640355 dev.ix.0.mac_stats.bcast_pkts_txd: 2 dev.ix.0.mac_stats.mcast_pkts_txd: 25 dev.ix.0.mac_stats.management_pkts_txd: 0 dev.ix.0.mac_stats.tx_frames_64: 39831 dev.ix.0.mac_stats.tx_frames_65_127: 166390 dev.ix.0.mac_stats.tx_frames_128_255: 72116 dev.ix.0.mac_stats.tx_frames_256_511: 2072 dev.ix.0.mac_stats.tx_frames_512_1023: 1339 dev.ix.0.mac_stats.tx_frames_1024_1522: 358607 and lastly, the default sysctl kern.ipc: kern.ipc.maxsockbuf: 2097152 kern.ipc.sockbuf_waste_factor: 8 kern.ipc.max_linkhdr: 16 kern.ipc.max_protohdr: 60 kern.ipc.max_hdr: 76 kern.ipc.max_datalen: 92 kern.ipc.maxmbufmem: 50194468864 kern.ipc.nmbclusters: 6127254 kern.ipc.nmbjumbop: 3063627 kern.ipc.nmbjumbo9: 2723223 kern.ipc.nmbjumbo16: 2042416 kern.ipc.nmbufs: 39214440 kern.ipc.maxpipekva: 1610002432 kern.ipc.pipekva: 147456 kern.ipc.pipefragretry: 0 kern.ipc.pipeallocfail: 0 kern.ipc.piperesizefail: 0 kern.ipc.piperesizeallowed: 1 kern.ipc.msgmax: 16384 kern.ipc.msgmni: 40 kern.ipc.msgmnb: 2048 kern.ipc.msgtql: 40 kern.ipc.msgssz: 8 kern.ipc.msgseg: 2048 kern.ipc.semmni: 50 kern.ipc.semmns: 340 kern.ipc.semmnu: 150 kern.ipc.semmsl: 340 kern.ipc.semopm: 100 kern.ipc.semume: 50 kern.ipc.semusz: 632 kern.ipc.semvmx: 32767 kern.ipc.semaem: 16384 kern.ipc.shmmax: 536870912 kern.ipc.shmmin: 1 kern.ipc.shmmni: 192 kern.ipc.shmseg: 128 kern.ipc.shmall: 131072 kern.ipc.shm_use_phys: 0 kern.ipc.shm_allow_removed: 0 kern.ipc.soacceptqueue: 128 kern.ipc.numopensockets: 79 kern.ipc.maxsockets: 3144540 kern.ipc.sendfile.readahead: 1