From owner-freebsd-net@FreeBSD.ORG Mon Oct 31 09:04:38 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D4B91065670 for ; Mon, 31 Oct 2011 09:04:38 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 920778FC17 for ; Mon, 31 Oct 2011 09:04:37 +0000 (UTC) Received: by wyh11 with SMTP id 11so1673351wyh.13 for ; Mon, 31 Oct 2011 02:04:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=NdRBb+dykEjzsMXYXZOJiWijPBi2auWqK5sAAULZieQ=; b=xyP85D2qlRnXcBk6zLtLRZ+m9sfJj638Cug0rwkHBkbQ5LJ7TgFOqHXRFYfashBvKa LATZk9WG8LnO+k4ZwzHW6+P9/ZCLDgym9lJaA6H6pHwJi91f5+J4xmdPZYJ2cKlQ1Vl/ ybRO95RXW6Xc8lb1yDhxekOsV+hYHyzGdTIBw= Received: by 10.227.208.77 with SMTP id gb13mr18619772wbb.4.1320051876158; Mon, 31 Oct 2011 02:04:36 -0700 (PDT) Received: from [127.0.0.1] ([84.241.57.181]) by mx.google.com with ESMTPS id fw16sm31189018wbb.13.2011.10.31.02.04.30 (version=SSLv3 cipher=OTHER); Mon, 31 Oct 2011 02:04:35 -0700 (PDT) Message-ID: <4EAE649C.6040106@gmail.com> Date: Mon, 31 Oct 2011 12:34:28 +0330 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: Emil Muratov References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> <4EA91836.2040508@gmail.com> <4EAE58BC.2090000@hotplug.ru> In-Reply-To: <4EAE58BC.2090000@hotplug.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Jack Vogel , Jason Wolfe Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Oct 2011 09:04:38 -0000 On 10/31/2011 11:43 AM, Emil Muratov wrote: > >> >> You may try these settings and see if they help: >> >> - hw.em.fc_setting=0 (in /boot/loader.conf) >> - hw.em.rxd="4096" (in /boot/loader.conf) >> - hw.em.txd="4096" (in /boot/loader.conf) >> - Fix speed and duplex at both link sides. After doing that, confirm on the freebsd >> box (with ifconfig) and the other device (with whatever command it provides) that >> the same speed and duplex is used by both devices. >> >> you also have high values for dev.em.0.rx/tx_[abs]_int_delay. If you >> have set them manually, remove them or replace them with these in loader.conf: >> >> hw.em.rx_int_delay=0 >> hw.em.tx_int_delay=66 >> hw.em.tx_abs_int_delay=66 >> hw.em.rx_abs_int_delay=66 >> >> these may be set via corresponding sysctls too. >> > > Still no luck with the above settings, I've got another lockups a couple of times. Here is the recent details > > > ================================= > 11.10.30-23:43:06 ... interface em0 is down... > we have Ierrs and no ingoing packets for 5 secs, interface em0 must be toggled > > 11:43PM up 1 day, 3:01, 2 users, load averages: 0.76, 0.64, 0.70 > > == vmstat -i == > interrupt total rate > irq18: ehci0 1145540 11 > irq22: nfe0 473895599 4872 > cpu0: timer 195004026 2005 > irq256: ahci0 12832958 131 > irq257: em0:rx 0 95571051 982 > irq258: em0:tx 0 88777545 912 > irq259: em0:link 946 0 > cpu3: timer 195003397 2005 > cpu1: timer 195003398 2005 > cpu2: timer 195003399 2005 > Total 1452237859 14932 > > == netstat -m == > 5424/1701/7125 mbufs in use (current/cache/total) > 719/1185/1904/51200 mbuf clusters in use (current/cache/total/max) > 719/582 mbuf+clusters out of packet secondary zone in use (current/cache) > 329/583/912/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > 4095/342/4437/12800 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 40978K/8205K/49183K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/6663503/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > == netstat -ind == > Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop > usbus 0 0 0 0 0 0 0 0 > usbus 0 0 0 0 0 0 0 0 > nfe0 1500 00:25:22:21:86:89 196018201 0 0 350650768 0 0 664 > nfe0 1500 fe80::225:22f fe80::225:22ff:fe 0 - - 0 - - - > nfe0 1500 10.16.128.0/1 10.16.189.71 6 - - 29787707 - - - > em0 9000 00:1b:21:ab:bf:4a 175676617 949 0 101627139 0 0 0 > em0 9000 192.168.168.0 192.168.168.1 7628423 - - 13654747 - - - > em0 9000 fe80::21b:21f fe80::21b:21ff:fe 45 - - 5747 - - - > em0 9000 2002:d5xx:xxx 2002:d5xx:xxxx:x: 153 - - 159 - - - > > Oct 30 23:43:06 ion kernel: Interface is RUNNING and INACTIVE > Oct 30 23:43:07 ion kernel: em0: hw tdh = 2656, hw tdt = 3271 > Oct 30 23:43:07 ion kernel: em0: hw rdh = 2112, hw rdt = 2111 > Oct 30 23:43:07 ion kernel: em0: Tx Queue Status = 1 > Oct 30 23:43:07 ion kernel: em0: TX descriptors avail = 3481 > Oct 30 23:43:07 ion kernel: em0: Tx Descriptors avail failure = 0 > Oct 30 23:43:07 ion kernel: em0: RX discarded packets = 0 > Oct 30 23:43:07 ion kernel: em0: RX Next to Check = 2112 > Oct 30 23:43:07 ion kernel: em0: RX Next to Refresh = 2111 > net.inet.ip.intr_queue_maxlen: 4096 > net.inet.ip.intr_queue_drops: 0 > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=0 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000 > dev.em.0.%parent: pci2 > dev.em.0.nvm: -1 > dev.em.0.debug: -1 > dev.em.0.rx_int_delay: 0 > dev.em.0.tx_int_delay: 66 > dev.em.0.rx_abs_int_delay: 66 > dev.em.0.tx_abs_int_delay: 66 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 0 > dev.em.0.eee_control: 0 > dev.em.0.link_irq: 956 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 1 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1074790984 > dev.em.0.rx_control: 100827170 > dev.em.0.fc_high_water: 11264 > dev.em.0.fc_low_water: 9764 > dev.em.0.queue0.txd_head: 2656 > dev.em.0.queue0.txd_tail: 3274 > dev.em.0.queue0.tx_irq: 88769608 > dev.em.0.queue0.no_desc_avail: 0 > dev.em.0.queue0.rxd_head: 2112 > dev.em.0.queue0.rxd_tail: 2111 > dev.em.0.queue0.rx_irq: 95554873 > dev.em.0.mac_stats.excess_coll: 0 > dev.em.0.mac_stats.single_coll: 0 > dev.em.0.mac_stats.multiple_coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_count: 0 > dev.em.0.mac_stats.symbol_errors: 0 > dev.em.0.mac_stats.sequence_errors: 0 > dev.em.0.mac_stats.defer_count: 0 > dev.em.0.mac_stats.missed_packets: 959 > dev.em.0.mac_stats.recv_no_buff: 0 > dev.em.0.mac_stats.recv_undersize: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_oversize: 0 > dev.em.0.mac_stats.recv_jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_errs: 0 > dev.em.0.mac_stats.coll_ext_errs: 0 > dev.em.0.mac_stats.xon_recvd: 0 > dev.em.0.mac_stats.xon_txd: 0 > dev.em.0.mac_stats.xoff_recvd: 0 > dev.em.0.mac_stats.xoff_txd: 0 > dev.em.0.mac_stats.total_pkts_recvd: 175670049 > dev.em.0.mac_stats.good_pkts_recvd: 175669090 > dev.em.0.mac_stats.bcast_pkts_recvd: 2928 > dev.em.0.mac_stats.mcast_pkts_recvd: 1082 > dev.em.0.mac_stats.rx_frames_64: 4539426 > dev.em.0.mac_stats.rx_frames_65_127: 10242159 > dev.em.0.mac_stats.rx_frames_128_255: 576238 > dev.em.0.mac_stats.rx_frames_256_511: 372297 > dev.em.0.mac_stats.rx_frames_512_1023: 2208708 > dev.em.0.mac_stats.rx_frames_1024_1522: 157730262 > dev.em.0.mac_stats.good_octets_recvd: 238712986216 > dev.em.0.mac_stats.good_octets_txd: 40512955499 > dev.em.0.mac_stats.total_pkts_txd: 108590895 > dev.em.0.mac_stats.good_pkts_txd: 108590895 > dev.em.0.mac_stats.bcast_pkts_txd: 4604 > dev.em.0.mac_stats.mcast_pkts_txd: 13969 > dev.em.0.mac_stats.tx_frames_64: 13905445 > dev.em.0.mac_stats.tx_frames_65_127: 69702633 > dev.em.0.mac_stats.tx_frames_128_255: 1442225 > dev.em.0.mac_stats.tx_frames_256_511: 755348 > dev.em.0.mac_stats.tx_frames_512_1023: 375982 > dev.em.0.mac_stats.tx_frames_1024_1522: 22409262 > dev.em.0.mac_stats.tso_txd: 4695644 > dev.em.0.mac_stats.tso_ctx_fail: 0 > dev.em.0.interrupts.asserts: 36 > dev.em.0.interrupts.rx_pkt_timer: 0 > dev.em.0.interrupts.rx_abs_timer: 0 > dev.em.0.interrupts.tx_pkt_timer: 0 > dev.em.0.interrupts.tx_abs_timer: 0 > dev.em.0.interrupts.tx_queue_empty: 0 > dev.em.0.interrupts.tx_queue_min_thresh: 0 > dev.em.0.interrupts.rx_desc_min_thresh: 0 > dev.em.0.interrupts.rx_overrun: 0 > > == ifconfig em0 == > > em0: flags=8843 metric 0 mtu 9000 > description: LAN > options=219b > ether 00:1b:21:ab:bf:4a > inet 192.168.168.1 netmask 0xffffffc0 broadcast 192.168.168.63 > inet6 fe80::21b:21ff:feab:bf4a%em0 prefixlen 64 scopeid 0x4 > inet6 2002:d5xx:xxxx:1::1 prefixlen 64 > nd6 options=1 > media: Ethernet 1000baseT (1000baseT ) > status: active > > > > > The patch I prepared for Jason contains a possible fix for your problem (Cc'ed). Can you give it a try? Before that, and to rule out other possibilities, you may disable MSIX and see if it helps: hw.em.enable_msix="0" (in /boot/loader.conf)