From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 08:37:21 2011 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53CBE106564A for <freebsd-net@freebsd.org>; Thu, 27 Oct 2011 08:37:21 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 05B0E8FC15 for <freebsd-net@freebsd.org>; Thu, 27 Oct 2011 08:37:20 +0000 (UTC) Received: by ggnq2 with SMTP id q2so3206848ggn.13 for <freebsd-net@freebsd.org>; Thu, 27 Oct 2011 01:37:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=Q87UQGNphDJx1rZbZbDlOk/NZVxQRKAs0dctJQ3A0S4=; b=d1gx3D+1AErsytdtQbE7/drWaPNMVcxQkD2UMqyTnJf+vrf4rr5+UDDv8IGhyemJ7O jjY5EDDYm+Zh/gT7IUVGT348OC40o2UxMJU6ER4fzwDmVeSnXOWKNYDsb8YjKftzLeI7 rvvcT7ZGY2Ovde/V+rwuHPP2Ft8qeee9/E0Os= Received: by 10.151.149.4 with SMTP id b4mr31417254ybo.33.1319704640074; Thu, 27 Oct 2011 01:37:20 -0700 (PDT) Received: from [127.0.0.1] ([84.241.57.181]) by mx.google.com with ESMTPS id l27sm12875024ani.21.2011.10.27.01.37.17 (version=SSLv3 cipher=OTHER); Thu, 27 Oct 2011 01:37:19 -0700 (PDT) Message-ID: <4EA91836.2040508@gmail.com> Date: Thu, 27 Oct 2011 12:07:10 +0330 From: Hooman Fazaeli <hoomanfazaeli@gmail.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: Emil Muratov <gpm@hotplug.ru> References: <CAAAm0r0RXEJo4UiKS=Ui0e5OQTg6sg-xcYf3mYB5+vk8i8557w@mail.gmail.com> <4E8F157A.40702@sentex.net> <CAAAm0r2JH43Rct7UxQK2duH1p43Nepnj5mpb6bXo==DPayhJLg@mail.gmail.com> <4E8F51D4.1060509@sentex.net> <CACqU3MVwLaepFymZJkaVk6p=SpykGhqs=VYFjLh9fP9S=AxDhg@mail.gmail.com> <CAAAm0r1DKvoL9=Ket9up=4+5xiCzTTZJK99FhF9jcCA28B0M+A@mail.gmail.com> <CAAAm0r3XdsMHZh+P_NF-txZasdExzwZ8ymmGQgGhJQds0fOiBQ@mail.gmail.com> <CAAAm0r1iS3z-7CBJ=xYDf+JOA1Q2nU0O54Twbyb7FjvgWHjKVw@mail.gmail.com> <4EA7E203.3020306@sepehrs.com> <CAAAm0r3Nr2t8cCetPkFnLQ-3KwqHw_0SpqbtvYPRUkSP=9n8CA@mail.gmail.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> In-Reply-To: <4EA8FA40.7010504@hotplug.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Jason Wolfe <nitroboost@gmail.com> Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 27 Oct 2011 08:37:21 -0000 On 10/27/2011 9:59 AM, Emil Muratov wrote: > > Hi Hooman > > Here is what I've got when the script triggered just in time when the interface was locked > > > 11.10.26-23:39:10 ... interface em0 is down... > > FreeBSD ion.hotplug.ru 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Oct 20 20:20:25 MSD 2011 root@epia.home > .lan:/usr/obj/usr/src/sys/ION6debug amd64 > 11:39PM up 1:12, 2 users, load averages: 0.26, 0.48, 0.58 > > > == vmstat -i == > interrupt total rate > irq22: nfe0 16644480 3865 > cpu0: timer 8610122 1999 > irq256: ahci0 606705 140 > irq257: em0:rx 0 3896622 904 > irq258: em0:tx 0 2762957 641 > irq259: em0:link 620 0 > cpu3: timer 8609499 1999 > cpu1: timer 8609499 1999 > cpu2: timer 8609499 1999 > Total 58350003 13550 > > == netstat -ind == > Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop > usbus 0 <Link#1> 0 0 0 0 0 0 0 > usbus 0 <Link#2> 0 0 0 0 0 0 0 > nfe0 1500 <Link#3> 00:25:22:21:86:89 7157140 0 0 12266747 0 0 0 > nfe0 1500 fe80::225:22f fe80::225:22ff:fe 0 - - 85 - - - > nfe0 1500 10.16.128.0/1 10.16.189.71 0 - - 48135 - - - > em0 9000 <Link#4> 00:1b:21:ab:bf:4a 5465087 623 0 2862028 0 0 113 > em0 9000 192.168.168.0 192.168.168.1 764085 - - 1005078 - - - > em0 9000 fe80::21b:21f fe80::21b:21ff:fe 45 - - 252 - - - > em0 9000 2002:d58d:871 2002:d58d:8715:1: 73 - - 38 - - - > wifi 1500 <Link#7> 00:1b:21:ab:bf:4a 347 0 0 350 0 0 0 > wifi 1500 192.168.168.6 192.168.168.65 0 - - 0 - - - > wifi 1500 fe80::225:x fe80::225:x:x 0 - - 349 - - - > wifi 1500 2002:x:x 2002:x:x:2: 0 - - 0 - - - > wifio 1500 <Link#8> 00:1b:21:ab:bf:4a 59559 0 0 114639 0 0 0 > wifio 1500 192.168.168.8 192.168.168.81 0 - - 160 - - - > wifio 1500 fe80::225:x fe80::225:x:x 0 - - 0 - - - > stf0 1280 <Link#9> 5725 0 0 6125 420 0 0 > stf0 1280 2002:x:x 2002:x:x::1 1878 - - 1121 - - - > ng0* 1500 <Link#10> 0 0 0 0 0 0 0 > ng1* 1500 <Link#11> 0 0 0 0 0 0 0 > ng2 1492 <Link#12> 7143733 0 0 12234436 0 0 0 > ng2 1492 213.141.x.x 213.141.x.x 4735932 - - 8480089 - - - > ng2 1492 fe80::x:x fe80::x:x:x 0 - - 1 - - - > tun0 1455 <Link#13> 350 0 0 172 0 0 0 > tun0 1455 fe80::225:x fe80::225:x:x 0 - - 2 - - - > tun0 1455 192.168.169.1 192.168.169.1 117 - - 167 - - - > > Oct 26 23:39:11 ion kernel: em0: hw tdh = 975, hw tdt = 944 > Oct 26 23:39:11 ion kernel: em0: hw rdh = 960, hw rdt = 959 > Oct 26 23:39:11 ion kernel: em0: Tx Queue Status = 1 > Oct 26 23:39:11 ion kernel: em0: TX descriptors avail = 31 > Oct 26 23:39:11 ion kernel: em0: Tx Descriptors avail failure = 0 > Oct 26 23:39:11 ion kernel: em0: RX discarded packets = 0 > Oct 26 23:39:11 ion kernel: em0: RX Next to Check = 960 > Oct 26 23:39:11 ion kernel: em0: RX Next to Refresh = 959 > > net.inet.ip.intr_queue_maxlen: 4096 > net.inet.ip.intr_queue_drops: 0 > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=0 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000 > dev.em.0.%parent: pci2 > dev.em.0.nvm: -1 > dev.em.0.debug: -1 > dev.em.0.rx_int_delay: 200 > dev.em.0.tx_int_delay: 200 > dev.em.0.rx_abs_int_delay: 4096 > dev.em.0.tx_abs_int_delay: 4096 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 3 > dev.em.0.eee_control: 0 > dev.em.0.link_irq: 648 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 0 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1477444168 > dev.em.0.rx_control: 100827170 > dev.em.0.fc_high_water: 11264 > dev.em.0.fc_low_water: 9764 > dev.em.0.queue0.txd_head: 975 > dev.em.0.queue0.txd_tail: 944 > dev.em.0.queue0.tx_irq: 2762762 > dev.em.0.queue0.no_desc_avail: 0 > dev.em.0.queue0.rxd_head: 960 > dev.em.0.queue0.rxd_tail: 959 > dev.em.0.queue0.rx_irq: 3895860 > dev.em.0.mac_stats.excess_coll: 0 > dev.em.0.mac_stats.single_coll: 0 > dev.em.0.mac_stats.multiple_coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_count: 0 > dev.em.0.mac_stats.symbol_errors: 0 > dev.em.0.mac_stats.sequence_errors: 0 > dev.em.0.mac_stats.defer_count: 0 > dev.em.0.mac_stats.missed_packets: 647 > dev.em.0.mac_stats.recv_no_buff: 0 > dev.em.0.mac_stats.recv_undersize: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_oversize: 0 > dev.em.0.mac_stats.recv_jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_errs: 0 > dev.em.0.mac_stats.coll_ext_errs: 0 > dev.em.0.mac_stats.xon_recvd: 438789 > dev.em.0.mac_stats.xon_txd: 366 > dev.em.0.mac_stats.xoff_recvd: 438789 > dev.em.0.mac_stats.xoff_txd: 1013 > dev.em.0.mac_stats.total_pkts_recvd: 5465524 > dev.em.0.mac_stats.good_pkts_recvd: 4587299 > dev.em.0.mac_stats.bcast_pkts_recvd: 1102 > dev.em.0.mac_stats.mcast_pkts_recvd: 162 > dev.em.0.mac_stats.rx_frames_64: 325765 > dev.em.0.mac_stats.rx_frames_65_127: 1029229 > dev.em.0.mac_stats.rx_frames_128_255: 118432 > dev.em.0.mac_stats.rx_frames_256_511: 11360 > dev.em.0.mac_stats.rx_frames_512_1023: 100708 > dev.em.0.mac_stats.rx_frames_1024_1522: 3001805 > dev.em.0.mac_stats.good_octets_recvd: 4648591681 > dev.em.0.mac_stats.good_octets_txd: 2203060494 > dev.em.0.mac_stats.total_pkts_txd: 3780652 > dev.em.0.mac_stats.good_pkts_txd: 3779273 > dev.em.0.mac_stats.bcast_pkts_txd: 89 > dev.em.0.mac_stats.mcast_pkts_txd: 534 > dev.em.0.mac_stats.tx_frames_64: 1323163 > dev.em.0.mac_stats.tx_frames_65_127: 850801 > dev.em.0.mac_stats.tx_frames_128_255: 193136 > dev.em.0.mac_stats.tx_frames_256_511: 64088 > dev.em.0.mac_stats.tx_frames_512_1023: 47149 > dev.em.0.mac_stats.tx_frames_1024_1522: 1300936 > dev.em.0.mac_stats.tso_txd: 429804 > dev.em.0.mac_stats.tso_ctx_fail: 0 > dev.em.0.interrupts.asserts: 44 > dev.em.0.interrupts.rx_pkt_timer: 0 > dev.em.0.interrupts.rx_abs_timer: 0 > dev.em.0.interrupts.tx_pkt_timer: 0 > dev.em.0.interrupts.tx_abs_timer: 0 > dev.em.0.interrupts.tx_queue_empty: 0 > dev.em.0.interrupts.tx_queue_min_thresh: 0 > dev.em.0.interrupts.rx_desc_min_thresh: 0 > dev.em.0.interrupts.rx_overrun: 0 > > ifconfig em0 > em0: flags=8c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 9000 > description: LAN > options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC> > ether 00:1b:21:ab:bf:4a > inet 192.168.168.1 netmask 0xffffffc0 broadcast 192.168.168.63 > inet6 fe80::21b:21ff:feab:bf4a%em0 prefixlen 64 scopeid 0x4 > inet6 2002:x:x:1::1 prefixlen 64 > nd6 options=1<PERFORMNUD> > media: Ethernet autoselect (1000baseT <full-duplex>) > status: active > What device is at the other end of link? The input errors netstat reports correspond to the missed_packets reported by the driver. Also the driver has been OACTIVE when the problem occurred. There are also a strange number of xon/xoff frames transmitted and received by the driver. You may try these settings and see if they help: - hw.em.fc_setting=0 (in /boot/loader.conf) - hw.em.rxd="4096" (in /boot/loader.conf) - hw.em.txd="4096" (in /boot/loader.conf) - Fix speed and duplex at both link sides. After doing that, confirm on the freebsd box (with ifconfig) and the other device (with whatever command it provides) that the same speed and duplex is used by both devices. you also have high values for dev.em.0.rx/tx_[abs]_int_delay. If you have set them manually, remove them or replace them with these in loader.conf: hw.em.rx_int_delay=0 hw.em.tx_int_delay=66 hw.em.tx_abs_int_delay=66 hw.em.rx_abs_int_delay=66 these may be set via corresponding sysctls too.