From owner-freebsd-net@FreeBSD.ORG Thu Mar 1 19:56:34 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 220C51065673 for ; Thu, 1 Mar 2012 19:56:34 +0000 (UTC) (envelope-from nitroboost@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id CEB9D8FC08 for ; Thu, 1 Mar 2012 19:56:33 +0000 (UTC) Received: by yenl9 with SMTP id l9so297911yen.13 for ; Thu, 01 Mar 2012 11:56:33 -0800 (PST) Received-SPF: pass (google.com: domain of nitroboost@gmail.com designates 10.60.14.101 as permitted sender) client-ip=10.60.14.101; Authentication-Results: mr.google.com; spf=pass (google.com: domain of nitroboost@gmail.com designates 10.60.14.101 as permitted sender) smtp.mail=nitroboost@gmail.com; dkim=pass header.i=nitroboost@gmail.com Received: from mr.google.com ([10.60.14.101]) by 10.60.14.101 with SMTP id o5mr2977408oec.20.1330631793244 (num_hops = 1); Thu, 01 Mar 2012 11:56:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=wuOq7h76tS5MxJs7uuz54/0QkWi87mtqmG7pQbcj8MM=; b=osJe9+6jx9ArOSsyAyDDNpTLwqMpYt1Vnqy8GIiHQXzrholl20Fm0tGkakCSMqxn70 zx6+q22+FdoXo0NV25gcESkC/cRnmkE3p8sVPyzHJiIfRRSI6Q3SzszsxYdCmwfVUoeB CCf+tmByziI7Eqj4+2nLxOe+rVWds8Mg+BlpclYMv22ZOmSrBCBbi5/KxMbPVZ94PUH8 VgZqbZJc/4Osvi2h3fqQ2yhEWR2/Z7p5+J9D26NcCKnd+1/bVc+bpO/MC66swQK6/VH2 +AHMkGDSJ5r+3BRoFNlcBASt/+7WwJb/BZUQ5rnlvWDApvzH7TRICADmvcaWVAlX6QXO eksw== MIME-Version: 1.0 Received: by 10.60.14.101 with SMTP id o5mr2500763oec.20.1330630289087; Thu, 01 Mar 2012 11:31:29 -0800 (PST) Received: by 10.182.29.196 with HTTP; Thu, 1 Mar 2012 11:31:29 -0800 (PST) Date: Thu, 1 Mar 2012 12:31:29 -0700 Message-ID: From: Jason Wolfe To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Intel 82574L interface wedging - em7.3.2/8.2-STABLE X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2012 19:56:34 -0000 So since the 7.3.0/7.3.2 code released out of the "Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled" thread I've been having some good results in 8.2-STABLE, and 'wedges' are much less common. I am however still seeing them rarely, using some fuzzy math based on uptime on the new code and number of boxes, about once every 250 days. MUCH better than prior, but wondering if there is something else still lingering? It appears to have the same symptoms as before with a full buffer, where dropped packets start climbing and packets out stall. These servers have MSI-X enabled. I bounced em0 because dropped packets incremented 1036224 to 1036874 and the interface is not incrementing packets out. 9:15PM up 18 days, 5:07, 0 users, load averages: 1.75, 2.25, 2.35 em0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:2c:c3:a5 inet6 fe80::225:90ff:fe2c:c3a4%em0 prefixlen 64 scopeid 0x1 nd6 options=1 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:2c:c3:a5 inet6 fe80::225:90ff:fe2c:c3a5%em1 prefixlen 64 scopeid 0x2 nd6 options=3 media: Ethernet autoselect (1000baseT ) status: active ipfw0: flags=8801 metric 0 mtu 65536 lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 69.28.143.32 netmask 0xffffffff inet 69.28.143.33 netmask 0xffffffff inet 69.28.143.58 netmask 0xffffffff inet 69.28.143.59 netmask 0xffffffff inet 69.28.143.60 netmask 0xffffffff inet 69.28.143.61 netmask 0xffffffff inet 68.142.118.4 netmask 0xffffffff inet 68.142.118.254 netmask 0xffffffff nd6 options=3 lagg0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:25:90:2c:c3:a5 inet 69.164.10.184 netmask 0xffffff80 broadcast 69.164.10.255 inet6 fe80::225:90ff:fe2c:c3a5%lagg0 prefixlen 64 scopeid 0x5 inet6 2607:f4e8:320:14:225:90ff:fe2c:c3a5 prefixlen 64 autoconf nd6 options=3 media: Ethernet autoselect status: active laggproto loadbalance laggport: em0 flags=4 laggport: em1 flags=4 interrupt total rate irq3: uart1 9150 0 cpu0: timer 3147281076 2000 irq256: em0:rx 0 92406843 58 irq257: em0:tx 0 7707454123 4897 irq258: em0:link 2 0 irq259: em1:rx 0 9059996943 5757 irq260: em1:tx 0 7697538938 4891 irq261: em1:link 10325 0 irq262: mps0 1109270744 704 cpu2: timer 3147265253 2000 cpu3: timer 3147265245 2000 cpu1: timer 3147265253 2000 Total 38255763895 24310 65436/10254/75690 mbufs in use (current/cache/total) 9092/4636/13728/5956826 mbuf clusters in use (current/cache/total/max) 9092/980 mbuf+clusters out of packet secondary zone in use (current/cache) 49150/877/50027/2978413 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 231143K/15343K/246486K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 195213 requests for I/O initiated by sendfile 0 calls to protocol drain routines Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop em0 1500 00:25:90:2c:c3:a5 91137540 0 0 49557533207 0 0 1039464 em0 1500 fe80::225:90f fe80::225:90ff:fe 0 - - 5 - - - em1 1500 00:25:90:2c:c3:a5 46770487879 52357 0 49003644601 0 0 132545 em1 1500 fe80::225:90f fe80::225:90ff:fe 0 - - 1 - - - lagg0 1500 00:25:90:2c:c3:a5 46861296630 0 0 98539892148 1172007 0 0 lagg0 1500 69.164.10.128 69.164.10.184 44434261867 - - 98536592195 - - - lagg0 1500 fe80::225:90f fe80::225:90ff:fe 2737 - - 2750 - - - lagg0 1500 2607:f4e8:320 2607:f4e8:320:14: 22188979 - - 22152621 - - - kern.msgbuf: Feb 27 21:15:07 cds447 kernel: Interface is RUNNING and ACTIVE Feb 27 21:15:07 cds447 kernel: em0: hw tdh = 1385, hw tdt = 1385 Feb 27 21:15:07 cds447 kernel: em0: hw rdh = 452, hw rdt = 451 Feb 27 21:15:07 cds447 kernel: em0: Tx Queue Status = 0 Feb 27 21:15:07 cds447 kernel: em0: TX descriptors avail = 2048 Feb 27 21:15:07 cds447 kernel: em0: Tx Descriptors avail failure = 0 Feb 27 21:15:07 cds447 kernel: em0: RX discarded packets = 0 Feb 27 21:15:07 cds447 kernel: em0: RX Next to Check = 452 Feb 27 21:15:07 cds447 kernel: em0: RX Next to Refresh = 451 Feb 27 21:15:07 cds447 kernel: Interface is RUNNING and ACTIVE Feb 27 21:15:07 cds447 kernel: em1: hw tdh = 221, hw tdt = 342 Feb 27 21:15:07 cds447 kernel: em1: hw rdh = 335, hw rdt = 233 Feb 27 21:15:07 cds447 kernel: em1: Tx Queue Status = 0 Feb 27 21:15:07 cds447 kernel: em1: TX descriptors avail = 2048 Feb 27 21:15:07 cds447 kernel: em1: Tx Descriptors avail failure = 0 Feb 27 21:15:07 cds447 kernel: em1: RX discarded packets = 0 Feb 27 21:15:07 cds447 kernel: em1: RX Next to Check = 832 Feb 27 21:15:07 cds447 kernel: em1: RX Next to Refresh = 838 net.inet.ip.intr_queue_maxlen: 512 net.inet.ip.intr_queue_drops: 0 dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.3.2 dev.em.0.%driver: em dev.em.0.%location: slot=0 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 subdevice=0x10d3 class=0x020000 dev.em.0.%parent: pci1 dev.em.0.nvm: -1 dev.em.0.debug: -1 dev.em.0.fc: 3 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.0.eee_control: 0 dev.em.0.link_irq: 2 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1074790984 dev.em.0.rx_control: 67141634 dev.em.0.fc_high_water: 18432 dev.em.0.fc_low_water: 16932 dev.em.0.queue0.txd_head: 1385 dev.em.0.queue0.txd_tail: 1385 dev.em.0.queue0.tx_irq: 7707451554 dev.em.0.queue0.no_desc_avail: 0 dev.em.0.queue0.rxd_head: 659 dev.em.0.queue0.rxd_tail: 658 dev.em.0.queue0.rx_irq: 92406848 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd: 91202036 dev.em.0.mac_stats.good_pkts_recvd: 91202036 dev.em.0.mac_stats.bcast_pkts_recvd: 91083791 dev.em.0.mac_stats.mcast_pkts_recvd: 53524 dev.em.0.mac_stats.rx_frames_64: 91112331 dev.em.0.mac_stats.rx_frames_65_127: 26634 dev.em.0.mac_stats.rx_frames_128_255: 52891 dev.em.0.mac_stats.rx_frames_256_511: 6099 dev.em.0.mac_stats.rx_frames_512_1023: 4081 dev.em.0.mac_stats.rx_frames_1024_1522: 0 dev.em.0.mac_stats.good_octets_recvd: 5845701200 dev.em.0.mac_stats.good_octets_txd: 66047819213524 dev.em.0.mac_stats.total_pkts_txd: 49557613335 dev.em.0.mac_stats.good_pkts_txd: 49557613335 dev.em.0.mac_stats.bcast_pkts_txd: 35 dev.em.0.mac_stats.mcast_pkts_txd: 10502 dev.em.0.mac_stats.tx_frames_64: 191214392 dev.em.0.mac_stats.tx_frames_65_127: 5048362310 dev.em.0.mac_stats.tx_frames_128_255: 37903087 dev.em.0.mac_stats.tx_frames_256_511: 73859562 dev.em.0.mac_stats.tx_frames_512_1023: 491425065 dev.em.0.mac_stats.tx_frames_1024_1522: 43714848919 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.interrupts.asserts: 3 dev.em.0.interrupts.rx_pkt_timer: 0 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.tx_pkt_timer: 0 dev.em.0.interrupts.tx_abs_timer: 0 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_queue_min_thresh: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.rx_overrun: 0 dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.3.2 dev.em.1.%driver: em dev.em.1.%location: slot=0 function=0 dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 subdevice=0x10d3 class=0x020000 dev.em.1.%parent: pci2 dev.em.1.nvm: -1 dev.em.1.debug: -1 dev.em.1.fc: 3 dev.em.1.rx_int_delay: 0 dev.em.1.tx_int_delay: 66 dev.em.1.rx_abs_int_delay: 66 dev.em.1.tx_abs_int_delay: 66 dev.em.1.rx_processing_limit: 100 dev.em.1.eee_control: 0 dev.em.1.link_irq: 9701 dev.em.1.mbuf_alloc_fail: 0 dev.em.1.cluster_alloc_fail: 0 dev.em.1.dropped: 0 dev.em.1.tx_dma_fail: 0 dev.em.1.rx_overruns: 0 dev.em.1.watchdog_timeouts: 0 dev.em.1.device_control: 1074790984 dev.em.1.rx_control: 67141634 dev.em.1.fc_high_water: 18432 dev.em.1.fc_low_water: 16932 dev.em.1.queue0.txd_head: 428 dev.em.1.queue0.txd_tail: 592 dev.em.1.queue0.tx_irq: 7697553110 dev.em.1.queue0.no_desc_avail: 0 dev.em.1.queue0.rxd_head: 190 dev.em.1.queue0.rxd_tail: 189 dev.em.1.queue0.rx_irq: 8990697150 dev.em.1.mac_stats.excess_coll: 0 dev.em.1.mac_stats.single_coll: 0 dev.em.1.mac_stats.multiple_coll: 0 dev.em.1.mac_stats.late_coll: 0 dev.em.1.mac_stats.collision_count: 0 dev.em.1.mac_stats.symbol_errors: 0 dev.em.1.mac_stats.sequence_errors: 0 dev.em.1.mac_stats.defer_count: 0 dev.em.1.mac_stats.missed_packets: 52357 dev.em.1.mac_stats.recv_no_buff: 12234 dev.em.1.mac_stats.recv_undersize: 0 dev.em.1.mac_stats.recv_fragmented: 0 dev.em.1.mac_stats.recv_oversize: 0 dev.em.1.mac_stats.recv_jabber: 0 dev.em.1.mac_stats.recv_errs: 0 dev.em.1.mac_stats.crc_errs: 0 dev.em.1.mac_stats.alignment_errs: 0 dev.em.1.mac_stats.coll_ext_errs: 0 dev.em.1.mac_stats.xon_recvd: 0 dev.em.1.mac_stats.xon_txd: 0 dev.em.1.mac_stats.xoff_recvd: 0 dev.em.1.mac_stats.xoff_txd: 0 dev.em.1.mac_stats.total_pkts_recvd: 46770604210 dev.em.1.mac_stats.good_pkts_recvd: 46770551853 dev.em.1.mac_stats.bcast_pkts_recvd: 91073541 dev.em.1.mac_stats.mcast_pkts_recvd: 53519 dev.em.1.mac_stats.rx_frames_64: 14339105496 dev.em.1.mac_stats.rx_frames_65_127: 23186724273 dev.em.1.mac_stats.rx_frames_128_255: 38625485 dev.em.1.mac_stats.rx_frames_256_511: 116531779 dev.em.1.mac_stats.rx_frames_512_1023: 165521693 dev.em.1.mac_stats.rx_frames_1024_1522: 8924043127 dev.em.1.mac_stats.good_octets_recvd: 16290987983606 dev.em.1.mac_stats.good_octets_txd: 65599822022242 dev.em.1.mac_stats.total_pkts_txd: 49003789456 dev.em.1.mac_stats.good_pkts_txd: 49003789452 dev.em.1.mac_stats.bcast_pkts_txd: 10085 dev.em.1.mac_stats.mcast_pkts_txd: 11 dev.em.1.mac_stats.tx_frames_64: 170250212 dev.em.1.mac_stats.tx_frames_65_127: 4806984601 dev.em.1.mac_stats.tx_frames_128_255: 36525028 dev.em.1.mac_stats.tx_frames_256_511: 71876326 dev.em.1.mac_stats.tx_frames_512_1023: 482360156 dev.em.1.mac_stats.tx_frames_1024_1522: 43435793133 dev.em.1.mac_stats.tso_txd: 0 dev.em.1.mac_stats.tso_ctx_fail: 0 dev.em.1.interrupts.asserts: 7943 dev.em.1.interrupts.rx_pkt_timer: 1 dev.em.1.interrupts.rx_abs_timer: 0 dev.em.1.interrupts.tx_pkt_timer: 0 dev.em.1.interrupts.tx_abs_timer: 1 dev.em.1.interrupts.tx_queue_empty: 0 dev.em.1.interrupts.tx_queue_min_thresh: 0 dev.em.1.interrupts.rx_desc_min_thresh: 0 dev.em.1.interrupts.rx_overrun: 6 hw.em.eee_setting: 0 hw.em.rx_process_limit: 100 hw.em.enable_msix: 1 hw.em.sbp: 0 hw.em.smart_pwr_down: 0 hw.em.txd: 2048 hw.em.rxd: 2048 hw.em.rx_abs_int_delay: 66 hw.em.tx_abs_int_delay: 66 hw.em.rx_int_delay: 0 hw.em.tx_int_delay: 66 Jason