From owner-freebsd-net@FreeBSD.ORG Wed Jul 7 00:09:10 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 373F3106567D for ; Wed, 7 Jul 2010 00:09:10 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id D59EC8FC1D for ; Wed, 7 Jul 2010 00:09:09 +0000 (UTC) Received: by qyk30 with SMTP id 30so2760261qyk.13 for ; Tue, 06 Jul 2010 17:09:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=VEKc2wRV7xNRTiihkcK947YKZauye8KbdKVM93Wv3hg=; b=GD5uqAQpTREw68uUObjptbbXT/JEtoCipGPn6IzcXXRwU7yydP2sdKmxnEH8cvuaL6 qh8yW2Fk76thbuiMxCOGroChVG/96enFPtHUVzoeMPts6M2J7K+nW7GXxRj6j7JOA+1F ckzRkMtkZOkfWgqH+wD8IJJDno1Gk63SVPiow= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=PRX2nElC9DElDkY4Qeyw2hPH8yBayPEE+tj0MtiW08P780svLHPS9RfRPmxFBdgYHe Gjh+C4P7F1r15cvdDZatC4gJfKD1tK7nlXY/tOWvc+m39jee1ij6nZ73ufFJAUEgyFTT fCCPxEW/ZlLLlm5Y0+w/PQfxGMST90uUvVRwk= MIME-Version: 1.0 Received: by 10.224.20.9 with SMTP id d9mr2966885qab.50.1278461340157; Tue, 06 Jul 2010 17:09:00 -0700 (PDT) Received: by 10.229.220.5 with HTTP; Tue, 6 Jul 2010 17:09:00 -0700 (PDT) In-Reply-To: <1278404933.20433.26.camel@stormi-desktop> References: <1278330234.10826.18.camel@stormi-desktop> <1278356796.10826.35.camel@stormi-desktop> <1278404933.20433.26.camel@stormi-desktop> Date: Tue, 6 Jul 2010 17:09:00 -0700 Message-ID: From: Jack Vogel To: Shtorm Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: Intel 82574L Gigabit Ethernet Controller X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2010 00:09:10 -0000 Yow, 30 vlans, but only em1 is using vlans not em0? Is only em1 having watchdogs? I noticed you appear to have flow control off, maybe turning it on would help. I would like to see the log messages from the watchdogs. Jack On Tue, Jul 6, 2010 at 1:28 AM, Shtorm wrote: > On Mon, 2010-07-05 at 22:02 -0700, Jack Vogel wrote: > > Cleaner in the Makefile, let me know how it goes. > > > > Jack > > > > > > On Mon, Jul 5, 2010 at 12:06 PM, Yuriy A. Korobko > > wrote: > > > > > On Mon, 2010-07-05 at 11:11 -0700, Jack Vogel wrote: > > > > Are you defining 'EM_MULTIQUEUE', its off by default and needs to be > > > > defined somewhere by you. > > > > > > > > You will only see the two queues used if you have two different > > > connections > > > > operating at once. > > > > > > > > Jack > > Recompiled module (actually whole kernel) with EM_MULTIQUEUE, but got a > some watchdog timeouts. > > Here is example of netstat when it happens: > > 2346 0 0 583370 2642 0 2988379 0 > 2250 0 0 550961 2384 0 2835276 0 > 2634 0 0 703410 2733 0 2971417 0 > 2741 0 0 695884 2748 0 3061573 0 > 2811 0 0 618520 2274 0 3273868 0 > 3220 0 0 687145 2068 0 3417857 0 > 901 0 0 231694 166 0 1565839 0 > 752 0 0 216135 0 0 270247 0 > 685 0 0 208745 0 0 243442 0 > 713 0 0 178089 0 0 230472 0 > 735 0 0 159555 0 0 178435 0 > 616 0 0 145222 0 0 179022 0 > input (em1) output > packets errs idrops bytes packets errs bytes colls > 616 0 0 129608 0 0 120929 0 > 659 0 0 113806 0 0 105707 0 > 622 0 0 106247 0 0 107825 0 > 645 0 0 101593 0 0 38023 0 > 483 0 0 61681 0 0 8547 0 > > Watchdog timeout on em1 message on console > > 0 0 0 0 0 1 19095 0 > 0 0 0 0 0 0 222 0 > 0 0 0 0 0 0 0 0 > 243 0 0 30224 1783 0 1708875 0 > 576 0 0 62350 105 0 7710 0 > 467 0 0 49758 4 0 4388 0 > 500 0 0 53415 23 0 24743 0 > 437 0 0 50135 21 0 16733 0 > > After this traffic increased again up to 3 kpps and another watchdog > timeout happened. > > em0: flags=8843 metric 0 mtu > 1500 > options=2098 > ether 00:30:48:bc:ab:ca > inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.y > media: Ethernet autoselect (1000baseT ) > status: active > > em1: flags=8843 metric 0 mtu > 1500 > options=2098 > ether 00:30:48:bc:ab:cb > media: Ethernet autoselect (1000baseT ) > status: active > > em1 have 30 vlans and mpd5 as pppoe server listening on each vlan. > > Here is sysctl dev.em > > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.0.5 > dev.em.0.%driver: em > dev.em.0.%location: slot=0 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 > subdevice=0x10d3 class=0x020000 > dev.em.0.%parent: pci1 > dev.em.0.nvm: -1 > dev.em.0.rx_int_delay: 200 > dev.em.0.tx_int_delay: 200 > dev.em.0.rx_abs_int_delay: 500 > dev.em.0.tx_abs_int_delay: 500 > dev.em.0.rx_processing_limit: 100 > dev.em.0.link_irq: 2 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 0 > dev.em.0.fc_high_water: 18432 > dev.em.0.fc_low_water: 16932 > dev.em.0.mac_stats.excess_coll: 0 > dev.em.0.mac_stats.symbol_errors: 0 > dev.em.0.mac_stats.sequence_errors: 0 > dev.em.0.mac_stats.defer_count: 0 > dev.em.0.mac_stats.missed_packets: 0 > dev.em.0.mac_stats.recv_no_buff: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_errs: 0 > dev.em.0.mac_stats.coll_ext_errs: 0 > dev.em.0.mac_stats.rx_overruns: 0 > dev.em.0.mac_stats.watchdog_timeouts: 0 > dev.em.0.mac_stats.xon_recvd: 0 > dev.em.0.mac_stats.xon_txd: 0 > dev.em.0.mac_stats.xoff_recvd: 0 > dev.em.0.mac_stats.xoff_txd: 0 > dev.em.0.mac_stats.total_pkts_recvd: 732916 > dev.em.0.mac_stats.good_pkts_recvd: 732916 > dev.em.0.mac_stats.bcast_pkts_recvd: 1238 > dev.em.0.mac_stats.mcast_pkts_recvd: 0 > dev.em.0.mac_stats.rx_frames_64: 84622 > dev.em.0.mac_stats.rx_frames_65_127: 128831 > dev.em.0.mac_stats.rx_frames_128_255: 34037 > dev.em.0.mac_stats.rx_frames_256_511: 30206 > dev.em.0.mac_stats.rx_frames_512_1023: 24919 > dev.em.0.mac_stats.rx_frames_1024_1522: 430301 > dev.em.0.mac_stats.good_octets_recvd: 0 > dev.em.0.mac_stats.good_octest_txd: 0 > dev.em.0.mac_stats.total_pkts_txd: 678078 > dev.em.0.mac_stats.good_pkts_txd: 678078 > dev.em.0.mac_stats.bcast_pkts_txd: 109 > dev.em.0.mac_stats.mcast_pkts_txd: 0 > dev.em.0.mac_stats.tx_frames_64: 324803 > dev.em.0.mac_stats.tx_frames_65_127: 196866 > dev.em.0.mac_stats.tx_frames_128_255: 47362 > dev.em.0.mac_stats.tx_frames_256_511: 31917 > dev.em.0.mac_stats.tx_frames_512_1023: 44997 > dev.em.0.mac_stats.tx_frames_1024_1522: 32133 > dev.em.0.mac_stats.tso_txd: 0 > dev.em.0.mac_stats.tso_ctx_fail: 0 > dev.em.0.interrupts.asserts: 0 > dev.em.0.interrupts.rx_pkt_timer: 0 > dev.em.0.interrupts.rx_abs_timer: 0 > dev.em.0.interrupts.tx_pkt_timer: 0 > dev.em.0.interrupts.tx_abs_timer: 0 > dev.em.0.interrupts.tx_queue_empty: 0 > dev.em.0.interrupts.tx_queue_min_thresh: 0 > dev.em.0.interrupts.rx_desc_min_thresh: 0 > dev.em.0.interrupts.rx_overrun: 0 > dev.em.0.host.breaker_tx_pkt: 0 > dev.em.0.host.host_tx_pkt_discard: 0 > dev.em.0.host.rx_pkt: 0 > dev.em.0.host.breaker_rx_pkts: 0 > dev.em.0.host.breaker_rx_pkt_drop: 0 > dev.em.0.host.tx_good_pkt: 0 > dev.em.0.host.breaker_tx_pkt_drop: 0 > dev.em.0.host.rx_good_bytes: 0 > dev.em.0.host.tx_good_bytes: 0 > dev.em.0.host.length_errors: 0 > dev.em.0.host.serdes_violation_pkt: 0 > dev.em.0.host.header_redir_missed: 0 > dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.0.5 > dev.em.1.%driver: em > dev.em.1.%location: slot=0 function=0 > dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 > subdevice=0x10d3 class=0x020000 > dev.em.1.%parent: pci2 > dev.em.1.nvm: -1 > dev.em.1.rx_int_delay: 200 > dev.em.1.tx_int_delay: 200 > dev.em.1.rx_abs_int_delay: 500 > dev.em.1.tx_abs_int_delay: 500 > dev.em.1.rx_processing_limit: 100 > dev.em.1.link_irq: 7 > dev.em.1.mbuf_alloc_fail: 0 > dev.em.1.cluster_alloc_fail: 0 > dev.em.1.dropped: 0 > dev.em.1.tx_dma_fail: 0 > dev.em.1.fc_high_water: 18432 > dev.em.1.fc_low_water: 16932 > dev.em.1.mac_stats.excess_coll: 0 > dev.em.1.mac_stats.symbol_errors: 0 > dev.em.1.mac_stats.sequence_errors: 0 > dev.em.1.mac_stats.defer_count: 0 > dev.em.1.mac_stats.missed_packets: 0 > dev.em.1.mac_stats.recv_no_buff: 0 > dev.em.1.mac_stats.recv_errs: 0 > dev.em.1.mac_stats.crc_errs: 0 > dev.em.1.mac_stats.alignment_errs: 0 > dev.em.1.mac_stats.coll_ext_errs: 0 > dev.em.1.mac_stats.rx_overruns: 0 > dev.em.1.mac_stats.watchdog_timeouts: 3 > dev.em.1.mac_stats.xon_recvd: 0 > dev.em.1.mac_stats.xon_txd: 0 > dev.em.1.mac_stats.xoff_recvd: 0 > dev.em.1.mac_stats.xoff_txd: 0 > dev.em.1.mac_stats.total_pkts_recvd: 638131 > dev.em.1.mac_stats.good_pkts_recvd: 638131 > dev.em.1.mac_stats.bcast_pkts_recvd: 49276 > dev.em.1.mac_stats.mcast_pkts_recvd: 0 > dev.em.1.mac_stats.rx_frames_64: 8091 > dev.em.1.mac_stats.rx_frames_65_127: 472864 > dev.em.1.mac_stats.rx_frames_128_255: 46165 > dev.em.1.mac_stats.rx_frames_256_511: 33998 > dev.em.1.mac_stats.rx_frames_512_1023: 44560 > dev.em.1.mac_stats.rx_frames_1024_1522: 32453 > dev.em.1.mac_stats.good_octets_recvd: 0 > dev.em.1.mac_stats.good_octest_txd: 0 > dev.em.1.mac_stats.total_pkts_txd: 668981 > dev.em.1.mac_stats.good_pkts_txd: 668981 > dev.em.1.mac_stats.bcast_pkts_txd: 0 > dev.em.1.mac_stats.mcast_pkts_txd: 12 > dev.em.1.mac_stats.tx_frames_64: 5594 > dev.em.1.mac_stats.tx_frames_65_127: 166028 > dev.em.1.mac_stats.tx_frames_128_255: 33061 > dev.em.1.mac_stats.tx_frames_256_511: 28853 > dev.em.1.mac_stats.tx_frames_512_1023: 24279 > dev.em.1.mac_stats.tx_frames_1024_1522: 411166 > dev.em.1.mac_stats.tso_txd: 0 > dev.em.1.mac_stats.tso_ctx_fail: 0 > dev.em.1.interrupts.asserts: 0 > dev.em.1.interrupts.rx_pkt_timer: 0 > dev.em.1.interrupts.rx_abs_timer: 0 > dev.em.1.interrupts.tx_pkt_timer: 0 > dev.em.1.interrupts.tx_abs_timer: 0 > dev.em.1.interrupts.tx_queue_empty: 0 > dev.em.1.interrupts.tx_queue_min_thresh: 0 > dev.em.1.interrupts.rx_desc_min_thresh: 0 > dev.em.1.interrupts.rx_overrun: 0 > dev.em.1.host.breaker_tx_pkt: 0 > dev.em.1.host.host_tx_pkt_discard: 0 > dev.em.1.host.rx_pkt: 0 > dev.em.1.host.breaker_rx_pkts: 0 > dev.em.1.host.breaker_rx_pkt_drop: 0 > dev.em.1.host.tx_good_pkt: 0 > dev.em.1.host.breaker_tx_pkt_drop: 0 > dev.em.1.host.rx_good_bytes: 0 > dev.em.1.host.tx_good_bytes: 0 > dev.em.1.host.length_errors: 0 > dev.em.1.host.serdes_violation_pkt: 0 > dev.em.1.host.header_redir_missed: 0 > > loader.conf variables: > > if_em_load=YES > hw.em.rxd=4096 > hw.em.txd=4096 > hw.em.tx_int_delay=200 > hw.em.rx_int_delay=200 > hw.em.tx_abs_int_delay=500 > hw.em.rx_abs_int_delay=500 > hw.em.enable_msix=1 > hw.em.msix_queues=2 > hw.em.rx_process_limit=100 > hw.em.fc_setting=0 > > > And here is vmstat -i > > interrupt total rate > irq16: uhci0 2673 2 > irq18: ehci0 uhci5 2 0 > irq19: uhci2 uhci4 70 0 > irq23: uhci3 ehci1 8835 9 > cpu0: timer 3743479 3957 > irq256: em0 662219 700 > irq257: em0 717531 758 > irq259: em0 12142 12 > irq260: em0 6100 6 > irq261: em1 694169 733 > irq262: em1 217388 229 > irq264: em1 463023 489 > irq265: em1 11 0 > cpu1: timer 3743411 3957 > cpu6: timer 3743408 3957 > cpu7: timer 3743407 3957 > cpu4: timer 3743408 3957 > cpu5: timer 3743407 3957 > cpu3: timer 3743408 3957 > cpu2: timer 3743409 3957 > Total 32731500 34599 > > > If you need more information just let me know, I can use this box for > tests at night until others can handle traffic. > > >