From owner-freebsd-net@FreeBSD.ORG Sun Feb 2 11:49:47 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1F158B9A for ; Sun, 2 Feb 2014 11:49:47 +0000 (UTC) Received: from mail-vc0-x232.google.com (mail-vc0-x232.google.com [IPv6:2607:f8b0:400c:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CF8C015ED for ; Sun, 2 Feb 2014 11:49:46 +0000 (UTC) Received: by mail-vc0-f178.google.com with SMTP id ik5so4208884vcb.9 for ; Sun, 02 Feb 2014 03:49:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=hSR4qZ2YiONrSAcX1P09qxoRN0RP8a7oE3LoHx7Cw8Y=; b=GsqeBNU7y7iv43NjntDnekC8AA2tSIwHuMdbEeLOk3ongh4OQnfI2PYe/520F2VAH6 XUzaSXjki/iR6FHs62LlF87Q5qyVupcwXvOGPyK2CcOA2OLv54zJ2DBnvK0ttDiJrL9f gPYrcudXXg4S2FlVXXnA/ATH/70lbW34m3ELFGqjQbn6oQo2iDuwxzqT17I8P1LLmwwc 0zBSHqmTqh9Jz/w/LOCjjMUxHkVZjTnYMacePj36xcBPdGngMCylsXTJmxOqmNMD8doK 1MmGswUD35ld1tEo03HS92Kvzj3LE+f3oHlozJ2/bhsiBM8xlyTAYk2kvLaXo4i8GL0g 5pAg== MIME-Version: 1.0 X-Received: by 10.58.66.137 with SMTP id f9mr24311566vet.11.1391341785965; Sun, 02 Feb 2014 03:49:45 -0800 (PST) Sender: ndenev@gmail.com Received: by 10.220.78.84 with HTTP; Sun, 2 Feb 2014 03:49:45 -0800 (PST) In-Reply-To: <52EC573B.109@sentex.net> References: <52EC573B.109@sentex.net> Date: Sun, 2 Feb 2014 11:49:45 +0000 X-Google-Sender-Auth: kP41reusPt2zQgoMYMpXXwQ9Nkw Message-ID: Subject: Re: missing missing packets in igb stats ? From: Nikolay Denev To: Mike Tancsa Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Feb 2014 11:49:47 -0000 On Sat, Feb 1, 2014 at 2:08 AM, Mike Tancsa wrote: > Hi Jack, > I was testing out forwarding and firewalling speeds of the igb driver on > RELENG_10 and noticed something odd. > > I have 2 boxes connected to a FreeBSD box in the middle > > FreeBSD-A(em1)-----------(igb1)Router-1(igb0)----------(em1)FreeBSD-B > > So Box A generates packets as fast as it can to FreeBSD box B's em1 nic. > Router-1 is a FreeBSD box as releng10. Watching ifstat on Router-1 as I > execute the command on FreeBSD-A > # ./netblast 1.1.1.2 500 100 20 > > start: 1391219372.477992294 > finish: 1391219392.496952108 > send calls: 10877557 > send errors: 0 > approx send rate: 543877 > approx error rate: 0 > > I see on the router-1 box > > igb0 igb1 > Kbps in Kbps out Kbps in Kbps out > 0.00 0.00 0.00 0.00 > 1600.61 191639.1 280888.7 0.00 > 3669.84 434348.6 636134.9 0.00 > 3706.56 438636.7 596650.5 0.00 > 3755.10 444358.9 562814.3 0.00 > 3714.89 439478.5 562056.4 0.00 > 3796.79 449397.9 562042.9 0.00 > 3786.02 447957.2 577561.4 0.00 > 3629.18 429453.4 601285.7 0.00 > 3728.48 441312.7 597785.3 0.00 > 3806.67 450401.2 596247.0 0.00 > 3854.79 456150.2 597865.7 0.00 > 3690.11 436552.1 596695.8 0.00 > 3676.08 435002.6 596462.8 0.00 > 3730.35 441535.2 597132.1 0.00 > 3680.43 435518.2 596960.3 0.00 > 3741.41 442685.3 597750.8 0.00 > 3691.93 436870.6 596236.9 0.00 > 3627.31 429120.5 594116.8 0.00 > 3661.97 433492.7 595812.0 0.00 > 3693.86 437169.0 597826.9 0.00 > 2046.18 240635.3 331656.3 0.00 > 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 > > Notice the rate of traffic coming in on igb1 is higher than what is going > out on igb0. Box A thinks it sent traffic at some 536,616 pkts per second > or 590Mb/s. However, traffic going out is slower, and what is seen at box B > is less. It sees the traffic at 286Mb/s and 357,873 pps > > Given the lost packets, should this not show up somewhere in the igb > statistics ? > > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PEG0.PEGP > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x8086 > subdevice=0xa03c class=0x020000 > dev.igb.0.%parent: pci1 > dev.igb.0.nvm: -1 > dev.igb.0.enable_aim: 1 > dev.igb.0.fc: 3 > dev.igb.0.rx_processing_limit: 100 > dev.igb.0.link_irq: 2 > dev.igb.0.dropped: 0 > dev.igb.0.tx_dma_fail: 0 > dev.igb.0.rx_overruns: 0 > dev.igb.0.watchdog_timeouts: 0 > dev.igb.0.device_control: 1488978497 > dev.igb.0.rx_control: 67141634 > dev.igb.0.interrupt_mask: 4 > dev.igb.0.extended_int_mask: 2147483655 > dev.igb.0.tx_buf_alloc: 0 > dev.igb.0.rx_buf_alloc: 0 > dev.igb.0.fc_high_water: 58976 > dev.igb.0.fc_low_water: 58960 > dev.igb.0.queue0.no_desc_avail: 19682298 > dev.igb.0.queue0.tx_packets: 20962740 > dev.igb.0.queue0.rx_packets: 1101622 > dev.igb.0.queue0.rx_bytes: 66097424 > dev.igb.0.queue0.lro_queued: 0 > dev.igb.0.queue0.lro_flushed: 0 > dev.igb.0.queue1.no_desc_avail: 32582207 > dev.igb.0.queue1.tx_packets: 50082567 > dev.igb.0.queue1.rx_packets: 6598 > dev.igb.0.queue1.rx_bytes: 462728 > dev.igb.0.queue1.lro_queued: 0 > dev.igb.0.queue1.lro_flushed: 0 > dev.igb.0.mac_stats.excess_coll: 0 > dev.igb.0.mac_stats.single_coll: 0 > dev.igb.0.mac_stats.multiple_coll: 0 > dev.igb.0.mac_stats.late_coll: 0 > dev.igb.0.mac_stats.collision_count: 0 > dev.igb.0.mac_stats.symbol_errors: 0 > dev.igb.0.mac_stats.sequence_errors: 0 > dev.igb.0.mac_stats.defer_count: 138912 > dev.igb.0.mac_stats.missed_packets: 0 > dev.igb.0.mac_stats.recv_no_buff: 0 > dev.igb.0.mac_stats.recv_undersize: 0 > dev.igb.0.mac_stats.recv_fragmented: 0 > dev.igb.0.mac_stats.recv_oversize: 0 > dev.igb.0.mac_stats.recv_jabber: 0 > dev.igb.0.mac_stats.recv_errs: 0 > dev.igb.0.mac_stats.crc_errs: 0 > dev.igb.0.mac_stats.alignment_errs: 0 > dev.igb.0.mac_stats.coll_ext_errs: 0 > dev.igb.0.mac_stats.xon_recvd: 550808 > dev.igb.0.mac_stats.xon_txd: 0 > dev.igb.0.mac_stats.xoff_recvd: 550808 > dev.igb.0.mac_stats.xoff_txd: 0 > dev.igb.0.mac_stats.total_pkts_recvd: 1108220 > dev.igb.0.mac_stats.good_pkts_recvd: 6604 > dev.igb.0.mac_stats.bcast_pkts_recvd: 0 > dev.igb.0.mac_stats.mcast_pkts_recvd: 0 > dev.igb.0.mac_stats.rx_frames_64: 1 > dev.igb.0.mac_stats.rx_frames_65_127: 6603 > dev.igb.0.mac_stats.rx_frames_128_255: 0 > dev.igb.0.mac_stats.rx_frames_256_511: 0 > dev.igb.0.mac_stats.rx_frames_512_1023: 0 > dev.igb.0.mac_stats.rx_frames_1024_1522: 0 > dev.igb.0.mac_stats.good_octets_recvd: 489608 > dev.igb.0.mac_stats.good_octets_txd: 10120060648 > dev.igb.0.mac_stats.total_pkts_txd: 71045307 > dev.igb.0.mac_stats.good_pkts_txd: 71045307 > dev.igb.0.mac_stats.bcast_pkts_txd: 2 > dev.igb.0.mac_stats.mcast_pkts_txd: 0 > dev.igb.0.mac_stats.tx_frames_64: 2 > dev.igb.0.mac_stats.tx_frames_65_127: 5051081 > dev.igb.0.mac_stats.tx_frames_128_255: 65994224 > dev.igb.0.mac_stats.tx_frames_256_511: 0 > dev.igb.0.mac_stats.tx_frames_512_1023: 0 > dev.igb.0.mac_stats.tx_frames_1024_1522: 0 > dev.igb.0.mac_stats.tso_txd: 0 > dev.igb.0.mac_stats.tso_ctx_fail: 0 > dev.igb.0.interrupts.asserts: 6564060 > dev.igb.0.interrupts.rx_pkt_timer: 1108207 > dev.igb.0.interrupts.rx_abs_timer: 0 > dev.igb.0.interrupts.tx_pkt_timer: 0 > dev.igb.0.interrupts.tx_abs_timer: 1108220 > dev.igb.0.interrupts.tx_queue_empty: 71044772 > dev.igb.0.interrupts.tx_queue_min_thresh: 0 > dev.igb.0.interrupts.rx_desc_min_thresh: 0 > dev.igb.0.interrupts.rx_overrun: 0 > dev.igb.0.host.breaker_tx_pkt: 0 > dev.igb.0.host.host_tx_pkt_discard: 0 > dev.igb.0.host.rx_pkt: 13 > dev.igb.0.host.breaker_rx_pkts: 0 > dev.igb.0.host.breaker_rx_pkt_drop: 0 > dev.igb.0.host.tx_good_pkt: 535 > dev.igb.0.host.breaker_tx_pkt_drop: 0 > dev.igb.0.host.rx_good_bytes: 70993032 > dev.igb.0.host.tx_good_bytes: 10120060648 > dev.igb.0.host.length_errors: 0 > dev.igb.0.host.serdes_violation_pkt: 0 > dev.igb.0.host.header_redir_missed: 0 > dev.igb.0.wake: 0 > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x8086 > subdevice=0xa03c class=0x020000 > dev.igb.1.%parent: pci1 > dev.igb.1.nvm: -1 > dev.igb.1.enable_aim: 1 > dev.igb.1.fc: 3 > dev.igb.1.rx_processing_limit: 100 > dev.igb.1.link_irq: 2 > dev.igb.1.dropped: 0 > dev.igb.1.tx_dma_fail: 0 > dev.igb.1.rx_overruns: 0 > dev.igb.1.watchdog_timeouts: 0 > dev.igb.1.device_control: 1488978497 > dev.igb.1.rx_control: 67141634 > dev.igb.1.interrupt_mask: 4 > dev.igb.1.extended_int_mask: 2147483655 > dev.igb.1.tx_buf_alloc: 0 > dev.igb.1.rx_buf_alloc: 0 > dev.igb.1.fc_high_water: 58976 > dev.igb.1.fc_low_water: 58960 > dev.igb.1.queue0.no_desc_avail: 0 > dev.igb.1.queue0.tx_packets: 14 > dev.igb.1.queue0.rx_packets: 27770289 > dev.igb.1.queue0.rx_bytes: 3632804418 > dev.igb.1.queue0.lro_queued: 0 > dev.igb.1.queue0.lro_flushed: 0 > dev.igb.1.queue1.no_desc_avail: 0 > dev.igb.1.queue1.tx_packets: 6599 > dev.igb.1.queue1.rx_packets: 58098597 > dev.igb.1.queue1.rx_bytes: 8250006086 > dev.igb.1.queue1.lro_queued: 0 > dev.igb.1.queue1.lro_flushed: 0 > dev.igb.1.mac_stats.excess_coll: 0 > dev.igb.1.mac_stats.single_coll: 0 > dev.igb.1.mac_stats.multiple_coll: 0 > dev.igb.1.mac_stats.late_coll: 0 > dev.igb.1.mac_stats.collision_count: 0 > dev.igb.1.mac_stats.symbol_errors: 0 > dev.igb.1.mac_stats.sequence_errors: 0 > dev.igb.1.mac_stats.defer_count: 0 > dev.igb.1.mac_stats.missed_packets: 0 > dev.igb.1.mac_stats.recv_no_buff: 0 > dev.igb.1.mac_stats.recv_undersize: 0 > dev.igb.1.mac_stats.recv_fragmented: 0 > dev.igb.1.mac_stats.recv_oversize: 0 > dev.igb.1.mac_stats.recv_jabber: 0 > dev.igb.1.mac_stats.recv_errs: 0 > dev.igb.1.mac_stats.crc_errs: 0 > dev.igb.1.mac_stats.alignment_errs: 0 > dev.igb.1.mac_stats.coll_ext_errs: 0 > dev.igb.1.mac_stats.xon_recvd: 0 > dev.igb.1.mac_stats.xon_txd: 0 > dev.igb.1.mac_stats.xoff_recvd: 0 > dev.igb.1.mac_stats.xoff_txd: 0 > dev.igb.1.mac_stats.total_pkts_recvd: 85868886 > dev.igb.1.mac_stats.good_pkts_recvd: 85868886 > dev.igb.1.mac_stats.bcast_pkts_recvd: 31 > dev.igb.1.mac_stats.mcast_pkts_recvd: 0 > dev.igb.1.mac_stats.rx_frames_64: 5 > dev.igb.1.mac_stats.rx_frames_65_127: 6211527 > dev.igb.1.mac_stats.rx_frames_128_255: 79657327 > dev.igb.1.mac_stats.rx_frames_256_511: 27 > dev.igb.1.mac_stats.rx_frames_512_1023: 0 > dev.igb.1.mac_stats.rx_frames_1024_1522: 0 > dev.igb.1.mac_stats.good_octets_recvd: 12226286048 > dev.igb.1.mac_stats.good_octets_txd: 490260 > dev.igb.1.mac_stats.total_pkts_txd: 6613 > dev.igb.1.mac_stats.good_pkts_txd: 6613 > dev.igb.1.mac_stats.bcast_pkts_txd: 4 > dev.igb.1.mac_stats.mcast_pkts_txd: 0 > dev.igb.1.mac_stats.tx_frames_64: 8 > dev.igb.1.mac_stats.tx_frames_65_127: 6605 > dev.igb.1.mac_stats.tx_frames_128_255: 0 > dev.igb.1.mac_stats.tx_frames_256_511: 0 > dev.igb.1.mac_stats.tx_frames_512_1023: 0 > dev.igb.1.mac_stats.tx_frames_1024_1522: 0 > dev.igb.1.mac_stats.tso_txd: 0 > dev.igb.1.mac_stats.tso_ctx_fail: 0 > dev.igb.1.interrupts.asserts: 8707927 > dev.igb.1.interrupts.rx_pkt_timer: 85867976 > dev.igb.1.interrupts.rx_abs_timer: 0 > dev.igb.1.interrupts.tx_pkt_timer: 0 > dev.igb.1.interrupts.tx_abs_timer: 85868886 > dev.igb.1.interrupts.tx_queue_empty: 6613 > dev.igb.1.interrupts.tx_queue_min_thresh: 0 > dev.igb.1.interrupts.rx_desc_min_thresh: 0 > dev.igb.1.interrupts.rx_overrun: 0 > dev.igb.1.host.breaker_tx_pkt: 0 > dev.igb.1.host.host_tx_pkt_discard: 0 > dev.igb.1.host.rx_pkt: 910 > dev.igb.1.host.breaker_rx_pkts: 0 > dev.igb.1.host.breaker_rx_pkt_drop: 0 > dev.igb.1.host.tx_good_pkt: 0 > dev.igb.1.host.breaker_tx_pkt_drop: 0 > dev.igb.1.host.rx_good_bytes: 12226288092 > dev.igb.1.host.tx_good_bytes: 490260 > dev.igb.1.host.length_errors: 0 > dev.igb.1.host.serdes_violation_pkt: 0 > dev.igb.1.host.header_redir_missed: 0 > > > Motherboard is Intel > > Base Board Information > Manufacturer: Intel Corporation > Product Name: DH87RL > Version: AAG74240-401 > Serial Number: BQRL330000Q9 > > > NIC is dual port > > igb0@pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086 > rev=0x01 hdr=0x00 > vendor = 'Intel Corporation' > device = '82576 Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xf7c20000, size 131072, > enabled > bar [14] = type Memory, range 32, base 0xf7800000, size 4194304, > enabled > bar [18] = type I/O Port, range 32, base 0xe020, size 32, enabled > bar [1c] = type Memory, range 32, base 0xf7c44000, size 16384, enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks > cap 11[70] = MSI-X supports 10 messages, enabled > Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] > cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR link x4(x4) > speed 2.5(2.5) ASPM disabled(L0s/L1) > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected > ecap 0003[140] = Serial 1 90e2baffff5eb48a > ecap 000e[150] = ARI 1 > ecap 0010[160] = SRIOV 1 > igb1@pci0:1:0:1: class=0x020000 card=0xa03c8086 chip=0x10c98086 > rev=0x01 hdr=0x00 > vendor = 'Intel Corporation' > device = '82576 Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xf7c00000, size 131072, > enabled > bar [14] = type Memory, range 32, base 0xf7000000, size 4194304, > enabled > bar [18] = type I/O Port, range 32, base 0xe000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xf7c40000, size 16384, enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks > cap 11[70] = MSI-X supports 10 messages, enabled > Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] > cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR link x4(x4) > speed 2.5(2.5) ASPM disabled(L0s/L1) > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected > ecap 0003[140] = Serial 1 90e2baffff5eb48a > ecap 000e[150] = ARI 1 > ecap 0010[160] = SRIOV 1 > > > root@intel4gen-9:/usr/home/mdtancsa # netstat -m > 6141/6489/12630 mbufs in use (current/cache/total) > 6139/5871/12010/487416 mbuf clusters in use (current/cache/total/max) > 6139/5861 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/5/5/243708 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/72209 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/40618 16k jumbo clusters in use (current/cache/total/max) > 13813K/13384K/27197K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > root@intel4gen-9:/usr/home/mdtancsa # > > > > > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/ > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" Just a guess, but this might be happening before the driver. Anything interesting in "netstat -s" for ip and udp? There is some inbound traffic on igb0. Are these ICMP udp port unreach? --Nikolay