From owner-freebsd-net@freebsd.org Thu Jun 2 21:40:16 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10D18B677AC for ; Thu, 2 Jun 2016 21:40:16 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vk0-x236.google.com (mail-vk0-x236.google.com [IPv6:2607:f8b0:400c:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C72641A13 for ; Thu, 2 Jun 2016 21:40:15 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by mail-vk0-x236.google.com with SMTP id c189so89841018vkb.1 for ; Thu, 02 Jun 2016 14:40:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=DvKftkQAt7gxZAoZ8424zrGzSDAi+842xlts2GRdlXw=; b=NYPb1Wird2OJGoVS+PcBrDcKclu/Hn0kQdin6AzH15JEiQs8aAFjUhwP5oYpGmD1mK xMPCj7gbbnnWgOH9H1Vki9aP936HUz//hb+1mTdtrTP/vtUfFCzTygRPhBErCDlBbI0C SUqE8Yiq6ck5CLM8UvvZ0VcRcc2oNwyC+QdRiceplFbG+y9gliqCER3tfSZxFbtA0hDc O1t3P/sxucI+Csh++o505u973oDUA9AJGmnEPAOEffbcVtAu4tbZqc8ll04aYvkfQIeT 28ZY3MxiCpxcvnV5unJtHo+WpU64Dt7ySVgLLMCQIQJf6HicBn4BJoYBY8EQNqql1iio wxWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=DvKftkQAt7gxZAoZ8424zrGzSDAi+842xlts2GRdlXw=; b=O97vwGVFCqJry9IeJS8OglVZdYHAmm/n8c0zoG0xEacOZt45bZmYCKTNEd4XJd7NXT F4G9CIZOusfEuHNkEJMtH/EUhIU7fZGHdEKU1rQCxXh5aJgki5Hv4Rl7W+ZEk/t2+29o N35ABc9QQ0Bp4yKszOWKZ8K+InvgNzjmFtFjkE//1bNOF7CKF5rJHsIuVgVDv9GbWEMw 3E9KQfuA3HRUCKY8SgITt4fQ1buEI63ZUUisLPRwshuMqayxty8cIj7YB+2ClEyMb31D esRuHjFFObs3bZ/bCR3OaIqRoOEpRc1sN8L9ssF6sjUVHaZ8dTb/QsgErx7EBlleaqkE dNqQ== X-Gm-Message-State: ALyK8tJ2vS6yPrE3fZvkGnpc9UlMVqFhYldxUk9xsQTYWz5hgL/kK/GtLflmtik0oRTO9uDllSQclWOoyGj4VQ== MIME-Version: 1.0 X-Received: by 10.31.237.129 with SMTP id l123mr127801vkh.110.1464903614901; Thu, 02 Jun 2016 14:40:14 -0700 (PDT) Received: by 10.31.171.145 with HTTP; Thu, 2 Jun 2016 14:40:14 -0700 (PDT) In-Reply-To: <9A903EE5-3F2C-46C0-B563-1150F81E3507@juniper.net> References: <20160602202015.GG8994@strugglingcoder.info> <9A903EE5-3F2C-46C0-B563-1150F81E3507@juniper.net> Date: Thu, 2 Jun 2016 14:40:14 -0700 Message-ID: Subject: Re: Possible transmit/stats problem in igb driver. From: Jack Vogel To: Sreekanth Rupavatharam Cc: hiren panchasara , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 21:40:16 -0000 Hmmm, so I no longer have access to any of this, but I'm wondering if there is some other stat that might report errors or some sort that would match the missing/bad packets? Maybe they were considered runts or something and dropped. Good luck :) Jack On Thu, Jun 2, 2016 at 2:11 PM, Sreekanth Rupavatharam wrote: > Inline > > >Apart from stats, do you see anything else going wrong? i.e. do you > >actually see less packets (arp replies??) than expected? > > [SR] The packets are not going out on the wire. The tool doesn=E2=80=99t = receive > the packets. That=E2=80=99s how I started noticing the issue. > > >Taking your example, tx_packets is something we count in the drivers and > >total_pkts_txd is calculated in the card and we just read it off of it > >to report (E1000_TPT). > > [SR] Correct. My main question would be under what circumstance would the > packet handed off to hardware will *not* be transmitted?. Especially > considering there are no transmit errors or pause frames received. There > are no dma tx failures either. That=E2=80=99s the baffling part. I tried = another > exercise where I used ping of various sizes going out, but that doesn=E2= =80=99t > seem to trigger the problem. > > > >To understand your setup better, ixia is the sender and your box with > >igb(4) is the receiver and your are sending arp requests to it. > > Yes, correct. > > >Can you post following for working (size <=3D 64bytes) and non-working > >(size > 64bytes) cases for before/after? > > > >sysctl dev.igb | grep tx_packets > >sysctl dev.igb | grep total_pkts_txd > >sysctl dev.igb | grep rx_packets > >sysctl dev.igb | grep total_pkts_recvd > > > Before(not working): > dev.igb.1.queue0.tx_packets: 24907933 > dev.igb.1.queue0.rx_packets: 18086575 > dev.igb.1.mac_stats.total_pkts_recvd: 25057359 > dev.igb.1.mac_stats.total_pkts_txd: 16647169 > > After(not working): > dev.igb.1.queue0.tx_packets: 24913324 > dev.igb.1.queue0.rx_packets: 18091832 > dev.igb.1.mac_stats.total_pkts_recvd: 25062618 > dev.igb.1.mac_stats.total_pkts_txd: 16647545 > >netstat -sp arp > > The difference is 5391 for queue0.tx_packets but for > mac_stats.total_pkts_txd is 376 > Everything else is matching up. > > Before (working) > dev.igb.1.queue0.tx_packets: 25359165 > dev.igb.1.queue0.rx_packets: 18526094 > dev.igb.1.mac_stats.total_pkts_recvd: 25508763 > dev.igb.1.mac_stats.total_pkts_txd: 16831587 > > > After(working) > dev.igb.1.queue0.tx_packets: 25364597 > dev.igb.1.queue0.rx_packets: 18531398 > dev.igb.1.mac_stats.total_pkts_recvd: 25514009 > dev.igb.1.mac_stats.total_pkts_txd: 16836833 > > > Another interesting stat is > before_notworking:dev.igb.1.interrupts.tx_queue_empty: 16646890 > after_notworking:dev.igb.1.interrupts.tx_queue_empty: 16647266 > > The difference here is exactly 376 which is the number of packets that th= e > device actually claims to have transmitted. It=E2=80=99s as though it did= n=E2=80=99t see > the other packets en-queued in the ring descriptor. > > > I can=E2=80=99t do netstat just for arp as these are coming in a tunnel(P= ackets > don=E2=80=99t=E2=80=99 show up as arp on the interface). However, I did s= ee the packet rate > was about 500 packets/sec > > > >(You can netstat -z to clear the counters between runs.) > > > >Cheers, > >Hiren > >