From owner-freebsd-net@FreeBSD.ORG Mon Mar 24 22:47:48 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7ADB6D3F; Mon, 24 Mar 2014 22:47:48 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 2C1605F3; Mon, 24 Mar 2014 22:47:47 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqUEALC1MFODaFve/2dsb2JhbABZg0FXgwe4NYZkUYE4dIIlAQEBAwEBAQEgBCcgCwUWDgoCAg0ZAiMGAQkmBggHBAEcBIdEAwkIDaxVmxANhzIXgSmLNIFMAQEbNAeCb4FJBJVzaoMfizaFSYNJITGBBDk X-IronPort-AV: E=Sophos;i="4.97,723,1389762000"; d="scan'208";a="108513058" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 24 Mar 2014 18:47:13 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 31360B403E; Mon, 24 Mar 2014 18:47:13 -0400 (EDT) Date: Mon, 24 Mar 2014 18:47:13 -0400 (EDT) From: Rick Macklem To: Christopher Forgeron Message-ID: <1149589960.2454997.1395701233190.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: 9.2 ixgbe tx queue hang MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: FreeBSD Net , Garrett Wollman , Jack Vogel , Markus Gebert X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 22:47:48 -0000 Christopher Forgeron wrote: > I'm going to split this into different posts to focus on each topic. > This > is about setting IP_MAXPACKET to 65495 > > Update on Last Night's Run: > > (Last night's run is a kernel with IP_MAXPACKET = 65495) > > - Uptime on this run: 10:53AM up 13:21, 5 users, load averages: > 1.98, > 2.09, 2.13 > - Ping logger records no ping errors for the entire run. > - At Mar 24th 10:57 I did a grep through the night's log for 'before' > (which is the printf logging that Rick suggested a few days ago), and > saved > it to before_total.txt > - With wc -l on before_total.txt I can see that we have 504 lines, > thus 504 > incidents of the packet being above IP_MAXPACKET during this run. > - I did tr -c '[:alnum:]' '[\n*]' < before_total.txt | sort | uniq -c > | > sort -nr | head -50 to list the most common words. Ignoring the > non-pklen > output. The relevant output is: > > 344 65498 (3) > 330 65506 (11) > 330 65502 (7) > This makes sense to me, since tp->t_tsomax is used in tcp_output() for the TCP/IP packet, which does not include the link level (ethernet) header. When that is added, I would expect the length to be up to 14 (or maybe 18 for vlan cases) greater than IP_MAXPACKET. Since none of these are greater than 65509, this looks fine to me. So, unless you get ones greater than (65495 + 18 = 65513), this makes sense and does not indicate a problem. In another post, you indicate that having the driver set if_hw_tsomax didn't set tp->t_tsomax to the same value. --> I believe that is a bug and would mean my ixgbe.patch would not fix the problem, because it is tp->t_tsomax that must be decreased to at least (65536 - 18 = 65518). --> Now, have you tried a case between 65495 and 65518 and seen any EFBIG errors? If so, then I don't understand why 65518 isn't small enough? rick > - First # being the # of times. (Each pklen is printed twice on the > log, > thus 2x the total line count). > - Last (#) being the byte overrun from 65495 > - A fairly even distribution of each type of packet overrun. > > You will recall that my IP_MAXPACKET is 65495, so each of these > packet > lengths represents a overshoot. > > The fact that we have only 3 different types of overrun is good - It > suggests a non-random event, more like a broken 'if' statement for a > particular case. > I think it just means that your load happens to do only 3 sizes of I/O that is a little less than 65536. > If IP_MAXPACKET was set to 65535 as it normally is, I would have had > 504 > incidents of errors, with a chance that any one of them could have > blocked > the queue for considerable time. > If tp->t_tsomax hasn't been set to a smaller value than 65535, the ixgbe.patch didn't do what I thought it would. > Question: Should there be logic that discards packets that are over > IP_MAXPACKET to ensure that we don't end up in a blocked queue > situation > again? > > > Moving forward, I am doing two things: > > 1) I'm running a longer test with TSO disabled on my ix0 adapter. I > want > to make sure that over say 4 hours I don't have even 1 packet over > 65495. > This will at least locate the issue to TSO related code. > > 2) I have tcpdump running, to see if I can capture the packets over > 65495. > Here is my command. Any suggestions on additional switches I should > include? > > tcpdump -ennvvXS greater 65495 > > I'll report in on this again once I have new info. > > Thanks for reading. > > On Mon, Mar 24, 2014 at 2:14 AM, Christopher Forgeron > wrote: > > > Hi, > > > > I'll follow up more tomorrow, as it's late and I don't have time > > for > > detail. > > > > The basic TSO patch didn't work, as packets were were still going > > over > > 65535 by a fair amount. I thought I wrote that earlier, but I am > > dumping a > > lot of info into a few threads, so I apologize if I'm not as > > concise as I > > could be. > > > > However, setting IP_MAXPACKET did. 4 hours of continuous run-time, > > no > > issues. No lost pings, no issues. Of course this isn't a fix - but > > it helps > > isolate the problem. > > > what the story is a few months down the road. > > > > > > > > > Thanks for the patches, will have to start giving them code-names > > > so > > > we can keep them straight. :-) I guess we have printf, tsomax, > > > and > > > this one. > > > > > > > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >