From owner-freebsd-net@FreeBSD.ORG Sun Mar 23 20:28:03 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 935164ED; Sun, 23 Mar 2014 20:28:03 +0000 (UTC) Received: from mail-qc0-x230.google.com (mail-qc0-x230.google.com [IPv6:2607:f8b0:400d:c01::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3EB521A8; Sun, 23 Mar 2014 20:28:03 +0000 (UTC) Received: by mail-qc0-f176.google.com with SMTP id m20so5089501qcx.35 for ; Sun, 23 Mar 2014 13:28:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qbHzEIWIpWCmaxJEPh1IfwuPRbDQCqwWw7GI0EKs0Wc=; b=pR/nows16+sudSch+BsEWHLEofs+C0k845zrPSlUjqtMcJCNEkFeVyF+EDhUAz1jB5 zYxcitkepTuuWBPwLMz1zd05wAoFFObwewjEO/4yFHh/DLDSDIAVJ6BiAZCvzorvP3Hb E8Kg+NpgJuAJY4r/G15r9voT8QEk8RZ25cRIZDlcjFx2knFV/AzEHbrOQv80UZoy32vu pl9/q8yQa73VsJFogP6jCIvyoJOjFcvMjqG5rn6CDSIt/FTyJ3FTTu53+WavbcU8Fk/u WCYs40OQ8bbnOAqZWRliZ/039b05evIdn7tg5+Dy3m83XRl0UtCDZWTj+UkLbZgDPbX/ xG0w== MIME-Version: 1.0 X-Received: by 10.140.90.80 with SMTP id w74mr2664956qgd.96.1395606482497; Sun, 23 Mar 2014 13:28:02 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Sun, 23 Mar 2014 13:28:02 -0700 (PDT) In-Reply-To: References: <1532337187.1452820.1395543491196.JavaMail.root@uoguelph.ca> Date: Sun, 23 Mar 2014 17:28:02 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Garrett Wollman , Jack Vogel , Markus Gebert X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2014 20:28:03 -0000 Update: For giggles, I set IP_MAXPACKET = 32768. Over a hour of runtime, and no issues. This is better than with the TSO patch and the 9.2 ixgbe, as that was just a drastic reduction in errors. Still have an 'angry' netstat -m on boot, and I'm still incrementing denied netbuf calls, so something else is wrong. I'm going to modify Rick's prinft in ixgbe to also output when we're over 32768. I'm sure it's still happening, but with an extra 32k of space, we're not busting like we did before. I notice a few interesting ip->ip_len changes since 9.2 - Like here, at line 720 http://fxr.watson.org/fxr/diff/netinet/ip_output.c?v=FREEBSD10;im=kwqeqdhhvovqn;diffval=FREEBSD92;diffvar=v Looks like older code didn't byteswap with ntohs - I see that often in tcp_output.c, and in tcp_options.c. I'm also curious about this:Line 524 http://fxr.watson.org/fxr/diff/netinet/ip_options.c?v=FREEBSD10;diffval=FREEBSD92;diffvar=v New 10 code: ip ->ip_len = htons (ntohs (ip ->ip_len) + optlen); Old 9.2 Code: ip ->ip_len += optlen; I wonder if there are any unexpected consequences of these changes, or perhaps a line someplace that doesn't make the change. Is there a dtrace command I could use to watch these functions and compare the new ip_len with ip->ip_len or other variables? On Sun, Mar 23, 2014 at 12:25 PM, Christopher Forgeron wrote: > > On Sat, Mar 22, 2014 at 11:58 PM, Rick Macklem wrote: > >> Christopher Forgeron wrote: >> > >> >> > Also should we not also subtract ETHER_VLAN_ENCAP_LEN from tsomax to >> > make sure VLANs fit? >> > >> I took a look and, yes, this does seem to be needed. It will only be >> needed for the case where a vlan is in use and hwtagging is disabled, >> if I read the code correctly. >> > > Yes, or in the rare care where you configure your switch to pass the v_lan > header through to the NIC. > >> >> Do you use vlans? >> > > (Answered in above email) > > >> >> I've attached an updated patch. >> >> It might be nice to have the printf() patch in the driver too, so >> we can see how big the ones that are too big are? >> >> > Yes, I'm going to leave those in until I know we have this fixed.. will > probably leave it in a while longer as it should only have a minor > performance impact to iter-loop like that, and I'd like to see what the > story is a few months down the road. > > Thanks for the patches, will have to start giving them code-names so we > can keep them straight. :-) I guess we have printf, tsomax, and this one. >