From owner-freebsd-net@FreeBSD.ORG Wed Mar 26 17:44:38 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 71ACA5C0; Wed, 26 Mar 2014 17:44:38 +0000 (UTC) Received: from mail-qa0-x234.google.com (mail-qa0-x234.google.com [IPv6:2607:f8b0:400d:c00::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1A79A1F8; Wed, 26 Mar 2014 17:44:38 +0000 (UTC) Received: by mail-qa0-f52.google.com with SMTP id m5so2621455qaj.11 for ; Wed, 26 Mar 2014 10:44:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bsA84drya82gxl/p3iPfMQUCsaPymhQTfIJ/DLpVZYo=; b=AH/eHKOf+rAzxppBLCw61u1316a85NLiFJpSTVobwc8Hk7Xg7J6DeIOaWGcJvQ6HKH 4RCQnxzlt9+FLVq3VXxXDyv87OQxciyjQwu5WajwuLlYzRaw5SocQvL2cy3HMNUuPN0m GXugddwvuV9CYHJu2U1e8JPb3aQenYmGDgPi6rtar+oeW0b1dpam4Abtv5ALwhgKITiJ hDtzMhB/jXokwYY/cIh/pVKoelcWHLy+g1m9kS7n5DVhz3IjABA5WtUFPU+mVRf8muTG npLqINHv30Jo15vIuf7PZGtehxSH8IZgMvpRLaRpF+IGj4swvn8McU7JsXBqkEtyyrPR iAhA== MIME-Version: 1.0 X-Received: by 10.140.48.199 with SMTP id o65mr51466713qga.16.1395855877261; Wed, 26 Mar 2014 10:44:37 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Wed, 26 Mar 2014 10:44:37 -0700 (PDT) In-Reply-To: References: <0BC10908-2081-45AC-A1C8-14220D81EC0A@hostpoint.ch> <1236110257.2510701.1395709458870.JavaMail.root@uoguelph.ca> <1197F2E5-F20C-43E4-B8C8-8732F45457C2@hostpoint.ch> Date: Wed, 26 Mar 2014 14:44:37 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Markus Gebert Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Rick Macklem , Garrett Wollman , Jack Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2014 17:44:38 -0000 Up for almost 19 hours under load without a single error. I would say the TSO patch does work, now I'm going to run lagg tests. The more I think of it, the more I wonder if setting tsomax in if.c at line 660 isn't the better idea, like below. 660: if (ifp->if_hw_tsomax == 0) 661: ifp->if_hw_tsomax = IP_MAXPACKET - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN); I know there are concerns about the impact on various cards, but right now if.c will set if_hw_tssomax to IP_MAXPACKET, which we know is bad for ixgbe, and I believe bad for lagg (tests will show) - If the driver isn't specifically setting it to a different setting, is there harm in limiting all if's to a default of IP_MAXPACKET - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN) if not specified otherwise? When is a TSO of 65535 going to be useful? I can confirm that with just the TSO patch in ixgbe, and lagg enabled, the problem still exists. Last night's tests never went above a packet of 65530. Now with lagg enabled, I'm seeing packets of 65543 within 5 minutes, so we're already breaking. On Tue, Mar 25, 2014 at 11:33 PM, Christopher Forgeron wrote: > > > On Tue, Mar 25, 2014 at 8:21 PM, Markus Gebert > wrote: > >> >> >> Is 65517 correct? With Ricks patch, I get this: >> >> dev.ix.0.hw_tsomax: 65518 >> > > Perhaps a difference between 9.2 and 10 for one of the macros? My code is: > > > ifp->if_hw_tsomax = IP_MAXPACKET - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN); > printf("CSF - 3 Init, ifp->if_hw_tsomax = %d\n", > ifp->if_hw_tsomax); > > (BTW, you should submit the hw_tsomax sysctl patch, that's useful to > others) > > >> Also the dtrace command you used excludes 65518... >> > > Oh, I thought it was giving every packet that is greater than or equal to > 65518 - Could you show me the proper command? That's the third time I've > used dtrace, so I'm making this up as I go. :-) >