From owner-freebsd-net@FreeBSD.ORG Sun Sep 1 20:51:20 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C7AD0B7D; Sun, 1 Sep 2013 20:51:20 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A24532515; Sun, 1 Sep 2013 20:51:19 +0000 (UTC) Received: by mail-wi0-f171.google.com with SMTP id hq12so1179379wib.10 for ; Sun, 01 Sep 2013 13:51:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=qwKV+wetXtX2L8fzprJcef2IwG9zoUZn6GZq4SNa1tk=; b=Fcxk7pwlrG4uXUWM1ZImr69Za9unx+vvRarWaGCABKOp11TvATZPIKU/a2bW3mu2cs Q+tFdYMjjt+ojSYFDmXieA2e0vzamEGGSX90xdHLeR09RzsbEh/Aw1NOqKdaQ+gakAVz RyHlkyavj46FhC8xPcXWVNty0qsolnqnqrj0SRrKo9kxNNb5pc3rz1atKI7+biazVrbe MiB1BE3v98z/cIh2i9QgnKAKs9ZY7T2FrYA8xywKPnQznTAwZ+BDM+MWDalDKOfQdNz/ Hj/JW3Yy+0fRzPjLC5SECkjjCseR6ptHXrBSXdzFWvnXxvCsu/b8Odx2ln7xD5S+wPqZ Q1qg== MIME-Version: 1.0 X-Received: by 10.180.37.164 with SMTP id z4mr10937379wij.30.1378068677832; Sun, 01 Sep 2013 13:51:17 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.146.2 with HTTP; Sun, 1 Sep 2013 13:51:17 -0700 (PDT) In-Reply-To: <1378050319.62710.YahooMailNeo@web121601.mail.ne1.yahoo.com> References: <521BBD21.4070304@freebsd.org> <521EE8DA.3060107@freebsd.org> <1377952913.44129.YahooMailNeo@web121605.mail.ne1.yahoo.com> <1378001733.36695.YahooMailNeo@web121606.mail.ne1.yahoo.com> <1378050319.62710.YahooMailNeo@web121601.mail.ne1.yahoo.com> Date: Sun, 1 Sep 2013 13:51:17 -0700 X-Google-Sender-Auth: Dsg0NSrMPg0YIKpmRWgs3S4mwxM Message-ID: Subject: Re: Flow ID, LACP, and igb From: Adrian Chadd To: Barney Cordoba Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Andre Oppermann , Alan Somers , "net@freebsd.org" , Jack F Vogel , "Justin T. Gibbs" , Luigi Rizzo , "T.C. Gubatayao" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Sep 2013 20:51:20 -0000 Yo, LRO is an interesting hack that seems to do a good trick of hiding the ridiculous locking and unfriendly cache behaviour that we do per-packet. It helps with LAN test traffic where things are going out in batches from the TCP layer so the RX layer "sees" these frames in-order and can do LRO. When you disable it, I don't easily get 10GE LAN TCP performance. That has to be fixed. Given how fast the CPU cores, bus interconnect and memory interconnects are, I don't think there should be any reason why we can't hit 10GE traffic on a LAN with LRO disabled (in both software and hardware.) Now that I have the PMC sandy bridge stuff working right (but no PEBS, I have to talk to Intel about that in a bit more detail before I think about hacking that in) we can get actual live information about this stuff. But the last time I looked, there's just too much per-packet latency going on. The root cause looks like it's a toss up between scheduling, locking and just lots of code running to completion per-frame. As I said, that all has to die somehow. 2c, -adrian On 1 September 2013 08:45, Barney Cordoba wrote: > > > Comcast sends packets OOO. With any decent number of internet hops you're > likely to encounter a load > balancer or packet shaper that sends packets OOO, so you just can't be > worried about it. In fact, your > designs MUST work with OOO packets. > > Getting balance on your load balanced lines is certainly a bigger upside > than the additional CPU used. > You can buy a faster processor for your "stack" for a lot less than you > can buy bandwidth. > > Frankly my opinion of LRO is that it's a science project suitable for labs > only. It's a trick to get more bandwidth > than your bus capacity; the answer is to not run PCIe2 if you need pcie3. > You can use it internally if you have > control of all of the machines. When I modify a driver the first thing > that I do is rip it out. > > BC > > > ________________________________ > From: Luigi Rizzo > To: Barney Cordoba > Cc: Andre Oppermann ; Alan Somers ; > "net@freebsd.org" ; Jack F Vogel ; > Justin T. Gibbs ; T.C. Gubatayao < > tgubatayao@barracuda.com> > Sent: Saturday, August 31, 2013 10:27 PM > Subject: Re: Flow ID, LACP, and igb > > > On Sun, Sep 1, 2013 at 4:15 AM, Barney Cordoba >wrote: > > > ... > > > > [your point on testing with realistic assumptions is surely a valid one] > > > > > > Of course there's nothing really wrong with OOO packets. We had this > > discussion before; lots of people > > have round robin dual homing without any ill effects. It's just not an > > issue. > > > > It depends on where you are. > It may not be an issue if the reordering is not large enough to > trigger retransmissions, but even then it is annoying as it causes > more work in the endpoint -- it prevents LRO from working, and even > on the host stack it takes more work to sort where an out of order > segment goes than appending an in-order one to the socket buffer. > > cheers > luigi > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >