From owner-freebsd-net@FreeBSD.ORG Wed Oct 30 11:51:10 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B27ED5AB for ; Wed, 30 Oct 2013 11:51:10 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2283C2A4C for ; Wed, 30 Oct 2013 11:51:09 +0000 (UTC) Received: (qmail 61485 invoked from network); 30 Oct 2013 12:21:32 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 30 Oct 2013 12:21:32 -0000 Message-ID: <5270F297.4090001@freebsd.org> Date: Wed, 30 Oct 2013 12:50:47 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: Navdeep Parhar , Luigi Rizzo , Randall Stewart , "freebsd-net@freebsd.org" Subject: Re: MQ Patch. References: <40948D79-E890-4360-A3F2-BEC34A389C7E@lakerest.net> <526FFED9.1070704@freebsd.org> <52701D8B.8050907@freebsd.org> <527022AC.4030502@FreeBSD.org> <527027CE.5040806@freebsd.org> <20131030064105.GV58155@funkthat.com> In-Reply-To: <20131030064105.GV58155@funkthat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Oct 2013 11:51:10 -0000 On 30.10.2013 07:41, John-Mark Gurney wrote: > Andre Oppermann wrote this message on Tue, Oct 29, 2013 at 22:25 +0100: >> b) the driver assigns the DMA rings to particular cores which by that, >> through >> a critnest++ can drive them lockless. The drivers (*if_transmit) will >> look >> up the core it got called on and push the traffic out on that DMA ring. >> The problem is the actual upper stacks affinity which is not guaranteed. >> This has to consequences: there may be reordering of packets of the same >> flow because the protocols send function happens to be called from a >> different core the second time. Or the drivers (*if_transmit) has to >> switch to the right core to complete the transmit for this flow if the >> upper stack migrated/bounced around. It is rather difficult to assure >> full affinity from userspace down through the upper stack and then to >> the driver. > > I'll point you to the paper: > http://arxiv.org/abs/1106.0443 > > Please don't reorder packets. > > Binding TX queues to cores seems not very useful, sure you can do a > lockless implementation, but is running the scheduler to change cpu's > really cheaper than paying the cost of migrating the lock? > > I'll admit I haven't run benchmarks, but I doubt it. Don't worry. My list was about the possible ways of dealing with it and their constrains/disadvantage. Packet reordering is one part of it that pretty much makes approach b) non-viable as you correctly point out. -- Andre