From owner-freebsd-net@FreeBSD.ORG Wed Oct 30 06:41:13 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EE720F6B; Wed, 30 Oct 2013 06:41:13 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AB73D27E2; Wed, 30 Oct 2013 06:41:13 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r9U6f6WC024909 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Oct 2013 23:41:07 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r9U6f502024907; Tue, 29 Oct 2013 23:41:05 -0700 (PDT) (envelope-from jmg) Date: Tue, 29 Oct 2013 23:41:05 -0700 From: John-Mark Gurney To: Andre Oppermann Subject: Re: MQ Patch. Message-ID: <20131030064105.GV58155@funkthat.com> Mail-Followup-To: Andre Oppermann , Navdeep Parhar , Luigi Rizzo , Randall Stewart , "freebsd-net@freebsd.org" References: <40948D79-E890-4360-A3F2-BEC34A389C7E@lakerest.net> <526FFED9.1070704@freebsd.org> <52701D8B.8050907@freebsd.org> <527022AC.4030502@FreeBSD.org> <527027CE.5040806@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <527027CE.5040806@freebsd.org> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 29 Oct 2013 23:41:07 -0700 (PDT) Cc: "freebsd-net@freebsd.org" , Luigi Rizzo , Navdeep Parhar , Randall Stewart X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Oct 2013 06:41:14 -0000 Andre Oppermann wrote this message on Tue, Oct 29, 2013 at 22:25 +0100: > b) the driver assigns the DMA rings to particular cores which by that, > through > a critnest++ can drive them lockless. The drivers (*if_transmit) will > look > up the core it got called on and push the traffic out on that DMA ring. > The problem is the actual upper stacks affinity which is not guaranteed. > This has to consequences: there may be reordering of packets of the same > flow because the protocols send function happens to be called from a > different core the second time. Or the drivers (*if_transmit) has to > switch to the right core to complete the transmit for this flow if the > upper stack migrated/bounced around. It is rather difficult to assure > full affinity from userspace down through the upper stack and then to > the driver. I'll point you to the paper: http://arxiv.org/abs/1106.0443 Please don't reorder packets. Binding TX queues to cores seems not very useful, sure you can do a lockless implementation, but is running the scheduler to change cpu's really cheaper than paying the cost of migrating the lock? I'll admit I haven't run benchmarks, but I doubt it. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."