From owner-freebsd-hackers@FreeBSD.ORG Mon Jun 9 17:12:32 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 639064A6; Mon, 9 Jun 2014 17:12:32 +0000 (UTC) Received: from mail-we0-x22d.google.com (mail-we0-x22d.google.com [IPv6:2a00:1450:400c:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C73262C39; Mon, 9 Jun 2014 17:12:31 +0000 (UTC) Received: by mail-we0-f173.google.com with SMTP id u57so6212154wes.32 for ; Mon, 09 Jun 2014 10:12:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=ntX8SivLTjWdASJmFoQwlryaw1KMR4vE+MNXaOItCdo=; b=u77YhYm8rl1hFh2UVsMx3M2UIpRFete/iWhmTWGgi+IbFNLZyfcwBIR3hRcvhFUkYK MtlDNPF39pgHbVuR7+ZOnjXCx40f76VHcVcNfkgAjMlrdxpXoaSECfzb9KgLv7hUgzly OS9yxjnkw1SGqU4SRnm5s1L8DU0/Qb235cENat+C8Y8lyGMo0jXY1NjJ3iBNAOv3xxG9 qMHv8+A67xmJyyKwqqdkY7prjXBFHxh/ZiTXORk4RxYzitjZHE46beHXr1YFgcQ8IHRO vxxbiAcoH4TZCq3F2FL+zeXxZK0+wt/LWGvxwJYx6I+7AwDhect6LAQ5c888SLVsYUex Na9g== X-Received: by 10.180.221.163 with SMTP id qf3mr32860447wic.56.1402333950010; Mon, 09 Jun 2014 10:12:30 -0700 (PDT) Received: from [192.168.2.30] ([2.176.208.29]) by mx.google.com with ESMTPSA id do5sm15762427wib.16.2014.06.09.10.12.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 09 Jun 2014 10:12:29 -0700 (PDT) Message-ID: <5395EAF6.4020505@gmail.com> Date: Mon, 09 Jun 2014 21:42:22 +0430 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130215 Thunderbird/17.0.3 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: Best practice for accepting TCP connections on multicore? References: <20140607010151.51751.qmail@f5-external.bushwire.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Mark Delany X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2014 17:12:32 -0000 On 6/7/2014 6:32 AM, Adrian Chadd wrote: > Hi, > > In the RSS model I'm hacking on: > > * you query the kernel for the current RSS bucket -> CPU mapping. > Right now it's one bucket -> one CPU but at some point it may be one > bucket -> cpuset; > * you spawn a thread for each RSS bucket; > * you pin each thread to the RSS bucket current CPU id; > * you create a listen socket in that thread, marked as BINDMULTI (ie, > multiple things can listen on this and the kernel will load balance > between them) and RSS_BUCKET (ie, please place this socket in the > given RSS bucket, rather than the global/wildcard list); > * then when a connection comes in, the kernel will first do a lookup > for a matching wildcard socket in the per-RSS PCBGROUP, rather than > the global wildcard table; > * if it finds it, that socket gets the incoming connection. > > At some point I'll add some notification via kqueue or what not that > the RSS buckets need rebalancing, and userland can then re-pin the > per-bucket threads. > > At the moment the hacks I've done only support one listen socket per > entry. My hope is that BINDMULTI will do some basic hash to load > balance within a set of matching PCB entries - and it'll be combined, > so if you do BINDMULTI without RSS, it'll just load balance between > multiple sockets with no CPU affinity knowledge. If you do RSS, it'll > distribute only CPU-local requests to a thread that's sitting in the > right RSS bucket. if you enable both, you can use a thread pool for > each RSS bucket CPU and (eventually, when I write it) it'll load > balance among those. > > But for now I'm assuming one incoming thread per RSS bucket will be > enough for people to experiment with. > > anyway, I guess I should email out the details: > > * http://github.com/erikarn/freebsd - the 'local/rss' branch has the > RSS changes to dev/e1000/if_igb.c and netinet/ would you please point to the exact URL for the branch? > * http://github.com/erikarn/freebsd-rss - has some RSS examples. Look > at rss-http. > > I haven't yet tested this at > 1GE because all I have at home are igb > and em NICs. If someone would like to donate ixgbe and T4 hardware, > i'll gratefully take it and do up RSS patches for those drivers. > > -- Best regards. Hooman Fazaeli