From owner-freebsd-hackers@FreeBSD.ORG Mon Jun 9 17:54:13 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9DAA5360 for ; Mon, 9 Jun 2014 17:54:13 +0000 (UTC) Received: from mail-qc0-x232.google.com (mail-qc0-x232.google.com [IPv6:2607:f8b0:400d:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C1692038 for ; Mon, 9 Jun 2014 17:54:13 +0000 (UTC) Received: by mail-qc0-f178.google.com with SMTP id c9so1013729qcz.23 for ; Mon, 09 Jun 2014 10:54:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=nku/9JivlqaJokhIkwTFbBp0Kh8eM/iraziJhXwJe1M=; b=WYT9YBt8T9tckx15qeiyxZaurf11Ph8Pt338OB74wXwBOU1s5jO7z9hzwqJSQQrfLI 7Al04+pT35uwTDaNlkOIoLbJ/UirqvIA+vaOwDaZ96khezIMWsWGAYvCg6mfMfdl+V5W mNe5/Ox3PECilLmh11Y0Ii7xjKoKxPYtc4f7H1eUqu4WHoX1wyIV60Oe2vRF9zyMLVnt ohctdMRQGjfdb5JEVscUVhwrTc6Qu0CKfBmRcc/wgyctFByTJTkQhpRHBBLJf09kaK1P DdtNPD3/8IEv+UK4Feiw7eJlbYAh8GslTd7QBrGGm+sSqJrEdHAYCLk9GE66n+k8wIaf CjEA== MIME-Version: 1.0 X-Received: by 10.140.91.5 with SMTP id y5mr33096889qgd.12.1402336452560; Mon, 09 Jun 2014 10:54:12 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.43.134 with HTTP; Mon, 9 Jun 2014 10:54:12 -0700 (PDT) In-Reply-To: <5395EAF6.4020505@gmail.com> References: <20140607010151.51751.qmail@f5-external.bushwire.net> <5395EAF6.4020505@gmail.com> Date: Mon, 9 Jun 2014 13:54:12 -0400 X-Google-Sender-Auth: XOjObkLWlldd7s-Hpx_GEBTw0jc Message-ID: Subject: Re: Best practice for accepting TCP connections on multicore? From: Adrian Chadd To: Hooman Fazaeli Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , Mark Delany X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2014 17:54:13 -0000 freebsd-rss is not freebsd; it's just the evaluation code Ive' written. 'master' is the branch there. -a On 9 June 2014 13:12, Hooman Fazaeli wrote: > On 6/7/2014 6:32 AM, Adrian Chadd wrote: >> >> Hi, >> >> In the RSS model I'm hacking on: >> >> * you query the kernel for the current RSS bucket -> CPU mapping. >> Right now it's one bucket -> one CPU but at some point it may be one >> bucket -> cpuset; >> * you spawn a thread for each RSS bucket; >> * you pin each thread to the RSS bucket current CPU id; >> * you create a listen socket in that thread, marked as BINDMULTI (ie, >> multiple things can listen on this and the kernel will load balance >> between them) and RSS_BUCKET (ie, please place this socket in the >> given RSS bucket, rather than the global/wildcard list); >> * then when a connection comes in, the kernel will first do a lookup >> for a matching wildcard socket in the per-RSS PCBGROUP, rather than >> the global wildcard table; >> * if it finds it, that socket gets the incoming connection. >> >> At some point I'll add some notification via kqueue or what not that >> the RSS buckets need rebalancing, and userland can then re-pin the >> per-bucket threads. >> >> At the moment the hacks I've done only support one listen socket per >> entry. My hope is that BINDMULTI will do some basic hash to load >> balance within a set of matching PCB entries - and it'll be combined, >> so if you do BINDMULTI without RSS, it'll just load balance between >> multiple sockets with no CPU affinity knowledge. If you do RSS, it'll >> distribute only CPU-local requests to a thread that's sitting in the >> right RSS bucket. if you enable both, you can use a thread pool for >> each RSS bucket CPU and (eventually, when I write it) it'll load >> balance among those. >> >> But for now I'm assuming one incoming thread per RSS bucket will be >> enough for people to experiment with. >> >> anyway, I guess I should email out the details: >> >> * http://github.com/erikarn/freebsd - the 'local/rss' branch has the >> RSS changes to dev/e1000/if_igb.c and netinet/ > > would you please point to the exact URL for the branch? > >> * http://github.com/erikarn/freebsd-rss - has some RSS examples. Look >> at rss-http. >> >> I haven't yet tested this at > 1GE because all I have at home are igb >> and em NICs. If someone would like to donate ixgbe and T4 hardware, >> i'll gratefully take it and do up RSS patches for those drivers. >> >> > > -- > > Best regards. > Hooman Fazaeli >