From owner-freebsd-net@freebsd.org Thu Jul 21 19:39:41 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 536FBBA12C3 for ; Thu, 21 Jul 2016 19:39:41 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2AF9F1D0A; Thu, 21 Jul 2016 19:39:41 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.200.3] (c-73-147-115-187.hsd1.va.comcast.net [73.147.115.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id 0BBB0F80530; Thu, 21 Jul 2016 15:31:48 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail; t=1469129509; bh=NHyvsN2G6ymlfY0Yw6oDeD9rtITCbHUH10LjDpR7Ohs=; h=Subject:To:From:Date; b=fQ64nyPiF2ERbAZcJquTXkUM6b7tSBOkOFYqIImH7SUyQVkG8XFnruL9t5R5O0FpM vDJF0rq1gmznu36M+IK0y27ZIRjEkbkj4C6tyNEaKs/2JlIcvsoRFKhKOHrkI+qk3y F6VglJ/mdlyzlN48JNjqcbyZIMyVAf1nBi9pbs++OGA7fnnhtQuhZZBXbRNXQa/2Zs Oh5PWlpd5rtbXboMDmSegyCOCshXap1pO277kDdlrNJeU9n0Cr6QmVjzhDrOw0adaR HgiIQm5xeWKaFwQ5wO8UuCtlv9yA+gFuuswn1UPu05qddzZ4W9yJ1bfikZvzFQxnkL e/0ZweZnR9X+A== Subject: Re: proposal: splitting NIC RSS up from stack RSS To: Adrian Chadd , FreeBSD Net References: From: Andrew Gallatin Message-ID: <306af514-70ff-f3bf-5b4f-da7ac1ec6580@cs.duke.edu> Date: Thu, 21 Jul 2016 15:31:46 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 19:39:41 -0000 On 07/14/2016 16:06, Adrian Chadd wrote: > I'd appreciate any other feedback/comments/suggestions. If you're > using RSS and you haven't told me then please let me know! Hi Adrian, I'm a huge fan of your RSS work. In fact, I did a backport of RSS to Netflix's stable-10 about 6 months ago. I was really interesting in breaking up the global network hashtables, where we see a lot of contention. PCBGROUP didn't help much (just spread contention around), so I was hoping that RSS would be the magic bullet. Things may have progressed since then, but the real deficiencies that I saw were: o RSS (at the time) would only use a power-of-two number of cores. Sadly, Intel and AMD are building lots of chips with oddball core counts. So in a workload like ours where most work is initiated via the NIC rx ithread, having a 14-core machine meant leaving almost 1/2 the machine mostly idle, while 8 cores were maxed out. o There is (or was at the time) no library interface for RSS, and no patches for popular web servers (like nginx) to use RSS. The only example RSS users I could find were a few things from your blog. These 2 things lead me to abandon the backport, as I didn't have time to address them on top of other work I was doing. I especially think getting a real API and a real example consumer would help a lot. Best regards, Drew