From owner-freebsd-current@FreeBSD.ORG Mon Dec 2 21:41:16 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 99EE4AEA; Mon, 2 Dec 2013 21:41:16 +0000 (UTC) Received: from mail-qa0-x229.google.com (mail-qa0-x229.google.com [IPv6:2607:f8b0:400d:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1D1961AA9; Mon, 2 Dec 2013 21:41:16 +0000 (UTC) Received: by mail-qa0-f41.google.com with SMTP id j5so4920197qaq.7 for ; Mon, 02 Dec 2013 13:41:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=EC57TpugfYxzzijezRYWYhKJlDnMDzob9xIx5rcHmhs=; b=emc8FrTlknWZQkrwuFVDodUQvlkf+wFO+HbO9ZO3Z0V8X6HdP9tFP+4cxTnC7LfM0h 420sOCTDlKA4PcapacIL1YSq0vWQop8bNWo0AhSPq6z/oYCQu/3WUAPUOniJBsc7Gf/q yJL/7HKRqsX+hxUJZPpOb/e22bmjJx0+QGikNWUOXGUkvPxK4xFcnHCGeGnHhRVEWnNF gS/ph5trkDFhP9agRYvENpYjdG5FTj9V/HPqaYAqrz1uBYqkS6p0DiBFYzQUvwN1QrlS DhI54HdejTfLCGTmlSD+yQ43oj3/4v4iSfNCPnx8dspEUgLs3hDCV3bAx/uQacHwK5mH wW6A== MIME-Version: 1.0 X-Received: by 10.49.131.5 with SMTP id oi5mr76884665qeb.38.1386020474382; Mon, 02 Dec 2013 13:41:14 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.53.200 with HTTP; Mon, 2 Dec 2013 13:41:14 -0800 (PST) In-Reply-To: References: <4053E074-EDC5-49AB-91A7-E50ABE36602E@freebsd.org> Date: Mon, 2 Dec 2013 13:41:14 -0800 X-Google-Sender-Auth: nFocPVewXGPEdhU8tnGKtSD_dUk Message-ID: Subject: Re: [PATCH] SO_REUSEADDR and SO_REUSEPORT behaviour From: Adrian Chadd To: Sepherosa Ziehau Content-Type: text/plain; charset=ISO-8859-1 Cc: =?ISO-8859-1?Q?Ermal_Lu=E7i?= , freebsd-net , Oleg Moskalenko , Tim Kientzle , "freebsd-current@freebsd.org" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Dec 2013 21:41:16 -0000 On 2 December 2013 03:45, Sepherosa Ziehau wrote: > > On Mon, Dec 2, 2013 at 1:02 PM, Adrian Chadd wrote: > >> Ok, so given this, how do you guarantee the UTHREAD stays on the given >> CPU? You assume it stays on the CPU that the initial listen socket was >> created on, right? If it's migrated to another CPU core then the >> listen queue still stays in the original hash group that's in a netisr >> on a different CPU? > > As I wrote in the above brief introduction, Dfly currently relies on the > scheduler doing the proper thing (the scheduler does do a very good job > during my tests). I need to export certain kind of socket option to make > that information available to user space programs. Force UTHREAD binding in > kernel is not helpful, given in reverse proxy application, things are > different. And even if that kind of binding information was exported to > user space, user space program still would have to poll it periodically (in > Dfly at least), since other programs binding to the same addr/port could > come and go, which will cause reorganizing of the inp localgroup in the > current Dfly implementation. Right. I kinda gathered that. It's fine, I was conceptually thinking of doing some thead pinning into this anyway. How do you see this scaling on massively multi-core machines? Like 32, 48, 64, 128 cores? I had some vague handwav-y notion of maybe limiting the concept of pcbgroup hash / netisr threads to a subset of CPUs, or have them be able to float between sockets but only have 1 (or n, maybe) per socket. Or just have a fixed, smaller pool. The idea then is the scheduler would need to be told that a given userland thread/process belongs to a given netisr thread, and to schedule them on the same CPU when possible. Anyway, thanks for doing this work. I only wish that you'd do it for FreeBSD. :-) -adrian