From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 18:02:20 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B717A7B5; Tue, 1 Jul 2014 18:02:20 +0000 (UTC) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6C4752D3C; Tue, 1 Jul 2014 18:02:20 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id v10so7883558qac.41 for ; Tue, 01 Jul 2014 11:02:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=k29Cqf3KrMEaMdlBk4kIeGaY3fzC2J0rdhfzdWPmX5w=; b=QtawPZXlnexi02LUPnKYHm6PhWCS1Z2ChdfG3knFSLDgXvjTWV1J45B+pClaoExHCo hBY3Pg2nkKx1xbaPQhIucshb5lWZOsYyK9Ko4S+b7qOrA+EJrTRGuNz7Kjnn+TFPi2Oi ZamKzKTde3nAtYQPOsG/52OMmExs6gTTM1WNxvMRwcfQykKu/IhTIk5AqRdxkCgbjrkk PXecvf6Ycr0aVXVe+AOYy6i2PIc4YGOxl4VAT75oeR4O7VZojKxpxJeSZ3Y7hTdnkEGF DBFn1g03+AvP82sdpyWklK8TCmHGfPmHd0Fqahl/LhZuS+QpNWLL6F6ZpFvOcVF0rxNr JOXw== MIME-Version: 1.0 X-Received: by 10.140.34.195 with SMTP id l61mr71334451qgl.87.1404237739505; Tue, 01 Jul 2014 11:02:19 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.202.193 with HTTP; Tue, 1 Jul 2014 11:02:19 -0700 (PDT) Date: Tue, 1 Jul 2014 11:02:19 -0700 X-Google-Sender-Auth: jit6IfaJ6F7YjVvAu1q_Vw0i7Nc Message-ID: Subject: [rfc] IP_BINDMULTI and IP_RSSBUCKETID support From: Adrian Chadd To: FreeBSD Net , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2014 18:02:20 -0000 Hi! This patch: http://people.freebsd.org/~adrian/rss/20140701-rss-1.diff Implements a few things: * It introduces IP_BINDMULTI, which allows TCP/UDP sockets to be bound to the same matching address:port. It's intended to be what Linux SO_REUSEADDR has become except I specifically wanted it to be a different option. * .. it only allows the PCBs to be created, it doesn't actually load balance between multiple sockets like this. That requires a bunch more work to happen in the general case. But! * IP_RSSBUCKETID maps an IPv4 TCP (and later UDP, and much later IPv6) socket into a specific RSS bucket rather than in the global PCB list. That way it only receives incoming connections that has to that RSS bucket ID. If userland queries the RSS setup (sysctl net.inet.rss) and creates a worker thread per RSS bucket _AND_ pins it to the correct CPU for the RSS bucket, all the transmit and receive side will stay on the same CPU. The lock contention is almost eliminated in that case. My plan: * get this into the tree and tested/debugged; * add IPv4 UDP extensions to RSS; * teach a couple of popular threaded UDP applications (eg memcached) about this; * bask in what hopefully will be a good set of benefits; * work on IP_BINDMULTI support so multiple threads/processes can listen in the same RSS bucket and get some load balancing; * add IPv6 support; * add RSS rebalancing support; * add multi-socket, multi-NIC RSS awareness. Thanks! -a