From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 17:18:09 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AE29C783 for ; Tue, 23 Jul 2013 17:18:09 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-x22a.google.com (mail-we0-x22a.google.com [IPv6:2a00:1450:400c:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 426ED2A82 for ; Tue, 23 Jul 2013 17:18:09 +0000 (UTC) Received: by mail-we0-f170.google.com with SMTP id w60so1229836wes.15 for ; Tue, 23 Jul 2013 10:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=VhodCoUe6rFJASOWuGre4MLzwlm5UacbDteFmW79Z2M=; b=bxE6PIkGXpzRw2poak/tHJNzdRnrddsIhw2mfbGsMWYmystm3abbJaVz+QIjAIj/1D W8KG8rrqwM72zHqHG4N7w3PtifIG8IrTrxZ1UmmanRVm9NStAsPC6fjDRYZ/IYXafzjV zGqhhKZBddLMpoYWawKvAxFL1cAUpw8e5sT7gPTmy2kACq/wVlKONg2ZqnltMjLx+Jbd B+aZ6vYHBLZlsI5lFBefRPfhQUChAwzM6rbWA8sX0bEBtMSQOkTKCYuLJgMwx+fMaj66 yMRCyV5tzTaURpa71QLSSa+usAa6c+Arym2BKSSmC4su4r4+//8s59mlc6dEPKZn2oaX jjZA== MIME-Version: 1.0 X-Received: by 10.194.63.229 with SMTP id j5mr23328764wjs.79.1374599887571; Tue, 23 Jul 2013 10:18:07 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Tue, 23 Jul 2013 10:18:07 -0700 (PDT) In-Reply-To: References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> <51EE2C2B.4020800@mail.ru> Date: Tue, 23 Jul 2013 10:18:07 -0700 X-Google-Sender-Auth: PymJ1vcNJrOpSpOPLkLSWp03yMI Message-ID: Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour From: Adrian Chadd To: trafdev Content-Type: text/plain; charset=ISO-8859-1 Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 17:18:09 -0000 Answering my own email: SO_REUSEPORT on FreeBSD doesn't load balance incoming connections. Test case: * 8 threads * each creates a TCP socket, listening on port 1667, with SO_REUSEPORT * only the first thread ever sees incoming requests. I think this load distribution feature is useful to implement, but it shouldn't be called SO_REUSEPORT. (Silly Linux, why would you do that too..) -adrian On 23 July 2013 07:39, Adrian Chadd wrote: > On 23 July 2013 00:09, trafdev wrote: >> It's like shared acceptor FD and N processes: > > [snip] looks like mine, but I use threads. > >> Accept conn callback is called in N processes on each connection, only one >> wins, >> others exit by errno == EAGAIN case. Overhead is almost zero. >> Problem is that "wins" distribution is far from equal. > > Right. I'm not at that stage yet, but I can totally see that happening. > > Ok. Time to hit up the TCP stack people to weigh in on the recent lkml > posts about this: > > http://lwn.net/Articles/542629/ > > With SO_REUSEPORT, we should create one listen FD per thread, and let > the OS balance how that gets distributed. Rather than one listen > socket that is shared between all processes/threads. I'll try that > locally and see if that works right. Would you mind trying it locally > and see if it improves the distribution of work? > > Thanks, > > > -adrian