From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 17:54:50 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1907D3AA; Tue, 23 Jul 2013 17:54:50 +0000 (UTC) (envelope-from trafdev@mail.ru) Received: from smtp49.i.mail.ru (smtp49.i.mail.ru [94.100.177.109]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 872772C08; Tue, 23 Jul 2013 17:54:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=TgxRVOR/HDk3pslwDTt+0/Fgj6xZ4bxuMvhzNhezyfU=; b=TUtE/y/mGkko3A/mLq0bFZ4ZRq3ZxrxOQfeeAs535ba6g1oWOUp0/zX1lhTtwyxuMC/ba6dCv5RsnihmhBVGzCPwuQcynNLh4cQO8DLL6yX+Y8wGtKyAe+TGIjmnYwlf/0bXJQq0K+mErqtSUB0QQCnM0cL22CkIl/+xVedCp2I=; Received: from [50.156.108.197] (port=50336 helo=[192.168.1.116]) by smtp49.i.mail.ru with esmtpa (envelope-from ) id 1V1gnN-0006Dp-TX; Tue, 23 Jul 2013 21:54:46 +0400 Message-ID: <51EEC361.9050806@mail.ru> Date: Tue, 23 Jul 2013 10:54:41 -0700 From: trafdev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130628 Thunderbird/17.0.7 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> <51EE2C2B.4020800@mail.ru> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 17:54:50 -0000 Yes, and if you kill this first thread - second thread will start to receive connections and so on. That's why I've used one processes-shared acceptor socket which behaves better (load is balancing between processes but equality of distribution is far from ideal). Btw as per https://lwn.net/Articles/542629/ Linux 3.1 SOLVES all these problems via SO_REUSEPORT. On Tue Jul 23 10:18:07 2013, Adrian Chadd wrote: > Answering my own email: > > SO_REUSEPORT on FreeBSD doesn't load balance incoming connections. > > Test case: > > * 8 threads > * each creates a TCP socket, listening on port 1667, with SO_REUSEPORT > * only the first thread ever sees incoming requests. > > I think this load distribution feature is useful to implement, but it > shouldn't be called SO_REUSEPORT. > > (Silly Linux, why would you do that too..) > > > > -adrian > > On 23 July 2013 07:39, Adrian Chadd wrote: >> On 23 July 2013 00:09, trafdev wrote: >>> It's like shared acceptor FD and N processes: >> >> [snip] looks like mine, but I use threads. >> >>> Accept conn callback is called in N processes on each connection, only one >>> wins, >>> others exit by errno == EAGAIN case. Overhead is almost zero. >>> Problem is that "wins" distribution is far from equal. >> >> Right. I'm not at that stage yet, but I can totally see that happening. >> >> Ok. Time to hit up the TCP stack people to weigh in on the recent lkml >> posts about this: >> >> http://lwn.net/Articles/542629/ >> >> With SO_REUSEPORT, we should create one listen FD per thread, and let >> the OS balance how that gets distributed. Rather than one listen >> socket that is shared between all processes/threads. I'll try that >> locally and see if that works right. Would you mind trying it locally >> and see if it improves the distribution of work? >> >> Thanks, >> >> >> -adrian > >