From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 21:26:35 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D8ADCC62; Mon, 22 Jul 2013 21:26:35 +0000 (UTC) (envelope-from trafdev@mail.ru) Received: from fallback3.mail.ru (fallback3.mail.ru [94.100.176.58]) by mx1.freebsd.org (Postfix) with ESMTP id 5883C25CC; Mon, 22 Jul 2013 21:26:35 +0000 (UTC) Received: from smtp49.i.mail.ru (smtp49.i.mail.ru [94.100.177.109]) by fallback3.mail.ru (mPOP.Fallback_MX) with ESMTP id C9552EF6DF60; Tue, 23 Jul 2013 01:26:33 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=6q1Le/jIPEZy2+u861Ht2Yr3ZGWfeDMWoCfzZlduIe0=; b=ETy1BJV90GQGKeMfJwiECgDvSop82UdlarLqqWlJqr0y7Hcoow/0R4UWJvduQyuJG5V1kkUAFpAOZFwTgw5xGzJfN3t+PVmDgP1fKoKL75vjb8hI0lKBuMWGTjSbwagNmJXsC3KmHoL80LmuMyPU2T4j4+dSvyrdPQJx44v3RmE=; Received: from [50.156.108.197] (port=42334 helo=[192.168.1.116]) by smtp49.i.mail.ru with esmtpa (envelope-from ) id 1V1Ncf-0006io-5F; Tue, 23 Jul 2013 01:26:25 +0400 Message-ID: <51EDA37A.9040200@mail.ru> Date: Mon, 22 Jul 2013 14:26:18 -0700 From: trafdev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130628 Thunderbird/17.0.7 MIME-Version: 1.0 To: John-Mark Gurney Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> In-Reply-To: <20130722200205.GO26412@funkthat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 21:26:36 -0000 Actually overhead is almost zero, the real problem is in non-equivalent load distribution between processes. As https://lwn.net/Articles/542629/ mentions - "At Google, they have seen a factor-of-three difference between the thread accepting the most connections and the thread accepting the fewest connections;" I'm getting almost same results On Mon Jul 22 13:02:05 2013, John-Mark Gurney wrote: > trafdev wrote this message on Mon, Jul 15, 2013 at 13:04 -0700: >> Yep I think it's wasting of resources, poll manager should somehow be >> configured to update only one process/thread. >> Anyone know how to do that? > > This isn't currently possible w/o a shared kqueue, since the event is > level triggered, not edge.. You could do it w/ a shared kqueue using > _ONESHOT (but then you'd also have a shared listen fd which obviously > isn't what the OP wants)... > > I guess it wouldn't be too hard to do a wake one style thing, where > kqueue only deliveres the event once per "item/level", but right now > kqueue doesn't know anything about the format of data (which would be > number of listeners waiting)... Even if it did, there would be this > dangerous contract that if an event is returned that the user land > process would handle it... How is kqueue suppose to handle a server > that crashes/dies between getting the event and accepting a connection? > How is userland suppose to know that an event wasn't handled, or is > just taking a long time? > > Sadly, if you want to avoid the thundering heards problem, I think > blocking on accept is the best method, or using a fd passing scheme > where only on process accept's connections... > >> On Mon Jul 15 12:53:55 2013, Adrian Chadd wrote: >>> i've noticed this when doing this stuff in a threaded program with >>> each thread listening on the same port. >>> >>> All threads wake up on each accepted connection, one thread wins and >>> the other threads get EAGAIN. >>> >>> >>> >>> -adrian >>> >>> On 15 July 2013 12:31, trafdev wrote: >>>> Thanks for reply. >>>> >>>> This approach produces lot of "resource temporary unavailable" (eagain) on >>>> accept-ing connections in N-1 processes. >>>> Is this possible to avoid this by e.g. tweaking kqueue? >>>> >>>> >>>> On Sun Jul 14 19:37:59 2013, Sepherosa Ziehau wrote: >>>>> >>>>> On Sat, Jul 13, 2013 at 1:16 PM, trafdev wrote: >>>>>> >>>>>> Hello. >>>>>> >>>>>> Could someone help with following problem of SO_REUSEPORT. >>>>> >>>>> >>>>> The most portable "load balance" between processes listening on the >>>>> same TCP addr/port probably is: >>>>> >>>>> s=socket(); >>>>> bind(s); >>>>> listen(s); >>>>> /* various socketopt and fcntl as you needed */ >>>>> pid=fork(); >>>>> if (pid==0) { >>>>> server_loop(s); >>>>> exit(1); >>>>> } >>>>> server_loop(s); >>>>> exit(1); >>>>> >>>>> Even in Linux or DragonFly SO_REUSEPORT "load balance" between >>>>> processes listening on the same TCP addr/port was introduced recently, >>>>> so you probably won't want to rely on it. >