From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 20:02:13 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 077E8497; Mon, 22 Jul 2013 20:02:13 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D908B21AE; Mon, 22 Jul 2013 20:02:12 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r6MK26Cd007072 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 22 Jul 2013 13:02:06 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r6MK25SH007071; Mon, 22 Jul 2013 13:02:05 -0700 (PDT) (envelope-from jmg) Date: Mon, 22 Jul 2013 13:02:05 -0700 From: John-Mark Gurney To: trafdev Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour Message-ID: <20130722200205.GO26412@funkthat.com> Mail-Followup-To: trafdev , Adrian Chadd , Sepherosa Ziehau , freebsd-net@freebsd.org References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <51E455D5.2090403@mail.ru> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Mon, 22 Jul 2013 13:02:06 -0700 (PDT) Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 20:02:13 -0000 trafdev wrote this message on Mon, Jul 15, 2013 at 13:04 -0700: > Yep I think it's wasting of resources, poll manager should somehow be > configured to update only one process/thread. > Anyone know how to do that? This isn't currently possible w/o a shared kqueue, since the event is level triggered, not edge.. You could do it w/ a shared kqueue using _ONESHOT (but then you'd also have a shared listen fd which obviously isn't what the OP wants)... I guess it wouldn't be too hard to do a wake one style thing, where kqueue only deliveres the event once per "item/level", but right now kqueue doesn't know anything about the format of data (which would be number of listeners waiting)... Even if it did, there would be this dangerous contract that if an event is returned that the user land process would handle it... How is kqueue suppose to handle a server that crashes/dies between getting the event and accepting a connection? How is userland suppose to know that an event wasn't handled, or is just taking a long time? Sadly, if you want to avoid the thundering heards problem, I think blocking on accept is the best method, or using a fd passing scheme where only on process accept's connections... > On Mon Jul 15 12:53:55 2013, Adrian Chadd wrote: > >i've noticed this when doing this stuff in a threaded program with > >each thread listening on the same port. > > > >All threads wake up on each accepted connection, one thread wins and > >the other threads get EAGAIN. > > > > > > > >-adrian > > > >On 15 July 2013 12:31, trafdev wrote: > >>Thanks for reply. > >> > >>This approach produces lot of "resource temporary unavailable" (eagain) on > >>accept-ing connections in N-1 processes. > >>Is this possible to avoid this by e.g. tweaking kqueue? > >> > >> > >>On Sun Jul 14 19:37:59 2013, Sepherosa Ziehau wrote: > >>> > >>>On Sat, Jul 13, 2013 at 1:16 PM, trafdev wrote: > >>>> > >>>>Hello. > >>>> > >>>>Could someone help with following problem of SO_REUSEPORT. > >>> > >>> > >>>The most portable "load balance" between processes listening on the > >>>same TCP addr/port probably is: > >>> > >>>s=socket(); > >>>bind(s); > >>>listen(s); > >>>/* various socketopt and fcntl as you needed */ > >>>pid=fork(); > >>>if (pid==0) { > >>> server_loop(s); > >>> exit(1); > >>>} > >>>server_loop(s); > >>>exit(1); > >>> > >>>Even in Linux or DragonFly SO_REUSEPORT "load balance" between > >>>processes listening on the same TCP addr/port was introduced recently, > >>>so you probably won't want to rely on it. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."