From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 14:39:42 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2E20FB74 for ; Tue, 23 Jul 2013 14:39:42 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B7BC121F2 for ; Tue, 23 Jul 2013 14:39:41 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id hq4so3097399wib.12 for ; Tue, 23 Jul 2013 07:39:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=iCuAuG/rjE4dfxajZkiEIQDOUBj8FLES5RcIEX5jNKI=; b=gxHRpbBGxqyOtB7AlbAfDj3bN7zblwgGpFYrOp07v9wK8xAMprIlafmpxg4G8Bf95d X30NGUTApUezRs2BTiVCavQbI1wFXbEy5yBCI/SWXC2HXDXaWCjSEQiCASI0Eri6/AHH arYkdO3erGRyrny9A2Z6VhOoS6wZq8IvqP4yHY7lwY5uumAj6VmfFatR2YitAARwPMY6 S24j8XHSy+9gmGbkwX8m7Hw+Cvm3w9giaTJ9KqChexwSyT8VL+DW2xjBjnbdrOhHNwd2 wfLx6HmJNF1q39AvUdnhjID3yt0KGMJ4FkhkoPdQlQkg0FBhrbQD4R1yHB0Vpem17N8g Q/SQ== MIME-Version: 1.0 X-Received: by 10.180.160.165 with SMTP id xl5mr33225492wib.46.1374590379987; Tue, 23 Jul 2013 07:39:39 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Tue, 23 Jul 2013 07:39:39 -0700 (PDT) In-Reply-To: <51EE2C2B.4020800@mail.ru> References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> <51EE2C2B.4020800@mail.ru> Date: Tue, 23 Jul 2013 07:39:39 -0700 X-Google-Sender-Auth: NwNRzoSr_De_ua_qQkP5cgHqYEw Message-ID: Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour From: Adrian Chadd To: trafdev Content-Type: text/plain; charset=ISO-8859-1 Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 14:39:42 -0000 On 23 July 2013 00:09, trafdev wrote: > It's like shared acceptor FD and N processes: [snip] looks like mine, but I use threads. > Accept conn callback is called in N processes on each connection, only one > wins, > others exit by errno == EAGAIN case. Overhead is almost zero. > Problem is that "wins" distribution is far from equal. Right. I'm not at that stage yet, but I can totally see that happening. Ok. Time to hit up the TCP stack people to weigh in on the recent lkml posts about this: http://lwn.net/Articles/542629/ With SO_REUSEPORT, we should create one listen FD per thread, and let the OS balance how that gets distributed. Rather than one listen socket that is shared between all processes/threads. I'll try that locally and see if that works right. Would you mind trying it locally and see if it improves the distribution of work? Thanks, -adrian