From owner-freebsd-net@FreeBSD.ORG  Sat Jul 13 05:18:29 2013
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by hub.freebsd.org (Postfix) with ESMTP id 86B376CF
 for <freebsd-net@freebsd.org>; Sat, 13 Jul 2013 05:18:29 +0000 (UTC)
 (envelope-from trafdev@mail.ru)
Received: from fallback1.mail.ru (fallback1.mail.ru [94.100.176.18])
 by mx1.freebsd.org (Postfix) with ESMTP id 40C651148
 for <freebsd-net@freebsd.org>; Sat, 13 Jul 2013 05:18:28 +0000 (UTC)
Received: from smtp32.i.mail.ru (smtp32.i.mail.ru [94.100.177.92])
 by fallback1.mail.ru (mPOP.Fallback_MX) with ESMTP id CD66C2243CCE
 for <freebsd-net@freebsd.org>; Sat, 13 Jul 2013 09:16:41 +0400 (MSK)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru;
 s=mail2; 
 h=Content-Transfer-Encoding:Content-Type:Subject:To:MIME-Version:From:Date:Message-ID;
 bh=zrjz5Q+QteNa2LVnZo8mtNw4FtK28IEiMEBu9E7pO3I=; 
 b=h8nxY3NLOaxpOOb0otjRhB26KMaQIO0R4/O88MNukrL15IIvKaPt6o/BDxmHVW/4NqNOYcDckxWbuuUbnX66LCKXZRPQvWawV1F2RuQaO1r+vyeofFEPDcnin2YMYWL5ye3fuChrSezekZQP/a9OaQbovMxj2weuheoPRgrkkHk=;
Received: from [50.156.108.197] (port=15916 helo=[192.168.1.116])
 by smtp32.i.mail.ru with esmtpa (envelope-from <trafdev@mail.ru>)
 id 1UxsCA-0001wy-Eu
 for freebsd-net@freebsd.org; Sat, 13 Jul 2013 09:16:34 +0400
Message-ID: <51E0E2AF.7090404@mail.ru>
Date: Fri, 12 Jul 2013 22:16:31 -0700
From: trafdev <trafdev@mail.ru>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:17.0) Gecko/20130628 Thunderbird/17.0.7
MIME-Version: 1.0
To: freebsd-net@freebsd.org
Subject: SO_REUSEPORT: strange kernel balancer behaviour
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam: Not detected
X-Mras: Ok
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 13 Jul 2013 05:18:29 -0000

Hello.

Could someone help with following problem of SO_REUSEPORT.

Created server:

int sockd_acceptor_;
...

if ((sockd_acceptor_ = socket(PF_INET, SOCK_STREAM, 0)) == -1) {
         LOG4CXX_ERROR_ERRNO(kLogger, "socket");
         return false;
     }
     struct sockaddr_in sa_in;
     memset(&sa_in, 0, sizeof(sa_in));
     sa_in.sin_family = AF_INET;
     sa_in.sin_port = htons(port);
     sa_in.sin_addr.s_addr = htonl(INADDR_ANY);
     int yes = 1;
     if (setsockopt(sockd_acceptor_, SOL_SOCKET, SO_REUSEPORT, &yes, 
sizeof (yes)) == -1) {
         LOG4CXX_ERROR_ERRNO(kLogger, "setsockopt");
         return false;
     }
     if (bind(sockd_acceptor_, (const struct sockaddr*)&sa_in, 
sizeof(sa_in)) == -1) {
         LOG4CXX_ERROR_ERRNO(kLogger, "bind");
         return false;
     }

     if (listen(sockd_acceptor_, listen_backlog) == -1) {
         LOG4CXX_ERROR_ERRNO(kLogger, "socket listen");
         return false;
     }
     if (!fcntl_set(sockd_acceptor_, O_NONBLOCK))
         return false;

Then libev is used as async dispatcher.

Server process 1 started, server process 2 started.
Everything is good so far, no bind errors.

Client started.

Server process 1 starts to reply, process 2 gets no requests yet.
This happens until process 1 is killed. Immediately after that process 2 
starts to respond (gets requests from client).

So it looks like there is a deque of processes sharing same port 
(SO_REUSEPORT) and only one process may respond.
Is this an expected behavior?