From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 07:09:45 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E989ABE0; Tue, 23 Jul 2013 07:09:44 +0000 (UTC) (envelope-from trafdev@mail.ru) Received: from smtp47.i.mail.ru (smtp47.i.mail.ru [94.100.177.107]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6598F2B02; Tue, 23 Jul 2013 07:09:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=TqOcck8PeG6L1MCeZW1/SzPqbuR3Ey8z390dZgAPZxw=; b=ZaLjEzUSRDGs6GCWEb1KdUEulym9P9WPZhd7hG/gpLVyDVbROhEjLtOeYoaZK5YJAZQuiDqB2m5UQjiuCdzluhRWA6Furrnm7lez3rAj0E2sZgFfXThorwxnTvUyOx/GZAT+CxwAiBjWQF5a7AN2ayYZaijK5WRad3aVslHCE0I=; Received: from [50.156.108.197] (port=25693 helo=[192.168.1.116]) by smtp47.i.mail.ru with esmtpa (envelope-from ) id 1V1Wj1-0000Pu-Ff; Tue, 23 Jul 2013 11:09:35 +0400 Message-ID: <51EE2C2B.4020800@mail.ru> Date: Tue, 23 Jul 2013 00:09:31 -0700 From: trafdev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130628 Thunderbird/17.0.7 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 07:09:45 -0000 It's like shared acceptor FD and N processes: Listening proc: bool HttpServer::Listen(unsigned short port, uint listen_backlog) { LOG4CXX_TRACE(kLogger, "Listen"); if ((sockd_acceptor_ = socket(AF_INET, SOCK_STREAM, 0)) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "socket"); return false; } struct sockaddr_in sa_in; memset(&sa_in, 0, sizeof(sa_in)); sa_in.sin_family = AF_INET; sa_in.sin_port = htons(port); sa_in.sin_addr.s_addr = htonl(INADDR_ANY); int yes = 1; if (setsockopt(sockd_acceptor_, SOL_SOCKET, SO_REUSEPORT, &yes, sizeof (yes)) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "setsockopt"); return false; } if (bind(sockd_acceptor_, (const struct sockaddr*)&sa_in, sizeof(sa_in)) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "bind"); return false; } if (listen(sockd_acceptor_, listen_backlog) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "socket listen"); return false; } if (!fcntl_set(sockd_acceptor_, O_NONBLOCK)) return false; sockd_acceptor_watcher_.set (this); sockd_acceptor_watcher_.start(sockd_acceptor_, ev::READ); int wrk = thread::hardware_concurrency(); while (--wrk) { pid_t pid = fork(); if (pid == 0) { LOG4CXX_INFO(kLogger, "process " << wrk << " started"); ev::default_loop().post_fork(); ev::default_loop().run(); return true; } } ev::default_loop().run(); return true; } Accept conn callback: void HttpServer::AcceptConnCb(ev::io &w, int revents) { LOG4CXX_TRACE(kLogger, "AcceptConnCb"); CHECK_LIBEV_ERROR struct sockaddr_in addr; socklen_t addr_len = sizeof (addr); int sockd; if ((sockd = accept(sockd_acceptor_, (struct sockaddr *) &addr, &addr_len)) == -1) { if (errno == EAGAIN || errno == EWOULDBLOCK) { return; } else { LOG4CXX_ERROR_ERRNO(kLogger, "accept"); return; } } if (!fcntl_set(sockd, O_NONBLOCK)) return; char str[INET_ADDRSTRLEN]; if (!inet_ntop(AF_INET, &(addr.sin_addr), str, INET_ADDRSTRLEN)) { LOG4CXX_ERROR(kLogger, "inet_ntop"); return; } // Internal logic follows... ConnIn* c = ConnPool::Instance().Add(sockd, str); // c->Recv(this); return; } Accept conn callback is called in N processes on each connection, only one wins, others exit by errno == EAGAIN case. Overhead is almost zero. Problem is that "wins" distribution is far from equal. On Mon Jul 22 23:37:57 2013, Adrian Chadd wrote: > eg: > > * one process, one listen thread, multiple dispatch threads? > * n processes, one listen FD per process, all listening on the same IP:port? > * one process, each thread listening on the same IP:port? > * something else? > > Thanks, > > > > -adrian > >