From owner-freebsd-net@freebsd.org Sun Feb 26 17:37:53 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 057B3CEE466 for ; Sun, 26 Feb 2017 17:37:53 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DCACBF88 for ; Sun, 26 Feb 2017 17:37:52 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id CDFB1CEE465; Sun, 26 Feb 2017 17:37:52 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CDAB4CEE464 for ; Sun, 26 Feb 2017 17:37:52 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (glebi.us [96.95.210.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebi.us", Issuer "cell.glebi.us" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9C3B8F81; Sun, 26 Feb 2017 17:37:52 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (localhost [127.0.0.1]) by cell.glebi.us (8.15.2/8.15.2) with ESMTPS id v1QHbiBN040610 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 26 Feb 2017 09:37:44 -0800 (PST) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebi.us (8.15.2/8.15.2/Submit) id v1QHbifa040609; Sun, 26 Feb 2017 09:37:44 -0800 (PST) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebi.us: glebius set sender to glebius@FreeBSD.org using -f Date: Sun, 26 Feb 2017 09:37:44 -0800 From: Gleb Smirnoff To: Sepherosa Ziehau Cc: Julien Charbon , Jason Eggleston , "freebsd-net@freebsd.org" , hiren@freebsd.org, jtl@freebsd.org, rrs@freebsd.org Subject: Re: listening sockets as non sockets Message-ID: <20170226173744.GH8899@FreeBSD.org> References: <20170127005251.GM2611@FreeBSD.org> <20170210063024.GE1973@FreeBSD.org> <20170216184903.GF58829@FreeBSD.org> <0858647a-ec3c-1a78-053f-d04397a82d8a@freebsd.org> <20170222232704.GJ8899@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.2 (2016-11-26) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 17:37:53 -0000 On Sun, Feb 26, 2017 at 11:37:59PM +0800, Sepherosa Ziehau wrote: S> Just got enough time to do some benchmark. The configuration is S> pretty simple, one server box w/ 2x82599, two client boxes w/ 1x82599, S> connected through DAC. nginx-devel is installed through pkg; access S> log is disabled, 24 workers, each worker w/ 16384 max connections. I S> use the wrk I mentioned to you. HTTP/1.1 and 1 request/connection, S> 15K concurrent connections from each client boxes, so 30K concurrent S> connections total. Each client has been verified to be able to do S> 160Kreqs/s w/ 1KB web object w/ 1 request/connection. 4 different S> sized staic web objects are benched, see below. The solisten was S> checked out on Feb 25th. Several GENERIC options (NFS and SCTP S> related) have to be commented out and kernel is built w/ -DNO_MODULES S> (CAM ctl module does not compile) and GENERIC-NODEBUG on the solisten S> branch. Each test lasts 2 minutes. MSL on the server side has been S> changed to 10ms, though I don't think it matters here. Server hw: S> 2x2620v2 (HT enabled), 32GB ddr3-1600. Client hw: i7-3770 (HT S> enabled), 16GB ddr3-1600. S> S> r314268 -> solisten S> S> 1KB: S> Performance (reqs/s) S> 77916.71 -> 26240.37 S> Latency average S> 121ms -> 294ms S> S> 8KB: S> Performance (reqs/s) S> 77803.72 -> 25968.87 S> Latency average S> 121ms -> 251ms S> S> 16KB: S> Performance (reqs/s) S> 75698.77 -> 25047.23 S> Latency average S> 173ms -> 377ms S> S> 24KB: S> Performance (reqs/s) S> 73736.80 -> 24946.36 S> Latency average S> 180ms -> 379ms S> S> So what I have seen is solisten's performance is 1/3 of r314268, and S> average latency doubles. Thanks a lot! Looks like I've put something on the accept path that degrades performance. :) I will look at this tomorrow and post an update. Can you please share exact wrk command you used? Do I understand it correct that there was only 1 listening socket on the server side? -- Totus tuus, Glebius.