From owner-freebsd-net@freebsd.org Thu Mar 2 22:00:59 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 350BBCF5CA7 for ; Thu, 2 Mar 2017 22:00:59 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 1DA1C336 for ; Thu, 2 Mar 2017 22:00:59 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1D03BCF5CA6; Thu, 2 Mar 2017 22:00:59 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CB3DCF5CA5 for ; Thu, 2 Mar 2017 22:00:59 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (glebi.us [96.95.210.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebi.us", Issuer "cell.glebi.us" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 03013331; Thu, 2 Mar 2017 22:00:58 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (localhost [127.0.0.1]) by cell.glebi.us (8.15.2/8.15.2) with ESMTPS id v22M0pTK039169 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 2 Mar 2017 14:00:51 -0800 (PST) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebi.us (8.15.2/8.15.2/Submit) id v22M0pgm039168; Thu, 2 Mar 2017 14:00:51 -0800 (PST) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebi.us: glebius set sender to glebius@FreeBSD.org using -f Date: Thu, 2 Mar 2017 14:00:51 -0800 From: Gleb Smirnoff To: Sepherosa Ziehau Cc: Julien Charbon , Jason Eggleston , "freebsd-net@freebsd.org" , hiren@freebsd.org, jtl@freebsd.org, rrs@freebsd.org Subject: Re: listening sockets as non sockets Message-ID: <20170302220051.GU1044@FreeBSD.org> References: <20170127005251.GM2611@FreeBSD.org> <20170210063024.GE1973@FreeBSD.org> <20170216184903.GF58829@FreeBSD.org> <0858647a-ec3c-1a78-053f-d04397a82d8a@freebsd.org> <20170222232704.GJ8899@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.2 (2016-11-26) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Mar 2017 22:00:59 -0000 On Sun, Feb 26, 2017 at 11:37:59PM +0800, Sepherosa Ziehau wrote: S> r314268 -> solisten S> S> 1KB: S> Performance (reqs/s) S> 77916.71 -> 26240.37 S> Latency average S> 121ms -> 294ms ... S> So what I have seen is solisten's performance is 1/3 of r314268, and S> average latency doubles. I did similar testing, and my results are the following, for three consecutive runs: solisten head (r306199) req/s 63k,63k,63k 46k,47k,44k latency 213,214,208 232,233,223 So, I don't see latency increase, neither req/s regression. I see the opposite. What is different about my test? First, this is NetflixBSD, both head and solisten installation. Head is based on r306199 and solisten is based on r314150 and cb79de4fd2912450c4ab808c017ae395fd636bd8 from my github. To my knowledge the parts of the stack that are different in NetflixBSD do not touch sonewconn(), accept4() and other parts we are interested at. I also didn't notice any drastical changes in head between r306199 and r314150. So imho it is fair to attribute the difference to my change. The hardware is different. It is Supermicro X9SRH-7F/7TF, Xeon(R) CPU E5-2697 v2 @ 2.70GHz, 256Gb RAM and Chelsio cxl(4) at 40Gbit/s. I got two boxes of this configuration one running head and other solisten. The client box runs same CPU and mainboard, but has lagg of two cxls, capping it to 80 Gbit/s, which isn't important but, what is important providing more parallelism at sending side. The nginx has multiple listening sockets, but we bombard only one that is at AF_INET4 *:80. The nginx is configured to 64 worker processes and accept_mutex is on. So, even with 1 socket, seems like I got some improvement. I run your wrk 498d70f6da5a201f109488eeaf31c8ba891dc163, and the command used on the sending side is: ./wrk -c 15000 -t 48 -d 120s --delay --latency --connreqs 1 http://host/file The difference to your command is only threads count. My box has much more cores. The file is of size 1657 bytes. Sephe, can you please get hwpmc dumps with your test on solisten/head? In the test that shows that solisten is 3x slower. Julien, your testing will also be much appreciated. Looks like Sephe's result shouldn't block your try. -- Totus tuus, Glebius.