From owner-freebsd-net@freebsd.org Sun Feb 26 15:38:01 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8B74CEE179 for ; Sun, 26 Feb 2017 15:38:01 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id A7D2E6E7 for ; Sun, 26 Feb 2017 15:38:01 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id A428ECEE178; Sun, 26 Feb 2017 15:38:01 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A211CCEE177 for ; Sun, 26 Feb 2017 15:38:01 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from mail-ua0-x234.google.com (mail-ua0-x234.google.com [IPv6:2607:f8b0:400c:c08::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4EFDF6E5; Sun, 26 Feb 2017 15:38:01 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by mail-ua0-x234.google.com with SMTP id 40so34881968uau.2; Sun, 26 Feb 2017 07:38:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=efeH44UV7NUlC1VzW/hDUAcGELcFQNSoImPzImU86M0=; b=BYuSXwLTKSkBprxkr9I6EIoJE4MMUjNnknx0hHIpLIHfKBvNDq0q2wxGolxmG4pFNn rsHg0ZycRsrEWbeWNDnFW7PUYGtnAjmbmgs5gt9V6Vazxz1H4JV6gbSbfX5WK5UV/5TL YnJyEZeIntIadM0TuyRShKeNLiGPuB0+Gl5/eAeWpjUSN7RC01Ap6Xty/sbzueskYcWo AtHAI9r+CoewEQ5IGhJzu8zQNurcjsASFAN/Zg6jTyp+1ykzvOLmxogVh7lyDt05oqJt gPzNhMr/YvQNtL1OPQHN4/0LCmx2U36UJ3fmMww1Vdc+6h0purMuhdg+GUCdAaSn2Mjw SBsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=efeH44UV7NUlC1VzW/hDUAcGELcFQNSoImPzImU86M0=; b=mV5FoEaAcZRFlQY4tuYHtwVayO0te4UP0iULM+iPIBTvEyn/pqCSBu+MECfjaa1qnl jtTz0o9bUWNlEEipVSvvv/v0r7aUZ9QcUYcSAIn0Qaazlyt42ZO5s3balX5A/TtoiB7H WUXSvFPmOND6KD9bf8EQgNmw4JdHdXl+EY6kwlPJ2iwaqoPn+TTnR+FN/7LzP9kdk6cd 8aOvfNqEjAg0aEX6z3WU5C8QWagyjdsuorTBf1oy2abtdzEuuWq82XX7XLwtZnf0ps2K VunYcz+aPGijdluFjT16L3wPC5d5ASPGxMQbzBTUm7vjQ5ttJEz/QJNXezEUmrIOI7uh vG9A== X-Gm-Message-State: AMke39n3wH4Gnxg/Gig0f5HQQte8cFNUdWFaUsr9X2A0EFitisd//EiMKLaEq+Pt3krJdKshV/E19XWQX6fXVw== X-Received: by 10.176.82.165 with SMTP id v34mr5572554uav.133.1488123479908; Sun, 26 Feb 2017 07:37:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.176.66.193 with HTTP; Sun, 26 Feb 2017 07:37:59 -0800 (PST) In-Reply-To: <20170222232704.GJ8899@FreeBSD.org> References: <20170127005251.GM2611@FreeBSD.org> <20170210063024.GE1973@FreeBSD.org> <20170216184903.GF58829@FreeBSD.org> <0858647a-ec3c-1a78-053f-d04397a82d8a@freebsd.org> <20170222232704.GJ8899@FreeBSD.org> From: Sepherosa Ziehau Date: Sun, 26 Feb 2017 23:37:59 +0800 Message-ID: Subject: Re: listening sockets as non sockets To: Gleb Smirnoff Cc: Julien Charbon , Jason Eggleston , "freebsd-net@freebsd.org" , hiren@freebsd.org, jtl@freebsd.org, rrs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 15:38:01 -0000 On Thu, Feb 23, 2017 at 7:27 AM, Gleb Smirnoff wrote: > On Tue, Feb 21, 2017 at 09:58:45AM +0100, Julien Charbon wrote: > J> > T> 1) The patch worked pretty okay, but the idea of separate file type is > J> > T> abandoned. With current filedescriptor code it is almost impossible > J> > T> to racelessly switch fileops and f_data. > J> > T> For more details read the commit message, URL below. > J> > T> > J> > T> 2) I moved the work to git. It allows for easier code sharing. > J> > T> > J> > T> https://github.com/glebius/FreeBSD/commit/3a52df429889ea9c6e61013f6913aad95939f159 > J> > > J> > The current 'solisten' branch at https://github.com/glebius/FreeBSD has > J> > been running successfully for 48 hours at Netflix. But as mentioned before, > J> > our connection rate is pretty low. > J> > > J> > So, testing from Julien and Hiren will be much appreciated. > J> > J> Sound good, I will have spare time to get performance results on our > J> side w/ and w/o your change before the end of this week. > > Thanks, Julien! Looking forward for your results. Meanwhile, the code has > been running for a week on a single Netflix box. And for 24 hours on a > set of boxes as A/B test against non-patched code. Since our connection > rate is low, and we never seen accept_mtx to be a major problem in our > profiling, the A/B test didn't show any statistically significant > improvements. But my biased eye of course sees some improvements in the > graphs :) Just got enough time to do some benchmark. The configuration is pretty simple, one server box w/ 2x82599, two client boxes w/ 1x82599, connected through DAC. nginx-devel is installed through pkg; access log is disabled, 24 workers, each worker w/ 16384 max connections. I use the wrk I mentioned to you. HTTP/1.1 and 1 request/connection, 15K concurrent connections from each client boxes, so 30K concurrent connections total. Each client has been verified to be able to do 160Kreqs/s w/ 1KB web object w/ 1 request/connection. 4 different sized staic web objects are benched, see below. The solisten was checked out on Feb 25th. Several GENERIC options (NFS and SCTP related) have to be commented out and kernel is built w/ -DNO_MODULES (CAM ctl module does not compile) and GENERIC-NODEBUG on the solisten branch. Each test lasts 2 minutes. MSL on the server side has been changed to 10ms, though I don't think it matters here. Server hw: 2x2620v2 (HT enabled), 32GB ddr3-1600. Client hw: i7-3770 (HT enabled), 16GB ddr3-1600. r314268 -> solisten 1KB: Performance (reqs/s) 77916.71 -> 26240.37 Latency average 121ms -> 294ms 8KB: Performance (reqs/s) 77803.72 -> 25968.87 Latency average 121ms -> 251ms 16KB: Performance (reqs/s) 75698.77 -> 25047.23 Latency average 173ms -> 377ms 24KB: Performance (reqs/s) 73736.80 -> 24946.36 Latency average 180ms -> 379ms So what I have seen is solisten's performance is 1/3 of r314268, and average latency doubles. Thanks, sephe -- Tomorrow Will Never Die