From owner-freebsd-net@freebsd.org Fri Mar 3 01:57:19 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13917CF5135 for ; Fri, 3 Mar 2017 01:57:19 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id E5F861010 for ; Fri, 3 Mar 2017 01:57:18 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id E27ABCF5132; Fri, 3 Mar 2017 01:57:18 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E2218CF512B for ; Fri, 3 Mar 2017 01:57:18 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from mail-ua0-x236.google.com (mail-ua0-x236.google.com [IPv6:2607:f8b0:400c:c08::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9AB7C100F; Fri, 3 Mar 2017 01:57:18 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by mail-ua0-x236.google.com with SMTP id f54so99266501uaa.1; Thu, 02 Mar 2017 17:57:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ir0YrlChUlzLU3g/0pY6jCRcH5NhAeZoI3+HW/gZOeQ=; b=eOS+pxOO5gVaxB3fIp5gVn7A2h0I23a0JygK4lxUETBy+Bj5hEsdasP7qfYoRVgqfd lN5k0ZJ3tUuk6MHmKOBLOA+TJJlE+3BK6fwHiYq/97M8eWYidqR8S+O3rftIOQZAzJhU Vu9qTAihazZZ9D3ZdsnB6UE7MBEvYeLpzGRomaFwkqXfAkKIOIwkIRsxVkGavaEuF6/L LB+1wIexRcBRI/rlvTenszC+hIzcXuzOsBb3hy0IBkt3+lqf5a2VrGDg9Ango0jWR8m7 o0B6sC6pTB0rakMQ9js1HOdAP3cDVl4VryiYcfM1SC3tHKkXLmszD7s62SOix3GiYalM cxcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ir0YrlChUlzLU3g/0pY6jCRcH5NhAeZoI3+HW/gZOeQ=; b=qRQBlnbkx5tc9Ff12tahdbZ2LaETz1dsNba5KmLNrU3KdNYZMaN8kq3SVr0n6Lz4Hz nIaGT03dqO6JGAlYYATuS7OULXNBCUzlVFB3FLLEYlqJzQTBlAfXP6oJIpZxdarn45ln owBtzgClYlNyPcsp0eUpthYDO+Gsb3YRva9E1vwGCmwNx1sgMoxXTqBPV8LggDjbOfw6 WB0r83NRenMNkT/8ygXZ8hGiDAnXmunWFUKG6QgvaifD8/cLQPevPnXIZtFGf/qaArrl RDETzboaDY+xGe4tdURWIV38YDKEP5w/dI/Ga2yud1bKjiR2jHS3A0QYzqmHzG35isDy zJew== X-Gm-Message-State: AMke39lw/3n8oPLKTVluPDcGIxdrTv3Ev1su4n3khY4RfDn9tkM3iYC7nBV0pV/Xbrb5awq3CaSOSSM/dgvksw== X-Received: by 10.176.6.233 with SMTP id g96mr130010uag.68.1488506237429; Thu, 02 Mar 2017 17:57:17 -0800 (PST) MIME-Version: 1.0 Received: by 10.176.66.193 with HTTP; Thu, 2 Mar 2017 17:57:16 -0800 (PST) In-Reply-To: <20170302220051.GU1044@FreeBSD.org> References: <20170127005251.GM2611@FreeBSD.org> <20170210063024.GE1973@FreeBSD.org> <20170216184903.GF58829@FreeBSD.org> <0858647a-ec3c-1a78-053f-d04397a82d8a@freebsd.org> <20170222232704.GJ8899@FreeBSD.org> <20170302220051.GU1044@FreeBSD.org> From: Sepherosa Ziehau Date: Fri, 3 Mar 2017 09:57:16 +0800 Message-ID: Subject: Re: listening sockets as non sockets To: Gleb Smirnoff Cc: Julien Charbon , Jason Eggleston , "freebsd-net@freebsd.org" , hiren@freebsd.org, jtl@freebsd.org, rrs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 01:57:19 -0000 On Fri, Mar 3, 2017 at 6:00 AM, Gleb Smirnoff wrote: > On Sun, Feb 26, 2017 at 11:37:59PM +0800, Sepherosa Ziehau wrote: > S> r314268 -> solisten > S> > S> 1KB: > S> Performance (reqs/s) > S> 77916.71 -> 26240.37 > S> Latency average > S> 121ms -> 294ms > ... > S> So what I have seen is solisten's performance is 1/3 of r314268, and > S> average latency doubles. > > I did similar testing, and my results are the following, for three > consecutive runs: > > solisten head (r306199) > req/s 63k,63k,63k 46k,47k,44k > latency 213,214,208 232,233,223 > > So, I don't see latency increase, neither req/s regression. I see > the opposite. > > What is different about my test? First, this is NetflixBSD, both head > and solisten installation. Head is based on r306199 and solisten > is based on r314150 and cb79de4fd2912450c4ab808c017ae395fd636bd8 from > my github. > > To my knowledge the parts of the stack that are different in NetflixBSD > do not touch sonewconn(), accept4() and other parts we are interested at. > I also didn't notice any drastical changes in head between r306199 and > r314150. So imho it is fair to attribute the difference to my change. > > The hardware is different. It is Supermicro X9SRH-7F/7TF, > Xeon(R) CPU E5-2697 v2 @ 2.70GHz, 256Gb RAM and Chelsio cxl(4) > at 40Gbit/s. I got two boxes of this configuration one running head > and other solisten. The client box runs same CPU and mainboard, but has > lagg of two cxls, capping it to 80 Gbit/s, which isn't important but, > what is important providing more parallelism at sending side. > > The nginx has multiple listening sockets, but we bombard only one that > is at AF_INET4 *:80. The nginx is configured to 64 worker processes > and accept_mutex is on. So, even with 1 socket, seems like I got some > improvement. > > I run your wrk 498d70f6da5a201f109488eeaf31c8ba891dc163, and the command > used on the sending side is: > > ./wrk -c 15000 -t 48 -d 120s --delay --latency --connreqs 1 http://host/file > > The difference to your command is only threads count. My box has much > more cores. Well, as I mentioned to you, I hooked up two client boxes, each runs 15000 concurrent connections; so 30K concurrent connections total. BTW, how heavy traffic your client box could generate? Each of mine client box could generate 160Kreqs/s for 1req/conn for 1KB web object, maybe that's the difference? > > The file is of size 1657 bytes. > > Sephe, can you please get hwpmc dumps with your test on solisten/head? In > the test that shows that solisten is 3x slower. Yeah, sure. I need to restore the freebsd on the server side; just wiped out the disk to do the same bench on linux. I will give you the information you want in one week or two. Could you give me the exact command I need to use for extracting the hwpmc stats? Thanks, sephe -- Tomorrow Will Never Die