From owner-freebsd-net@freebsd.org Wed Mar 8 15:03:51 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EF5D2D02574 for ; Wed, 8 Mar 2017 15:03:51 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D2A7643 for ; Wed, 8 Mar 2017 15:03:51 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-wm0-x230.google.com with SMTP id n11so117129804wma.1 for ; Wed, 08 Mar 2017 07:03:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=h7wFPJuamD9C4EJcFczG27zj9qBNTK7e14JP058/2jU=; b=dFi7bH6tQYgvkr7fTUMgP2UgQSQPSFkz+YFDuBZDCUDjDD7xmH7tD681fjO967+pbv O9rVgGSJ9Z3vNLnxM5tmeFYao6ruQ9weYzkPYYlIkNqwPSYo0jWaG48S8KGbAEMf65bv 7XKx2yLWmsPPeksAHGGAG4brds5inV4znM09CHKyqeRc7QgbyIuwgH9hZzV4LJXEHD50 tSMqZ2B0I4mnx2vq8a4SXhP3kZqXOgSzi6FnbhVrsKQ4PP86fNlIz8Skl14VTYvnBTnk WO19ZBUXHXhbsYc6y/FwG5bn7nQbEl99REp0Po4xlbFjwvFmP0v74xGp5J6YVK8VBWdF qHkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=h7wFPJuamD9C4EJcFczG27zj9qBNTK7e14JP058/2jU=; b=PiIwFcTrbN+OuSAaucxFwXOrdyT0Effe+Kln+SaB2Y1tHSpH6K4VmwrZI5MMkyrN80 H/W8+JZ3qBGNjp8R0B7psbrDu60P4cLIsbc0X4fQxmaEQu81FRtgQekzkBO0ruFOMSrL fdMjfCiYtHG/Ce1iBs5IWmGEI86O9HEPJ1vGMWdYLjiDbs9ypZdonbo7wAGkIBhh+LOC yN+M2d+6CRpeFMA0/6vnjYTdWztwpWCPpU97uhWOKvLkxU0SCS8yTsNJnw4KOwp5e9ek g2fL0f7S2wB1SDj6lJ7jWX2lP/Xdt2M5nL19g43q248FecowGssjGC7cgnsPfgdePABc tR1g== X-Gm-Message-State: AMke39nXVDanUKIwhbsJJF3sECyvOAl6SRzonx9V5yKj2oxpN3wRXdLYevgal5aVTbBKOg== X-Received: by 10.28.74.221 with SMTP id n90mr5900219wmi.114.1488985429707; Wed, 08 Mar 2017 07:03:49 -0800 (PST) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by smtp.gmail.com with ESMTPSA id j26sm4485219wrb.69.2017.03.08.07.03.48 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 08 Mar 2017 07:03:48 -0800 (PST) Date: Wed, 8 Mar 2017 16:03:46 +0100 From: Mateusz Guzik To: Slawa Olhovchenkov Cc: Kevin Bowling , freebsd-net , "Eugene M. Zheganin" Subject: Re: about that DFBSD performance test Message-ID: <20170308150346.GA32269@dft-labs.eu> References: <20170308125710.GS15630@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20170308125710.GS15630@zxy.spb.ru> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Mar 2017 15:03:52 -0000 On Wed, Mar 08, 2017 at 03:57:10PM +0300, Slawa Olhovchenkov wrote: > On Wed, Mar 08, 2017 at 05:25:57AM -0700, Kevin Bowling wrote: > > > Right off the bat, FreeBSD doesn't really understand NUMA in any sufficient > > capacity. Unfortunately at companies like the one I work at, we take that > > to mean "OK buy a high bin CPU and only populate one socket" which serves > > NUMA applicable only to high-localy computed tasks. > http/https/any_network_related serving is not related to this. > Indeed, on modern CPU is not important to bind NIC irq handlers to > same CPU/sockets as NIC. > Well, for both benchmarks this is both true and false. First and foremost there is general kernel scalability. Certain counters and most locks are purely managed with atomic operations. An atomic operation grabs the entire cacheline with the particular variable (64 bytes in total) in exclusive mode. If you have to do an atomic operation you are somewhat slower than you be otherwise. If you have to do an atomic operation and another cpu has the cacheline, you are visibly slower. And if the cacheline travels a lot between cpus (e.g. because the lock is contended), the performance degrades rapidly. NUMA increases the cost of cacheline bounces, making the already bad situation even worse. Locking primitives are affected by NUMA significantly more than they have to be (I'm working on that), but any fixes in the area are just bandaids. For instancee, I reproduce the http benchmark and indeed I have about 75k req/s on 2 * 10 * 2 box, although I'm only using one client. Profiling shows excessive contention on the 'accept lock' and something else from the socket layer. The latter comes from kqueue being extremely inefficient by acquiring and releasing the same lock about 4 times per call on average (if it took it *once* it would significantly reduce lock bouncing around, including across the socket to a different node). But even taking it once is likely too bad - no matter how realistically fast this can get, if all nginx processes serialize on this lock this is not going to scale. That said, the end result would be significantly higher if lock granularity was better and I suspect numa-awareness would not be a significant factor in the http benchmark - provided locks are granular enough, they would travel across the socket only if they get pushed out of the cache (which would be rare), but there would be no contention. This is a small excerpt from a reply I intend to write to the other thread where the 'solisten' patch is discussed. It gets rid of the accept lock contention, but this increases the load on other lock and thattemporarily slows things down. -- Mateusz Guzik