From owner-freebsd-net@freebsd.org Tue Aug 16 19:36:49 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89061BBCE23 for ; Tue, 16 Aug 2016 19:36:49 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-it0-x244.google.com (mail-it0-x244.google.com [IPv6:2607:f8b0:4001:c0b::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DB6A11A9 for ; Tue, 16 Aug 2016 19:36:49 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-it0-x244.google.com with SMTP id d65so6339358ith.0 for ; Tue, 16 Aug 2016 12:36:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=3inaiuo5i62Z+nPFdWjA7+tJLp2tkP/qb8+ozf7NVGo=; b=u2Ojajx74CA4PtovqG3DTRO82n6ReIN0Q2NvblHh6I47rrnMw5sKfe1aYrNXdKPZ52 DMoCuZVRU3E3Pp8zSckuuvXvwnAMERmoqgePJtSwx3YhA4XloTs3HWk+p0w/NjKOhln4 V5gyyxih1WacgunYh6Fjp4XO3bRwkK4xqep8cSmNBR1yHChrj0YuS/xTDdTH7pC4OBz8 Na2nhnjBzGX/RpBuh0FbA8FPrIjlglGzWqBcTzIiE5Q36jXXlY0IU2fbe2jUvS/aACGz RlO493p4LgP8rquoukyBdsvUTpvUyahCEr5BRErqgXyHYKBQIkPrHDA3UoQqI3q84ECE rUCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=3inaiuo5i62Z+nPFdWjA7+tJLp2tkP/qb8+ozf7NVGo=; b=knPchXrn8uvGkNwpLZ5uhMOLlXTYxFCeUolwPPlnYh6OBnK8GGgBrXhkRf0IhtjjT+ sZ/H4260GXSbNhZ0BmWkjEdLn+9X8OZfsWwN5bq9/ZT4Iz1qZNC8x0orsicdFHu0zxG4 oOVlXRjjbj0Dwha5hJsoH2fClLz5+PDIokT3MOLE+OI5CJonmwSzBsTzYiwSyQ4+jAxr eqrqR2J5r8ewb71/UMO0wf//QAglMn58bH68iOlAlQYGcPJ4cMI488rv+cgP4NMNd8P2 mhwrSYzAcIwpaZQdnnxRDbftNvhGSCofZ0EtonpEGPeGvGzRMikW6/ATc/s6CEGNBSiA XCKw== X-Gm-Message-State: AEkoouvxx0YohKS0ROeFqHWW3TmRaojs9bYvjP1P2zxi/jK68wb52mRLPJ0y9PjxlbVMeGlmoqmuS4VI4fWEoQ== X-Received: by 10.36.242.68 with SMTP id j65mr22532230ith.25.1471376206753; Tue, 16 Aug 2016 12:36:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.36.141.129 with HTTP; Tue, 16 Aug 2016 12:36:45 -0700 (PDT) In-Reply-To: <5EAA9DC3-9A76-4F67-9A03-31FFCB08DF7D@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <7DD30CE7-32E6-4D26-91D4-C1D4F2319655@gmail.com> <91AEB1BD-44EA-43AD-A9A1-6DEBF367DF9B@gmail.com> <5EAA9DC3-9A76-4F67-9A03-31FFCB08DF7D@gmail.com> From: Adrian Chadd Date: Tue, 16 Aug 2016 12:36:45 -0700 Message-ID: Subject: Re: Unstable local network throughput To: Ben RUBSON Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Aug 2016 19:36:49 -0000 On 16 August 2016 at 02:58, Ben RUBSON wrote: > >> On 16 Aug 2016, at 03:45, Adrian Chadd wrote: >> >> Hi, >> >> ok, can you try 5) but also running with the interrupt threads pinned to CPU 1? > > What do you mean by interrupt threads ? > > Perhaps you mean the NIC interrupts ? > In this case see 6) and 7) where NIC IRQs are pinned to CPUs 0-11 (6) and 11-23 (7) ? Hm, interesting. ok. So, I wonder what the maximum per-domain memory throughput is. I don't have any other easy things to instrument right now - the "everything disabled" method likely works best because of how the system is interleaving memory for you (instead of the OS trying to do it). Not pinning things means latency can be kept down to work around lock contention (ie, if a lock is held by thread A, and thread B needs to make some progress, it can make progress on another CPU , keeping CPU A held for a shorter period of time.) Would you mind compiling in LOCK_PROFILING and doing say, these tests with lock profiling enabled? It'll impact performance, sure, but I'd like to see what the locking looks like. sysctl debug.lock.prof.reset=1 sysctl debug.lock.prof.enable=1 (run test for a few seconds) sysctl debug.lock.prof.enable=0 sysctl debug.lock.prof.stats (and capture) * interrupts - domain 0, work - domain 1 * interrupts - domain 1, work - domain 1 * interrupts - domain 1, work - domain 0 Thanks! -adrian