From owner-freebsd-net@freebsd.org Thu Aug 11 18:39:36 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 239DBBB68B8 for ; Thu, 11 Aug 2016 18:39:36 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A79AA1A54 for ; Thu, 11 Aug 2016 18:39:35 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x230.google.com with SMTP id d196so1142041wmd.0 for ; Thu, 11 Aug 2016 11:39:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=lASs5WUtLWAy6hJU67Y2m92KiThgZ70smfLDVyBMEXs=; b=KH0CDBgI59GfU1Gv969i/EOgxxjxYIJxyPDqv8J4dEhsv0d1EZefzGs4e9Wys+vCYn 41jNFf+owqcS2WG4zVjTCpmxjzmp6Abokg3v7LlPY1Qg3+VqaoAf6IZvfOtnyaFrElBV uIFeBcIKP7qM0UnTUDgMy7EcWxyRbgfRHT7fZw6BzJnG3Kc5ux60E6CYGcqeXPe/BacM 8VU0u61OwDyT1QP+jjec5HTQJWTgFpqB44Uc+w8Ibb1ZJQ+AbpxueWGARFNAeCFHk8hh h6XwLv24hIiqeSMYggV3sOjLjNicGHVvHNsYJHKTyMtgerx8mV35J8YjM0iEY4h48srI Fwvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=lASs5WUtLWAy6hJU67Y2m92KiThgZ70smfLDVyBMEXs=; b=KzK1xk/SDzPOKBA3ayAW0Q4x8FxG2jYDx1Z4qTxSB0lsirt7oM2x8X9Kwn0PFqRT+P lFByYyRbNAMsf48I21LD2hWJ2SwMuFxK3crwji0Ik6fkiWrpUwKZgi+yM0waC+aeWojU lA2H93HpDI2JUHbMok3W7gBgpe9Ov85pLmal04Yuz55tjKVOr0dlr1YxmWzmtnwtqJln aIzq9rpbA63feyzDNq7oLjHFrnHMKR1JW6PAWC1E9YVxcIT6GBrYu92rKFkNlQ7RT4el R4Zl0npnPBvMtyGLOVx353NEnFPWuWdRqNLuxtgeWRQ0QXA+6xjQEM4Pobrz1U2gTv1Y PYKw== X-Gm-Message-State: AEkoouvm2YUDPSAAnZjuygOYdU6KrU76f/dAzzMdVPPjO1x6CXd9yPS9KaPGuOQsRwDKBg== X-Received: by 10.195.2.42 with SMTP id bl10mr9416651wjd.21.1470940774011; Thu, 11 Aug 2016 11:39:34 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id ub8sm3899546wjc.39.2016.08.11.11.39.33 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 11 Aug 2016 11:39:33 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: Date: Thu, 11 Aug 2016 20:39:32 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <7DD30CE7-32E6-4D26-91D4-C1D4F2319655@gmail.com> To: FreeBSD Net X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2016 18:39:36 -0000 > On 11 Aug 2016, at 18:36, Adrian Chadd wrote: >=20 > Hi! >=20 > mlx4_core0: mem > 0xfbe00000-0xfbefffff,0xfb000000-0xfb7fffff irq 64 at device 0.0 > numa-domain 1 on pci16 > mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 > (Aug 11 2016) >=20 > so the NIC is in numa-domain 1. Try pinning the worker threads to > numa-domain 1 when you run the test: >=20 > numactl -l first-touch-rr -m 1 -c 1 ./test-program >=20 > You can also try pinning the NIC threads to numa-domain 1 versus 0 (so > the second set of CPUs, not the first set.) >=20 > vmstat -ia | grep mlx (get the list of interrupt thread ids) > then for each: >=20 > cpuset -d 1 -x >=20 > Run pcm-memory.x each time so we can see the before and after effects > on local versus remote memory access. >=20 > Thanks! Waiting for the correct commands to use, I made some tests with : cpuset -l 0-11 or cpuset -l 12-23 and : c=3D0 vmstat -ia | grep mlx | sed 's/^irq\(.*\):.*/\1/' | while read i do cpuset -l $c -x $i ; ((c++)) ; [[ $c -gt 11 ]] && c=3D0 done or=20 c=3D12 vmstat -ia | grep mlx | sed 's/^irq\(.*\):.*/\1/' | while read i do cpuset -l $c -x $i ; ((c++)) ; [[ $c -gt 23 ]] && c=3D12 done Results : No pinning http://pastebin.com/raw/CrK1CQpm Pinning workers to 0-11 Pinning NIC IRQ to 0-11 http://pastebin.com/raw/kLEQ6TKL Pinning workers to 12-23 Pinning NIC IRQ to 12-23 http://pastebin.com/raw/qGxw9KL2 Pinning workers to 12-23 Pinning NIC IRQ to 0-11 http://pastebin.com/raw/tFjii629 Comments : Strangely, the best iPer throughput results are when there is no = pinning. Whereas before running kernel with your new options, the best results = were with everything pinned to 0-11. Feel free to ask me further testing. Ben