From owner-freebsd-net@freebsd.org Thu Aug 11 22:35:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4157FBB77C1 for ; Thu, 11 Aug 2016 22:35:23 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C51A0198B for ; Thu, 11 Aug 2016 22:35:22 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x232.google.com with SMTP id q128so17364885wma.1 for ; Thu, 11 Aug 2016 15:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=47jJgd/f7A310VLX+O3ofGdxakP3eRggquXR8SNh59o=; b=R07K5jZ/6nAciGC1IvpahocT/93qnEmetvNrreJ4lu1YjlaCuH4D3Lcr1hmI1MZ4ld MRvTFK5VAWh7F42ynoM05VazGUAdiCXK37Z1Tt5XihcEPssiynLo2wHBgsTHB65sj6Pe jPPHiSH83TUSB8zhu+7RQpYWESZtUbDyYiFSo+lZjG3LXfARe4kvYNjhTFpo3DfesWFs XratP2jBpavJ7R4UdhTB9ICCJSlWEAy/Uzti6xAU66VD8DLWMj6IP/kLO13QxqyvK1XY RWQ9xjqPzMJ9QGQFa3uHSFbk/6IxZgt9F/WhaZrhkOPRApPQTdzbxMUVvlDsR7fOLbxm LPng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=47jJgd/f7A310VLX+O3ofGdxakP3eRggquXR8SNh59o=; b=XIkSeGIxKXilA9D4yzeo6OvfnDmn4208phq9qUC1Dh06Gg578tk00Q5Uw6ESx+5IuC wBeZB1qAq+tK9s3j6Vn0IMsEZzJTRIccjj0shoNCeSJraGVKWzJQIFxYmGhvFiXI9S7s wM6hGBrzVmuTBg4E3H5YBOtuAoRwkZ3fGUuKHL9EdQF4WXRU8+0zBPupBb4QiWj/jn+3 20w4fKkfbHSdABiaeN6UMuMMQSsQhQD3dAmEYkEzXaR/oGMhxJIXO9GX4JGXef5e463D oirUCSjHXSuHHZVnpTClkZhpnaX1wxL+tjbMHHc99cZlonDb36V4A7+KSEUgMboIhhII EKxw== X-Gm-Message-State: AEkoouuORZSBr5h8TK8rXjkUhRHFuD3NEJN58YH3vFQkrEc9KyoNoanVbviK7GZqM4j2tQ== X-Received: by 10.28.127.138 with SMTP id a132mr11070241wmd.72.1470954920763; Thu, 11 Aug 2016 15:35:20 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id c139sm2095098wme.4.2016.08.11.15.35.19 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 11 Aug 2016 15:35:20 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: Date: Fri, 12 Aug 2016 00:35:18 +0200 Content-Transfer-Encoding: 7bit Message-Id: References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <7DD30CE7-32E6-4D26-91D4-C1D4F2319655@gmail.com> To: FreeBSD Net X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2016 22:35:23 -0000 > On 11 Aug 2016, at 18:36, Adrian Chadd wrote: > > Hi! > > mlx4_core0: mem > 0xfbe00000-0xfbefffff,0xfb000000-0xfb7fffff irq 64 at device 0.0 > numa-domain 1 on pci16 > mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 > (Aug 11 2016) > > so the NIC is in numa-domain 1. Try pinning the worker threads to > numa-domain 1 when you run the test: > > numactl -l first-touch-rr -m 1 -c 1 ./test-program > > You can also try pinning the NIC threads to numa-domain 1 versus 0 (so > the second set of CPUs, not the first set.) > > vmstat -ia | grep mlx (get the list of interrupt thread ids) > then for each: > > cpuset -d 1 -x > > Run pcm-memory.x each time so we can see the before and after effects > on local versus remote memory access. > > Thanks! Adrian, here are the results : Idle system : http://pastebin.com/raw/K1iMVHVF No pinning : http://pastebin.com/raw/w5KuexQ3 CPU : http://pastebin.com/raw/8zgRaazN numactl -l fixed-domain-rr -m 1 -c 1 : http://pastebin.com/raw/VWweYF9H CPU : http://pastebin.com/raw/QjaVH32X numactl -l fixed-domain-rr -m 0 -c 0 : http://pastebin.com/raw/71hfGJdw CPU : http://pastebin.com/raw/hef058Na numactl -l fixed-domain-rr -m 1 -c 1 + cpuset -l -x : http://pastebin.com/raw/nEQkgMK2 CPU : http://pastebin.com/raw/R652KAdJ numactl -l fixed-domain-rr -m 0 -c 0 + cpuset -l -x : http://pastebin.com/raw/GdYJHyae CPU : http://pastebin.com/raw/Ggfx9uF9 No pinning, default kernel (no NUMA option) : http://pastebin.com/raw/iQ2u8d8k CPU : http://pastebin.com/raw/Xr77KpcM default kernel (no NUMA option) + cpuset -l + cpuset -l -x : http://pastebin.com/raw/VBWg4SZs default kernel (no NUMA option) + cpuset -l + cpuset -l -x : http://pastebin.com/raw/SrJLZxuT No pinning, default kernel (no NUMA option), NUMA BIOS disabled : http://pastebin.com/raw/P5LrUASN I would say : - FreeBSD <= 10.3 : disable NUMA in BIOS - FreeBSD >= 11 : disable NUMA in BIOS or enable NUMA in kernel. But let's wait your analysis :) Ben