From owner-freebsd-net@freebsd.org Thu Aug 11 06:59:14 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51C78BB615B for ; Thu, 11 Aug 2016 06:59:14 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com [IPv6:2a00:1450:400c:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E0A071C0E for ; Thu, 11 Aug 2016 06:59:13 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x231.google.com with SMTP id i5so13516143wmg.0 for ; Wed, 10 Aug 2016 23:59:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=Qb0xKPuFjY7wojw3JQpi941kYwT0fLt4GByAIFpuSg4=; b=K3iNsdebuZqn1sxuPeq624Fui4yplEuw5tHXNRY8tSZD2X2lrHlDaK79/ogsolig3A 9lBHXLe232El1gpuOrZvNZjp/9I0aUGvR8jUxooe+4mPV1iCTlU+O7KrEcu9Vk3NvQF5 6ub6tmBgO25wLVAVzVVnGcKscl/qwtBqVpuJJ5caPdD86GjSk69tSQMhUNHguhoq+YfP 0rsCY3cpGljtQi1NExTm40V4lhG49qXgPMr+cLVlY1Aaf5jJ4dFWA3JtLZ4qGiGtbFnn Y8QBB5tTtg93RuGQGpO3AOilyvePuRynduqol13vM2VNqty579bUA44srKxWmcBBrLFU J0eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=Qb0xKPuFjY7wojw3JQpi941kYwT0fLt4GByAIFpuSg4=; b=KtqhgE0n5p6Ljh7hIoDuqFM9sXe1c2vwGn06zZ9L/vgddYyG+xIpE/AWuR6lOKHByu kFxhQc3phq2cI+FLttUBY0Zk0AZY+wrEaDPj0qj9vKP2pDl8spgn6d/ILEdzLHzEM656 OMzw631BnoWrcvdKk7OcSTIQBVHC4wU1aewuCZGv/8NUvLaYDbtLChJ3QbknLunZ8snp 5P4FzvZDMlsmJL0A5kMdOi3HJAqzfsNR/D0CHJQs4//ssQp1sAKd0UM8WVk+pbPdPI6a SANwcEEDN0etVQTYcINJYnXLhbh9oaj40iV+JmVV9DvuvesIXFJbpNRsRWTSYUDsoeVb H7jQ== X-Gm-Message-State: AEkoouuiONCVK54pwf4VG6mFR0JuUiE56hocZu1xIqXk31LixR8Ihs2GsAaQksIZFRWd1w== X-Received: by 10.28.229.1 with SMTP id c1mr7924995wmh.0.1470898750652; Wed, 10 Aug 2016 23:59:10 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id b123sm11864267wmg.17.2016.08.10.23.59.09 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Aug 2016 23:59:09 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: Date: Thu, 11 Aug 2016 08:59:09 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <7DD30CE7-32E6-4D26-91D4-C1D4F2319655@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> To: FreeBSD Net X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2016 06:59:14 -0000 > On 11 Aug 2016, at 00:11, Adrian Chadd wrote: >=20 > hi, >=20 > ok, lets start by getting the NUMA bits into the kernel so you can > mess with things. >=20 > add this to the kernel >=20 > options MAXMEMDOM=3D8 > (which hopefully is enough) > options VM_NUMA_ALLOC > options DEVICE_NUMA >=20 > Then reboot and post your 'dmesg' output to the list. This should show > exactly which domain devices are in. http://pastebin.com/raw/yaYEytME > Install the 'intel-pcm' package. There's a 'pcm-numa.x' command - do > kldload cpuctl, then run pcm-numa.x and see if it works. It should > give us some useful information about NUMA. > (Same as pcm-memory.x, pcm-pcie.x, etc.) Yes these tools work : # pcm-numa.x Intel(r) Performance Counter Monitor: NUMA monitoring utility=20 Copyright (c) 2009-2016 Intel Corporation Number of physical cores: 12 Number of logical cores: 24 Number of online logical cores: 24 Threads (logical cores) per physical core: 2 Num sockets: 2 Physical cores per socket: 6 Core PMU (perfmon) version: 3 Number of core PMU generic (programmable) counters: 4 Width of generic (programmable) counters: 48 bits Number of core PMU fixed counters: 3 Width of fixed counters: 48 bits Nominal core frequency: 2400000000 Hz Package thermal spec power: 85 Watt; Package minimum power: 31 Watt; = Package maximum power: 170 Watt;=20 ERROR: QPI LL monitoring device (0:127:9:2) is missing. The QPI = statistics will be incomplete or missing. Socket 0: 2 memory controllers detected with total number of 5 channels. = 1 QPI ports detected. ERROR: QPI LL monitoring device (0:255:9:2) is missing. The QPI = statistics will be incomplete or missing. Socket 1: 2 memory controllers detected with total number of 5 channels. = 1 QPI ports detected. Socket 0 Max QPI link 0 speed: 16.0 GBytes/second (8.0 GT/second) Socket 1 Max QPI link 0 speed: 16.0 GBytes/second (8.0 GT/second) Detected Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz "Intel(r) = microarchitecture codename Haswell-EP/EN/EX" Update every 1.0 seconds Time elapsed: 1010 ms Core | IPC | Instructions | Cycles | Local DRAM accesses | Remote = DRAM Accesses=20 0 0.70 1158 K 1655 K 577 245 = =20 1 0.33 186 K 557 K 160 15 = =20 2 0.43 317 K 745 K 385 31 = =20 3 0.36 260 K 718 K 232 33 = =20 4 0.31 186 K 602 K 188 11 = =20 5 0.39 314 K 806 K 371 43 = =20 6 0.36 235 K 659 K 257 46 = =20 7 0.35 200 K 576 K 133 44 = =20 8 0.42 423 K 1011 K 226 20 = =20 9 0.60 1309 K 2199 K 379 104 = =20 10 0.34 192 K 562 K 161 26 = =20 11 0.38 257 K 684 K 158 44 = =20 12 0.35 185 K 528 K 39 121 = =20 13 0.32 199 K 616 K 51 171 = =20 14 0.31 184 K 594 K 34 130 = =20 15 0.35 272 K 783 K 47 256 = =20 16 0.31 178 K 579 K 26 127 = =20 17 0.37 272 K 729 K 87 204 = =20 18 0.52 485 K 942 K 35 204 = =20 19 0.40 285 K 723 K 16 147 = =20 20 0.31 195 K 620 K 10 134 = =20 21 0.33 201 K 615 K 30 114 = =20 22 0.29 176 K 612 K 24 110 = =20 23 0.52 896 K 1716 K 86 895 = =20 = --------------------------------------------------------------------------= ----------------------------------------- * 0.43 8575 K 19 M 3712 3275 = =20 > Then next is playing around with interrupt thread / userland cpuset > and memory affinity. We can look at that next. Waiting for your instructions ! Ben