From owner-freebsd-net@freebsd.org Thu Aug 11 18:45:03 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04202BB6AC4 for ; Thu, 11 Aug 2016 18:45:03 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [199.48.133.146]) by mx1.freebsd.org (Postfix) with ESMTP id DEBCF1EB2 for ; Thu, 11 Aug 2016 18:45:02 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from sweettea.beer.town (unknown [76.164.8.130]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 3959556488; Thu, 11 Aug 2016 13:44:56 -0500 (CDT) Subject: Re: Unstable local network throughput To: Ben RUBSON , FreeBSD Net References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <7DD30CE7-32E6-4D26-91D4-C1D4F2319655@gmail.com> <246672E7-78D9-4421-9214-DE66C32D47FA@gmail.com> <0B3D077E-120B-471C-B2B5-1ABCD85E572E@gmail.com> From: Eric van Gyzen Message-ID: Date: Thu, 11 Aug 2016 13:44:55 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <0B3D077E-120B-471C-B2B5-1ABCD85E572E@gmail.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2016 18:45:03 -0000 On 08/11/16 12:54 PM, Ben RUBSON wrote: > >> On 11 Aug 2016, at 19:51, Ben RUBSON wrote: >> >> >>> On 11 Aug 2016, at 18:36, Adrian Chadd wrote: >>> >>> Hi! >> >> Hi Adrian, >> >>> mlx4_core0: mem >>> 0xfbe00000-0xfbefffff,0xfb000000-0xfb7fffff irq 64 at device 0.0 >>> numa-domain 1 on pci16 >>> mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 >>> (Aug 11 2016) >>> >>> so the NIC is in numa-domain 1. Try pinning the worker threads to >>> numa-domain 1 when you run the test: >>> >>> numactl -l first-touch-rr -m 1 -c 1 ./test-program >> >> # numactl -l first-touch-rr -m 1 -c 1 /usr/local/bin/iperf -c 192.168.2.1 -l 128KB -P 16 -i 2 -t 6000 >> Could not parse policy: '128KB' >> >> I did not manage to give arguments to command. Any idea ? > > I answer to myself, this should do the trick : > numactl -l first-touch-rr -m 1 -c 1 -- /usr/local/bin/iperf -c 192.168.2.1 -l 128KB -P 16 -i 2 -t 6000 This has annoyed me quite a bit, too. Setting the POSIXLY_CORRECT environment variable would also make it behave "correctly". > However of course it still gives the error below : > >> # numactl -l first-touch-rr -m 1 -c 1 /usr/local/bin/iperf >> numactl: numa_setaffinity: Invalid argument >> >> And sounds like -m is not allowed with first-touch-rr. >> What should I use ? Adrian probably meant fixed-domain-rr. >> Thank you ! >> >>> You can also try pinning the NIC threads to numa-domain 1 versus 0 (so >>> the second set of CPUs, not the first set.) >>> >>> vmstat -ia | grep mlx (get the list of interrupt thread ids) >>> then for each: >>> >>> cpuset -d 1 -x >>> >>> Run pcm-memory.x each time so we can see the before and after effects >>> on local versus remote memory access. >>> >>> Thanks! >>> >>> >>> >>> -adrian >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >