Date: Wed, 10 Aug 2016 15:11:40 -0700 From: Adrian Chadd <adrian.chadd@gmail.com> To: Ben RUBSON <ben.rubson@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Unstable local network throughput Message-ID: <CAJ-Vmo=Mfcvd41gtrt8GJfEtP-DQFfXt7pZ8eRLQzu73M=sX4A@mail.gmail.com> In-Reply-To: <B4D77A84-8F02-43E7-AD65-5B92423FC344@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <bed13ae3-0b8f-b1af-7418-7bf1b9fc74bc@selasky.org> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <BD0B68D1-CDCD-4E09-AF22-34318B6CEAA7@gmail.com> <CAJ-VmomW0Wth-uQU-OPTfRAsXW1kTDy-VyO2w-pgNosb-N1o=Q@mail.gmail.com> <B4D77A84-8F02-43E7-AD65-5B92423FC344@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10 August 2016 at 12:50, Ben RUBSON <ben.rubson@gmail.com> wrote: > >> On 10 Aug 2016, at 21:47, Adrian Chadd <adrian.chadd@gmail.com> wrote: >> >> hi, >> >> yeah, I'd like you to do some further testing with NUMA. Are you able >> to run freebsd-11 or -HEAD on these boxes? > > Hi Adrian, > > Yes I currently have 11 BETA3 running on them. > I could also run BETA4. hi, ok, lets start by getting the NUMA bits into the kernel so you can mess with things. add this to the kernel options MAXMEMDOM=8 (which hopefully is enough) options VM_NUMA_ALLOC options DEVICE_NUMA Then reboot and post your 'dmesg' output to the list. This should show exactly which domain devices are in. Install the 'intel-pcm' package. There's a 'pcm-numa.x' command - do kldload cpuctl, then run pcm-numa.x and see if it works. It should give us some useful information about NUMA. (Same as pcm-memory.x, pcm-pcie.x, etc.) Then next is playing around with interrupt thread / userland cpuset and memory affinity. We can look at that next. Currently the kernel doesn't know about NUMA local memory for device driver memory, kernel allocations for mbufs, etc, but we should still get a "good enough" idea about things. We can talk about that here once the above steps are done. Thanks! -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=Mfcvd41gtrt8GJfEtP-DQFfXt7pZ8eRLQzu73M=sX4A>