Date: Tue, 16 Aug 2016 12:36:45 -0700 From: Adrian Chadd <adrian.chadd@gmail.com> To: Ben RUBSON <ben.rubson@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Unstable local network throughput Message-ID: <CAJ-Vmon8dkUzm-FtwLNU307QhZhJsEEvMp=JQOJ-Rq30Uos7Pg@mail.gmail.com> In-Reply-To: <5EAA9DC3-9A76-4F67-9A03-31FFCB08DF7D@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <bed13ae3-0b8f-b1af-7418-7bf1b9fc74bc@selasky.org> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <BD0B68D1-CDCD-4E09-AF22-34318B6CEAA7@gmail.com> <CAJ-VmomW0Wth-uQU-OPTfRAsXW1kTDy-VyO2w-pgNosb-N1o=Q@mail.gmail.com> <B4D77A84-8F02-43E7-AD65-5B92423FC344@gmail.com> <CAJ-Vmo=Mfcvd41gtrt8GJfEtP-DQFfXt7pZ8eRLQzu73M=sX4A@mail.gmail.com> <7DD30CE7-32E6-4D26-91D4-C1D4F2319655@gmail.com> <CAJ-VmongwvbY3QqKBV%2BFJCHOfSdr-=v9CmLH1z=Tqwz19AtUpg@mail.gmail.com> <AF923C63-2414-4DCE-9FD9-CAE02E3AC8CE@gmail.com> <CAJ-VmonL8kVs3=BBg02cbzXA9NpAh-trdCBh4qkjw29dOCau-g@mail.gmail.com> <91AEB1BD-44EA-43AD-A9A1-6DEBF367DF9B@gmail.com> <CAJ-VmonTQApYBZ6Zc8Hr3ZvrSKmvg2ix=rvWwwBH3JyVSUQifg@mail.gmail.com> <5EAA9DC3-9A76-4F67-9A03-31FFCB08DF7D@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 16 August 2016 at 02:58, Ben RUBSON <ben.rubson@gmail.com> wrote: > >> On 16 Aug 2016, at 03:45, Adrian Chadd <adrian.chadd@gmail.com> wrote: >> >> Hi, >> >> ok, can you try 5) but also running with the interrupt threads pinned to CPU 1? > > What do you mean by interrupt threads ? > > Perhaps you mean the NIC interrupts ? > In this case see 6) and 7) where NIC IRQs are pinned to CPUs 0-11 (6) and 11-23 (7) ? Hm, interesting. ok. So, I wonder what the maximum per-domain memory throughput is. I don't have any other easy things to instrument right now - the "everything disabled" method likely works best because of how the system is interleaving memory for you (instead of the OS trying to do it). Not pinning things means latency can be kept down to work around lock contention (ie, if a lock is held by thread A, and thread B needs to make some progress, it can make progress on another CPU , keeping CPU A held for a shorter period of time.) Would you mind compiling in LOCK_PROFILING and doing say, these tests with lock profiling enabled? It'll impact performance, sure, but I'd like to see what the locking looks like. sysctl debug.lock.prof.reset=1 sysctl debug.lock.prof.enable=1 (run test for a few seconds) sysctl debug.lock.prof.enable=0 sysctl debug.lock.prof.stats (and capture) * interrupts - domain 0, work - domain 1 * interrupts - domain 1, work - domain 1 * interrupts - domain 1, work - domain 0 Thanks! -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmon8dkUzm-FtwLNU307QhZhJsEEvMp=JQOJ-Rq30Uos7Pg>