Date: Tue, 2 Aug 2016 21:35:48 +0200 From: Hans Petter Selasky <hps@selasky.org> To: Ben RUBSON <ben.rubson@gmail.com>, freebsd-net@freebsd.org Subject: Re: Unstable local network throughput Message-ID: <bed13ae3-0b8f-b1af-7418-7bf1b9fc74bc@selasky.org> In-Reply-To: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 08/02/16 20:43, Ben RUBSON wrote: > Hello, > > I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a ConnectX-3 Mellanox network adapter. > > FreeBSD 10.3 just installed, last updates performed. > Network adapters running last firmwares / last drivers. > No workload at all, just iPerf as the benchmark tool. > Hi, The CX-3 driver doesn't bind the worker threads to specific CPU cores by default, so if your CPU has more than one so-called numa, you'll end up that the bottle-neck is the high-speed link between the CPU cores and not the card. A quick and dirty workaround is to "cpuset" iperf and the interrupt and taskqueue threads to specific CPU cores. Are you using "options RSS" and "options PCBGROUP" in your kernel config? Are you also testing CX-4 cards from Mellanox? --HPS
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bed13ae3-0b8f-b1af-7418-7bf1b9fc74bc>