Date: Mon, 13 Mar 2017 02:18:26 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: John Jasen <jjasen@gmail.com> Cc: "Caraballo-vega, Jordan A." <jordancaraballo87@gmail.com>, freebsd-net@freebsd.org Subject: Re: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 Message-ID: <20170312231826.GV15630@zxy.spb.ru> In-Reply-To: <b460cc49-364a-34fb-e212-582730546dee@gmail.com> References: <ebb04a3e-bcde-6d50-af63-348e8d06fcba@gmail.com> <40a413f3-2c44-ee9d-9961-67114d8dffca@gmail.com> <20170205175531.GA20287@dwarf> <7d349edd-0c81-2e3f-d3b9-27af232de76d@gmail.com> <20170209153409.GG41673@dwarf> <6ad029e0-86c6-af3d-8fc3-694d4bcdc683@gmail.com> <b460cc49-364a-34fb-e212-582730546dee@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Mar 12, 2017 at 06:13:46PM -0400, John Jasen wrote: > I think I am able to confirm Mr. Caraballo's findings. > > I pulled a Dell PowerEdge 720 out of production, and upgraded it to > 11-RELEASE-p8. > > Currently, as in the R530, it has a single Chelsio T5-580, but has two > v2 Intel E5-26xx CPUs versus the newer ones in the R530. > > Both ports are configured for jumbo frames, and lro/tso are off. One is > pointed at 172.16.2.0/24 as the load receivers; the other is pointed to > 172.16.1.0/24 where the generators reside. Each side has 24 systems. > > I've played around a little with the number of queues, cpuset interrupt > binding, and net.isr values -- the only differences were going from > pathetic scores (1.7 million packets-per-second) to absolutely pathetic > (1.3 million when QPI was hit). > > In these runs, it seems that no matter what we try on the system, not > all the CPUs are engaged, and the receive queues are also unbalanced. As > an example, in the last run, only 4 of the CPUs were engaged, and > tracking rx queues using > https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/usr/local/bin/nic-queue-usage, > they ranges from 800k/second to 0/second, depending on the queues (this > run used Chelsio defaults of 8 rx queues/16 tx queues). Interrupts also > seem to confirm there is an unbalance, as current totals on the > 'receive' chelsio port range from 935,000 to 9,200,000 (vmstat -ai). > > Any idea whats going on? what traffic you generated (TCP? UDP? ICMP? other?), what reported in dmesg | grep txq ?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170312231826.GV15630>