Date: Sun, 12 Mar 2017 18:13:46 -0400 From: John Jasen <jjasen@gmail.com> To: "Caraballo-vega, Jordan A." <jordancaraballo87@gmail.com>, freebsd-net@freebsd.org Subject: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 Message-ID: <b460cc49-364a-34fb-e212-582730546dee@gmail.com> In-Reply-To: <6ad029e0-86c6-af3d-8fc3-694d4bcdc683@gmail.com> References: <ebb04a3e-bcde-6d50-af63-348e8d06fcba@gmail.com> <40a413f3-2c44-ee9d-9961-67114d8dffca@gmail.com> <20170205175531.GA20287@dwarf> <7d349edd-0c81-2e3f-d3b9-27af232de76d@gmail.com> <20170209153409.GG41673@dwarf> <6ad029e0-86c6-af3d-8fc3-694d4bcdc683@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I think I am able to confirm Mr. Caraballo's findings. I pulled a Dell PowerEdge 720 out of production, and upgraded it to 11-RELEASE-p8. Currently, as in the R530, it has a single Chelsio T5-580, but has two v2 Intel E5-26xx CPUs versus the newer ones in the R530. Both ports are configured for jumbo frames, and lro/tso are off. One is pointed at 172.16.2.0/24 as the load receivers; the other is pointed to 172.16.1.0/24 where the generators reside. Each side has 24 systems. I've played around a little with the number of queues, cpuset interrupt binding, and net.isr values -- the only differences were going from pathetic scores (1.7 million packets-per-second) to absolutely pathetic (1.3 million when QPI was hit). In these runs, it seems that no matter what we try on the system, not all the CPUs are engaged, and the receive queues are also unbalanced. As an example, in the last run, only 4 of the CPUs were engaged, and tracking rx queues using https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/usr/local/bin/n= ic-queue-usage, they ranges from 800k/second to 0/second, depending on the queues (this run used Chelsio defaults of 8 rx queues/16 tx queues). Interrupts also seem to confirm there is an unbalance, as current totals on the 'receive' chelsio port range from 935,000 to 9,200,000 (vmstat -ai). Any idea whats going on? On 02/27/2017 09:13 PM, Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER SCIENCE CORP] wrote: > As a summarywe have a Dell R530 with a Chelsio T580 cardwith -CURRENT.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b460cc49-364a-34fb-e212-582730546dee>