From owner-freebsd-net@freebsd.org Sun Mar 12 23:18:36 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02BE5D0AE35 for ; Sun, 12 Mar 2017 23:18:36 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B6C481062 for ; Sun, 12 Mar 2017 23:18:35 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1cnCks-000G17-8n; Mon, 13 Mar 2017 02:18:26 +0300 Date: Mon, 13 Mar 2017 02:18:26 +0300 From: Slawa Olhovchenkov To: John Jasen Cc: "Caraballo-vega, Jordan A." , freebsd-net@freebsd.org Subject: Re: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 Message-ID: <20170312231826.GV15630@zxy.spb.ru> References: <40a413f3-2c44-ee9d-9961-67114d8dffca@gmail.com> <20170205175531.GA20287@dwarf> <7d349edd-0c81-2e3f-d3b9-27af232de76d@gmail.com> <20170209153409.GG41673@dwarf> <6ad029e0-86c6-af3d-8fc3-694d4bcdc683@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Mar 2017 23:18:36 -0000 On Sun, Mar 12, 2017 at 06:13:46PM -0400, John Jasen wrote: > I think I am able to confirm Mr. Caraballo's findings. > > I pulled a Dell PowerEdge 720 out of production, and upgraded it to > 11-RELEASE-p8. > > Currently, as in the R530, it has a single Chelsio T5-580, but has two > v2 Intel E5-26xx CPUs versus the newer ones in the R530. > > Both ports are configured for jumbo frames, and lro/tso are off. One is > pointed at 172.16.2.0/24 as the load receivers; the other is pointed to > 172.16.1.0/24 where the generators reside. Each side has 24 systems. > > I've played around a little with the number of queues, cpuset interrupt > binding, and net.isr values -- the only differences were going from > pathetic scores (1.7 million packets-per-second) to absolutely pathetic > (1.3 million when QPI was hit). > > In these runs, it seems that no matter what we try on the system, not > all the CPUs are engaged, and the receive queues are also unbalanced. As > an example, in the last run, only 4 of the CPUs were engaged, and > tracking rx queues using > https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/usr/local/bin/nic-queue-usage, > they ranges from 800k/second to 0/second, depending on the queues (this > run used Chelsio defaults of 8 rx queues/16 tx queues). Interrupts also > seem to confirm there is an unbalance, as current totals on the > 'receive' chelsio port range from 935,000 to 9,200,000 (vmstat -ai). > > Any idea whats going on? what traffic you generated (TCP? UDP? ICMP? other?), what reported in dmesg | grep txq ?