From owner-freebsd-net@freebsd.org Sun Mar 12 22:13:49 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFB6CD0A25F for ; Sun, 12 Mar 2017 22:13:49 +0000 (UTC) (envelope-from jjasen@gmail.com) Received: from mail-qt0-x241.google.com (mail-qt0-x241.google.com [IPv6:2607:f8b0:400d:c0d::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 669911A25 for ; Sun, 12 Mar 2017 22:13:49 +0000 (UTC) (envelope-from jjasen@gmail.com) Received: by mail-qt0-x241.google.com with SMTP id r5so4405201qtb.2 for ; Sun, 12 Mar 2017 15:13:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:subject:to:references:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=0TaDUVD08i8JcHB9yYRSpkQWkvV83KDWkNZ1DctFjXk=; b=S6GyPLgkHv7bvhrT2sq41i5+8DMyStzJDcg3jQMtv4Xhz2hUeEZaZ8v0RQwWjWTzP+ 59epgzex/zF9pkrCDJdvN2vhVU6fC9H3C1uZVgZLBe0pAMzrCXieCkNRexaxlZge1afG 94zvtixNHL4NhLpJNdRtVGziQhCeAwt7Nrk6W+2u0Iz19rVuQvchFq0oPBA/p+pwxWOd Fs0d7P9sNn8sePlzDJnRe5s0jlFCOh+1Md9f8Or6A4Q76mk/sUqAkjjJZ5N+ZKFbVDy7 U1UL6GIdt6MhHBrnsTPwpz2MN+RZy3tK1pvAYnF1WLLo5i1SCDH3D9/WMTrffC1CZYU4 R6ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=0TaDUVD08i8JcHB9yYRSpkQWkvV83KDWkNZ1DctFjXk=; b=bEfU7pMEOdlDZd7stFDuKiqnDYBrc3XY0IJKG7gn7R4+OAm+/QOPZ3Qzu2C0yRjEPM 88T8avpOPMxw6Ueag7r+ag7FJz/jLH+cg0LQ4t/r1nShq6hKwsgxBG77kni6nekhpLy0 zrTpsq4BG8msxoqzH+RSzpDW6vQ/gehtPCZ6Z3r9BZoWAPkJ2Ew/roohAKx1UcTxhtpg xzp6mfWRdK/sHrxFCi+8TAqPyJpRJAVdXKHs3e0VgvTW+Tb9MvYrS0YNQeZZNeVX4w8Y 5rm4uA7ynZL8woCRzcnDda3VTo70HN+++yG3VI5B1kNkO6GWVnB56xKA+a9rClLwERnn sgiw== X-Gm-Message-State: AMke39mLfavismZ9pIrU8FcWOaWg5Q8rT8aceQogRvopnNB1/x2ig4p1D4QUlrjU/L/HPw== X-Received: by 10.237.55.99 with SMTP id i90mr28985854qtb.262.1489356828487; Sun, 12 Mar 2017 15:13:48 -0700 (PDT) Received: from [10.0.0.230] (pool-71-244-225-114.bltmmd.fios.verizon.net. [71.244.225.114]) by smtp.googlemail.com with ESMTPSA id e5sm10992795qtc.6.2017.03.12.15.13.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 12 Mar 2017 15:13:47 -0700 (PDT) From: John Jasen Subject: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 To: "Caraballo-vega, Jordan A." , freebsd-net@freebsd.org References: <40a413f3-2c44-ee9d-9961-67114d8dffca@gmail.com> <20170205175531.GA20287@dwarf> <7d349edd-0c81-2e3f-d3b9-27af232de76d@gmail.com> <20170209153409.GG41673@dwarf> <6ad029e0-86c6-af3d-8fc3-694d4bcdc683@gmail.com> Message-ID: Date: Sun, 12 Mar 2017 18:13:46 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <6ad029e0-86c6-af3d-8fc3-694d4bcdc683@gmail.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Mar 2017 22:13:49 -0000 I think I am able to confirm Mr. Caraballo's findings. I pulled a Dell PowerEdge 720 out of production, and upgraded it to 11-RELEASE-p8. Currently, as in the R530, it has a single Chelsio T5-580, but has two v2 Intel E5-26xx CPUs versus the newer ones in the R530. Both ports are configured for jumbo frames, and lro/tso are off. One is pointed at 172.16.2.0/24 as the load receivers; the other is pointed to 172.16.1.0/24 where the generators reside. Each side has 24 systems. I've played around a little with the number of queues, cpuset interrupt binding, and net.isr values -- the only differences were going from pathetic scores (1.7 million packets-per-second) to absolutely pathetic (1.3 million when QPI was hit). In these runs, it seems that no matter what we try on the system, not all the CPUs are engaged, and the receive queues are also unbalanced. As an example, in the last run, only 4 of the CPUs were engaged, and tracking rx queues using https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/usr/local/bin/n= ic-queue-usage, they ranges from 800k/second to 0/second, depending on the queues (this run used Chelsio defaults of 8 rx queues/16 tx queues). Interrupts also seem to confirm there is an unbalance, as current totals on the 'receive' chelsio port range from 935,000 to 9,200,000 (vmstat -ai). Any idea whats going on? On 02/27/2017 09:13 PM, Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER SCIENCE CORP] wrote: > As a summarywe have a Dell R530 with a Chelsio T580 cardwith -CURRENT.