From owner-freebsd-net@FreeBSD.ORG Sat Jul 12 00:58:28 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EED35E20 for ; Sat, 12 Jul 2014 00:58:27 +0000 (UTC) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com [IPv6:2607:f8b0:4001:c05::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA8332A62 for ; Sat, 12 Jul 2014 00:58:27 +0000 (UTC) Received: by mail-ig0-f176.google.com with SMTP id r10so45038igi.15 for ; Fri, 11 Jul 2014 17:58:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=vQ4nFFjxfmo/2ROBBnmvQNwkrkGxf3pK64OC8ll96tQ=; b=fYogbGntyAiyedjZOHO3nWP6HjmvtqbNwKsvxJpHuMp/4lO6RdhqRXpKLcMBUzfh1D W+bF1XN65Xj7oMuvY5sGiWAz+9oFZHrbK5EPjCqYzZcife1BIEGJQrO/1CtH3yn07UqE 3mzUogFB771Js363NqFe4tFNnNCuYh+vGrMEglqlqWT+zJneBUraa1yce7x7YwRYyEYZ lnaYTwCNTWMy3s5EShcTAfTxao1v5l6Q0IEpxhb/Bp7/s8ZINcquehh1Wcj9A+R/dkwx ck3N8GumdYeYPXsg05PZGqVtmIAenPIq9yue7EQP+WqscXe36efkaljT9uqs0dt8aQCw +ulA== X-Received: by 10.50.112.132 with SMTP id iq4mr8826387igb.45.1405126707002; Fri, 11 Jul 2014 17:58:27 -0700 (PDT) Received: from [10.0.0.215] ([96.234.167.12]) by mx.google.com with ESMTPSA id ro10sm661350igb.18.2014.07.11.17.58.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 11 Jul 2014 17:58:23 -0700 (PDT) Message-ID: <53C0882D.5070100@gmail.com> Date: Fri, 11 Jul 2014 20:58:21 -0400 From: John Jasem User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Navdeep Parhar , FreeBSD Net Subject: Re: tuning routing using cxgbe and T580-CR cards? References: <53C01EB5.6090701@gmail.com> <53C03BB4.2090203@gmail.com> In-Reply-To: <53C03BB4.2090203@gmail.com> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jul 2014 00:58:28 -0000 On 07/11/2014 03:32 PM, Navdeep Parhar wrote: > On 07/11/14 10:28, John Jasem wrote: >> In testing two Chelsio T580-CR dual port cards with FreeBSD 10-STABLE, >> I've been able to use a collection of clients to generate approximately >> 1.5-1.6 million TCP packets per second sustained, and routinely hit >> 10GB/s, both measured by netstat -d -b -w1 -W (I usually use -h for the >> quick read, accepting the loss of granularity). > When forwarding, the pps rate is often more interesting, and almost > always the limiting factor, as compared to the total amount of data > being passed around. 10GB at this pps probably means 9000 MTU. Try > with 1500 too if possible. Yes, I am generally more interested/concerned with the pps. Using 1500-sized packets, I've seen around 2 million pps. I'll have hard numbers for the list, with netstat and vmstat output Monday. >> a) One of the first things I did in prior testing was to turn >> hyperthreading off. I presume this is still prudent, as HT doesn't help >> with interrupt handling? > It is always worthwhile to try your workload with and without > hyperthreading. Testing Mellanox cards, HT was severely detrimental. However, in almost every case so far, Mellanox and Chelsio have resulted in opposite conclusions (cpufreq, net.isr.*). >> c) the defaults for the cxgbe driver appear to be 8 rx queues, and N tx >> queues, with N being the number of CPUs detected. For a system running >> multiple cards, routing or firewalling, does this make sense, or would >> balancing tx and rx be more ideal? And would reducing queues per card >> based on NUMBER-CPUS and NUM-CHELSIO-PORTS make sense at all? > The defaults are nrxq = min(8, ncores) and ntxq = min(16, ncores). The > man page mentions this. The reason for 8 vs. 16 is that tx queues are > "cheaper" as they don't have to be backed by rx buffers. It only needs > some memory for the tx descriptor ring and some hardware resources. > > It appears that your system has >= 16 cores. For forwarding it probably > makes sense to have nrxq = ntxq. If you're left with 8 or fewer cores > after disabling hyperthreading you'll automatically get 8 rx and tx > queues. Otherwise you'll have to fiddle with the hw.cxgbe.nrxq10g and > ntxq10g tunables (documented in the man page). I promise I did look through the man page before posting. :) This is actually a 12 core box with HT turned off. Mining the cxl stat entries in sysctl, it appears that the queues per port are reasonably well balanced, so I may be concerned over nothing. >> g) Are there other settings I should be looking at, that may squeeze out >> a few more packets? > The pps rates that you've observed are within the chip's hardware limits > by at least an order of magnitude. Tuning the kernel rather than the > driver may be the best bang for your buck. If I am missing obvious configurations for kernel tuning in this regard, it would not the be the first time. Thanks again! -- John Jasen (jjasen@gmail.com)