From owner-freebsd-net@freebsd.org Sat Mar 25 00:51:21 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BACBFD1BF59 for ; Sat, 25 Mar 2017 00:51:21 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-wm0-x243.google.com (mail-wm0-x243.google.com [IPv6:2a00:1450:400c:c09::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 45686113A for ; Sat, 25 Mar 2017 00:51:21 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-wm0-x243.google.com with SMTP id x124so1004758wmf.3 for ; Fri, 24 Mar 2017 17:51:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=Cb9YX4JrRAsq5zqJWuxoikqgJ00f4NWgx8jh4icAktY=; b=EOIlW8zl+lcyTfk1vgqbyQUhfX2aXug52GC90v8Z2B+jBJAWjQbk7RXgHgjMYH0ORJ mRltHI3jucu6ti+dcc8QUPwBqYi1F0cpkXtJXAkJV/MIlx6Yuv+ovqSZtmsA/sO90WzP aOd3h3z6HgCWop/vjFoQf6RbTfK3D9lH+DN7qvnwMY3r4lO0QLwYzGpzxHxgAgNc1YLy nahaG+WahbOCPdm6pB3iscZ+5Tz4fxeQ2Iy8A8BFqO3el6fZAQz355/lQDW+GM2xroMw y+BOkvEgnkQKuuojyjntBNppdSkMRbJmgBBJ6PpGH/IBp+qtz0K+h2ZwkQTh1ZGc6U2P uXsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:cc:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=Cb9YX4JrRAsq5zqJWuxoikqgJ00f4NWgx8jh4icAktY=; b=M6XNgEMKeWKnXC66UbOoIrAgepBkdGddgXjK7D4jJhmaTopFPgkOb4bWX4BCjVvaHE TMUUR9pLmDc3cBijNnnDmXw1BjQyh6du34KcFb74B3+RFB8+y13LR+3m+g76sNLiDqkI z973qAaEmMolcuvxtRclpa4faWEKmkCsdDpeU16DaU/APIFmlJFyvZU2Islr6XoXdL4k yu+9HrIpTluelRxjLnbOmmgfK51rKazwql/3LcD5ksKULiRz+2K3sHbz+Q0pk60JSb0M SDzg6n28hUJa8184t+QAi5WmO8EU4AmzbJn9v9CoFkgowQBjgkBRZ7C5Q65iN8onLzpj idxQ== X-Gm-Message-State: AFeK/H3A81umC8kB7e2rHYf+C2DhG7S0IaoAKLDu6d1pNSwlHQ2W4WBFBRqcs3cNUuM0Jw== X-Received: by 10.28.126.133 with SMTP id z127mr28272wmc.60.1490403079679; Fri, 24 Mar 2017 17:51:19 -0700 (PDT) Received: from [10.192.166.0] (stargate.chelsio.com. [12.32.117.8]) by smtp.googlemail.com with ESMTPSA id h65sm4853511wrh.32.2017.03.24.17.51.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Mar 2017 17:51:18 -0700 (PDT) Sender: Navdeep Parhar Subject: Re: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 To: "Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER SCIENCE CORP]" Cc: freebsd-net@freebsd.org, slw@zxy.spb.ru, "jjasen@gmail.com >> John Jasen" References: <20170312231826.GV15630@zxy.spb.ru> <74654520-b8b6-6118-2e46-902a8ea107ac@gmail.com> <173fffac-7ae2-786a-66c0-e9cd7ab78f44@gmail.com> <20170317100814.GN70430@zxy.spb.ru> <9924b2d5-4a72-579c-96c6-4dbdacc07c95@gmail.com> <9694e9f2-daec-924d-e9f6-7b22a634acb5@gmail.com> <20170318052837.GA21730@ox> <0a4e3073-bf5f-9bf8-533f-bd9ec3c0f60c@FreeBSD.org> From: Navdeep Parhar Message-ID: Date: Fri, 24 Mar 2017 17:51:15 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2017 00:51:21 -0000 On 03/24/2017 16:53, Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER SCIENCE CORP] wrote: > It looks like netmap is there; however, is there a way of figuring out > if netmap is being used? If you're not running netmap-fwd or some other netmap application, it's not being used. You have just 1 txq/rxq and that would explain the difference between cxl and vcxl. > cxl0: 16 txq, 8 rxq (NIC) > vcxl0: 1 txq, 1 rxq (NIC); 2 txq, 2 rxq (netmap) > ... > And yes, we are using UDP 64 bytes tests. That's strange then. The "input packets" counter counts every single frame that the chip saw on the wire that matched any of its MAC addresses, including frames that the chip drops. There's no way to explain why vcxl sees ~640K pps incoming vs. 2.8M pps for cxl. That number shouldn't depend on your router configuration at all -- it's entirely dependent on the traffic generators. Are you sure you aren't getting PAUSE frames out of the chip? There's nothing else that could slow down UDP senders. # sysctl -a | grep tx_pause Regards, Navdeep > > On 3/24/17 7:39 PM, Navdeep Parhar wrote: >> On 03/24/2017 16:07, Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER >> SCIENCE CORP] wrote: >>> At the time of implementing the vcxl* interfaces we get very bad >>> results. >> >> You're probably not using netmap with the vcxl interfaces, and the >> number of "normal" tx and rx queues is just 2 for these interfaces. >> >> Even if you _are_ using netmap, the hw.cxgbe.nnmtxq10g/rxq10g tunables >> don't work anymore. Use these to control the number of queues for >> netmap: >> hw.cxgbe.nnmtxq_vi >> hw.cxgbe.nnmrxq_vi >> >> You should see a line like this in dmesg for all cxl/vcxl interfaces >> and that tells you exactly how many queues the driver configured: >> cxl0: 4 txq, 4 rxq (NIC); 4 txq, 2 rxq (TOE) >> >>> >>> packets errs idrops bytes packets errs bytes colls drops >>> 629k 4.5k 0 66M 629k 0 66M >>> 0 0 >>> 701k 5.0k 0 74M 701k 0 74M >>> 0 0 >>> 668k 4.8k 0 70M 668k 0 70M >>> 0 0 >>> 667k 4.8k 0 70M 667k 0 70M >>> 0 0 >>> 645k 4.5k 0 68M 645k 0 68M >>> 0 0 >>> 686k 4.9k 0 72M 686k 0 72M >>> 0 0 >>> >>> And by using just the cxl* interfaces we were getting about >>> >>> input (Total) output >>> packets errs idrops bytes packets errs bytes colls >>> drops >>> 2.8M 0 1.2M 294M 1.6M 0 171M >>> 0 0 >>> 2.8M 0 1.2M 294M 1.6M 0 171M >>> 0 0 >>> 2.8M 0 1.2M 294M 1.6M 0 171M >>> 0 0 >>> 2.8M 0 1.2M 295M 1.6M 0 172M >>> 0 0 >>> 2.8M 0 1.2M 295M 1.6M 0 171M >>> 0 0 >>> >>> These are our configurations for now. Any advice or suggestion will be >>> appreciated. >> >> What I don't understand is that you have PAUSE disabled and congestion >> drops enabled but still the number of packets coming in (whether they >> are dropped eventually or not is irrelevant here) is very low in your >> experiments. It's almost as if the senders are backing off in the >> face of packet loss. Are you using TCP or UDP? Always use UDP for >> pps testing -- the senders need to be relentless. >> >> Regards, >> Navdeep >> >>> >>> /etc/rc.conf configurations >>> >>> ifconfig_cxl0="up" >>> ifconfig_cxl1="up" >>> ifconfig_vcxl0="inet 172.16.2.1/24 -tso -lro mtu 9000" >>> ifconfig_vcxl1="inet 172.16.1.1/24 -tso -lro mtu 9000" >>> gateway_enable="YES" >>> >>> /boot/loader.conf configurations >>> >>> # Chelsio Modules >>> t4fw_cfg_load="YES" >>> t5fw_cfg_load="YES" >>> if_cxgbe_load="YES" >>> >>> # rx and tx size >>> dev.cxl.0.qsize_txq=8192 >>> dev.cxl.0.qsize_rxq=8192 >>> dev.cxl.1.qsize_txq=8192 >>> dev.cxl.1.qsize_rxq=8192 >>> >>> # drop toecaps to increase queues >>> dev.t5nex.0.toecaps=0 >>> dev.t5nex.0.rdmacaps=0 >>> dev.t5nex.0.iscsicaps=0 >>> dev.t5nex.0.fcoecaps=0 >>> >>> # Controls the hardware response to congestion. -1 disables >>> # congestion feedback and is not recommended. 0 instructs the >>> # hardware to backpressure its pipeline on congestion. This >>> # usually results in the port emitting PAUSE frames. 1 instructs >>> # the hardware to drop frames destined for congested queues. From cxgbe >>> dev.t5nex.0.cong_drop=1 >>> >>> # Saw these recomendations in Vicenzo email thread >>> hw.cxgbe.num_vis=2 >>> hw.cxgbe.fl_pktshift=0 >>> hw.cxgbe.toecaps_allowed=0 >>> hw.cxgbe.nnmtxq10g=8 >>> hw.cxgbe.nnmrxq10g=8 >>> >>> /etc/sysctl.conf configurations >>> >>> # Turning off pauses >>> dev.cxl.0.pause_settings=0 >>> dev.cxl.1.pause_settings=0 >>> # John Jasen suggestion - March 24, 2017 >>> net.isr.bindthreads=0 >>> net.isr.maxthreads=24 >>> >>> >>> On 3/18/17 1:28 AM, Navdeep Parhar wrote: >>>> On Fri, Mar 17, 2017 at 11:43:32PM -0400, John Jasen wrote: >>>>> On 03/17/2017 03:32 PM, Navdeep Parhar wrote: >>>>> >>>>>> On Fri, Mar 17, 2017 at 12:21 PM, John Jasen >>>>>> wrote: >>>>>>> Yes. >>>>>>> We were hopeful, initially, to be able to achieve higher packet >>>>>>> forwarding rates through either netmap-fwd or due to enhancements >>>>>>> based >>>>>>> off https://wiki.freebsd.org/ProjectsRoutingProposal >>>>>> Have you tried netmap-fwd? I'd be interested in how that did in >>>>>> your tests. >>>>> We have. On this particular box, (11-STABLE, netmap-fwd fresh from >>>>> git) >>>>> it took about 1.7m pps in, dropped 500k, and passed about 800k. >>>>> >>>>> I'm lead to believe that vcxl interfaces may yield better results? >>>> Yes, those are the ones with native netmap support. Any netmap based >>>> application should use the vcxl interfaces. If you used them on the >>>> main cxl interfaces you were running netmap in emulated mode. >>>> >>>> Regards, >>>> Navdeep >>> >> >