From owner-freebsd-net@freebsd.org Fri Mar 24 23:53:11 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A798BD1CA96 for ; Fri, 24 Mar 2017 23:53:11 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: from mail-yw0-x244.google.com (mail-yw0-x244.google.com [IPv6:2607:f8b0:4002:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C9A2103B; Fri, 24 Mar 2017 23:53:11 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: by mail-yw0-x244.google.com with SMTP id i203so517486ywc.3; Fri, 24 Mar 2017 16:53:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=iwNUDvblU3hMLYCNfabXW+Pq6/jZT4mmXUqC6yluLXg=; b=P4+Vur9DFGurYTxF3+5c8H268aMQ6uKb9VVHq+X6ubj/zB/bFlvfqEo54IhDaa19Mc 8WcN0hIEfNKLyQY/wV8Fg+uXYQYSTQTP858squeNZHigNZhman07k3bW/Zdxjz+e92vG gvVK+SG3h7rwV7J0vc40vFrrd81XKc4JqqCWT/sM5ivJy1I93IcXNUq7KHVm23/3Q9oy Gb5Qk6225CyRVFO97qZ62nuCSVdrZXIxeKnVjIMJ3oTBGjJl4r5XxSOZ3BhGCLJwSdjS jRDYKytWPyQI51xm6xW8gXTU9QexgQJfTSkV0AHvWE3gx6XjejUyw7yhPnogyqw03r44 asNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=iwNUDvblU3hMLYCNfabXW+Pq6/jZT4mmXUqC6yluLXg=; b=iSaG9WcJBXTRKnVgV3cxPG7I8mXtvCGAU0WNzv2TuWm28ECIWLImQzaZMgYLxCX53g wPBSWi6FYfVPyti2NsayMxr7EtrcLQno6t8Rvm+j1GmIakgtx0SG2+C6aanUIc9jquuB hODZ30gSaJUXqnVYgQ2gfJewHHyIJgJ0mwiUf9HIsB2zu0H5Df6ug1SW3w+Ve5D3mZWY ikRBzghnq51DNTfBGUoBqpvVRWSPfwRLxxF7SJ0lh18oQb6dNivkQFfgHYNF+h/yhQwt v06X26GuCbGP7cVhV39oR2se2ZWP9c09ZvHY4aaBUPjCiz896UcMCOeC50Q4tm1p8q9d s3Jw== X-Gm-Message-State: AFeK/H1MP9Un4BFoJVzY+LWDSE/r3rXqXy+hkaO2DVCRfStsBxyvHHUVzUwqbPQ2iI4+Yw== X-Received: by 10.129.146.211 with SMTP id j202mr8670606ywg.41.1490399590364; Fri, 24 Mar 2017 16:53:10 -0700 (PDT) Received: from [10.0.0.67] (adsl-64-237-185-120.prtc.net. [64.237.185.120]) by smtp.gmail.com with ESMTPSA id 1sm1460227ywb.72.2017.03.24.16.53.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Mar 2017 16:53:09 -0700 (PDT) Subject: Re: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 To: Navdeep Parhar References: <20170312231826.GV15630@zxy.spb.ru> <74654520-b8b6-6118-2e46-902a8ea107ac@gmail.com> <173fffac-7ae2-786a-66c0-e9cd7ab78f44@gmail.com> <20170317100814.GN70430@zxy.spb.ru> <9924b2d5-4a72-579c-96c6-4dbdacc07c95@gmail.com> <9694e9f2-daec-924d-e9f6-7b22a634acb5@gmail.com> <20170318052837.GA21730@ox> <0a4e3073-bf5f-9bf8-533f-bd9ec3c0f60c@FreeBSD.org> Cc: freebsd-net@freebsd.org, slw@zxy.spb.ru, "jjasen@gmail.com >> John Jasen" From: "Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER SCIENCE CORP]" Message-ID: Date: Fri, 24 Mar 2017 19:53:07 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <0a4e3073-bf5f-9bf8-533f-bd9ec3c0f60c@FreeBSD.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2017 23:53:11 -0000 It looks like netmap is there; however, is there a way of figuring out if netmap is being used? root@router1:~ # dmesg | grep cxl cxl0: on t5nex0 cxl0: Ethernet address: 00:07:43:2c:ac:50 cxl0: 16 txq, 8 rxq (NIC) vcxl0: on cxl0 vcxl0: netmap queues/slots: TX 2/1023, RX 2/1024 vcxl0: 1 txq, 1 rxq (NIC); 2 txq, 2 rxq (netmap) cxl1: on t5nex0 cxl1: 16 txq, 8 rxq (NIC) vcxl1: on cxl1 vcxl1: netmap queues/slots: TX 2/1023, RX 2/1024 vcxl1: 1 txq, 1 rxq (NIC); 2 txq, 2 rxq (netmap) cxl0: link state changed to UP vcxl0: link state changed to UP cxl1: link state changed to UP vcxl1: link state changed to UP And yes, we are using UDP 64 bytes tests. On 3/24/17 7:39 PM, Navdeep Parhar wrote: > On 03/24/2017 16:07, Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER > SCIENCE CORP] wrote: >> At the time of implementing the vcxl* interfaces we get very bad >> results. > > You're probably not using netmap with the vcxl interfaces, and the > number of "normal" tx and rx queues is just 2 for these interfaces. > > Even if you _are_ using netmap, the hw.cxgbe.nnmtxq10g/rxq10g tunables > don't work anymore. Use these to control the number of queues for > netmap: > hw.cxgbe.nnmtxq_vi > hw.cxgbe.nnmrxq_vi > > You should see a line like this in dmesg for all cxl/vcxl interfaces > and that tells you exactly how many queues the driver configured: > cxl0: 4 txq, 4 rxq (NIC); 4 txq, 2 rxq (TOE) > >> >> packets errs idrops bytes packets errs bytes colls drops >> 629k 4.5k 0 66M 629k 0 66M >> 0 0 >> 701k 5.0k 0 74M 701k 0 74M >> 0 0 >> 668k 4.8k 0 70M 668k 0 70M >> 0 0 >> 667k 4.8k 0 70M 667k 0 70M >> 0 0 >> 645k 4.5k 0 68M 645k 0 68M >> 0 0 >> 686k 4.9k 0 72M 686k 0 72M >> 0 0 >> >> And by using just the cxl* interfaces we were getting about >> >> input (Total) output >> packets errs idrops bytes packets errs bytes colls >> drops >> 2.8M 0 1.2M 294M 1.6M 0 171M >> 0 0 >> 2.8M 0 1.2M 294M 1.6M 0 171M >> 0 0 >> 2.8M 0 1.2M 294M 1.6M 0 171M >> 0 0 >> 2.8M 0 1.2M 295M 1.6M 0 172M >> 0 0 >> 2.8M 0 1.2M 295M 1.6M 0 171M >> 0 0 >> >> These are our configurations for now. Any advice or suggestion will be >> appreciated. > > What I don't understand is that you have PAUSE disabled and congestion > drops enabled but still the number of packets coming in (whether they > are dropped eventually or not is irrelevant here) is very low in your > experiments. It's almost as if the senders are backing off in the > face of packet loss. Are you using TCP or UDP? Always use UDP for > pps testing -- the senders need to be relentless. > > Regards, > Navdeep > >> >> /etc/rc.conf configurations >> >> ifconfig_cxl0="up" >> ifconfig_cxl1="up" >> ifconfig_vcxl0="inet 172.16.2.1/24 -tso -lro mtu 9000" >> ifconfig_vcxl1="inet 172.16.1.1/24 -tso -lro mtu 9000" >> gateway_enable="YES" >> >> /boot/loader.conf configurations >> >> # Chelsio Modules >> t4fw_cfg_load="YES" >> t5fw_cfg_load="YES" >> if_cxgbe_load="YES" >> >> # rx and tx size >> dev.cxl.0.qsize_txq=8192 >> dev.cxl.0.qsize_rxq=8192 >> dev.cxl.1.qsize_txq=8192 >> dev.cxl.1.qsize_rxq=8192 >> >> # drop toecaps to increase queues >> dev.t5nex.0.toecaps=0 >> dev.t5nex.0.rdmacaps=0 >> dev.t5nex.0.iscsicaps=0 >> dev.t5nex.0.fcoecaps=0 >> >> # Controls the hardware response to congestion. -1 disables >> # congestion feedback and is not recommended. 0 instructs the >> # hardware to backpressure its pipeline on congestion. This >> # usually results in the port emitting PAUSE frames. 1 instructs >> # the hardware to drop frames destined for congested queues. From cxgbe >> dev.t5nex.0.cong_drop=1 >> >> # Saw these recomendations in Vicenzo email thread >> hw.cxgbe.num_vis=2 >> hw.cxgbe.fl_pktshift=0 >> hw.cxgbe.toecaps_allowed=0 >> hw.cxgbe.nnmtxq10g=8 >> hw.cxgbe.nnmrxq10g=8 >> >> /etc/sysctl.conf configurations >> >> # Turning off pauses >> dev.cxl.0.pause_settings=0 >> dev.cxl.1.pause_settings=0 >> # John Jasen suggestion - March 24, 2017 >> net.isr.bindthreads=0 >> net.isr.maxthreads=24 >> >> >> On 3/18/17 1:28 AM, Navdeep Parhar wrote: >>> On Fri, Mar 17, 2017 at 11:43:32PM -0400, John Jasen wrote: >>>> On 03/17/2017 03:32 PM, Navdeep Parhar wrote: >>>> >>>>> On Fri, Mar 17, 2017 at 12:21 PM, John Jasen >>>>> wrote: >>>>>> Yes. >>>>>> We were hopeful, initially, to be able to achieve higher packet >>>>>> forwarding rates through either netmap-fwd or due to enhancements >>>>>> based >>>>>> off https://wiki.freebsd.org/ProjectsRoutingProposal >>>>> Have you tried netmap-fwd? I'd be interested in how that did in >>>>> your tests. >>>> We have. On this particular box, (11-STABLE, netmap-fwd fresh from >>>> git) >>>> it took about 1.7m pps in, dropped 500k, and passed about 800k. >>>> >>>> I'm lead to believe that vcxl interfaces may yield better results? >>> Yes, those are the ones with native netmap support. Any netmap based >>> application should use the vcxl interfaces. If you used them on the >>> main cxl interfaces you were running netmap in emulated mode. >>> >>> Regards, >>> Navdeep >> >