From owner-freebsd-net@freebsd.org Fri Mar 24 23:07:56 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EDBAD1BD2A for ; Fri, 24 Mar 2017 23:07:56 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: from mail-yw0-x242.google.com (mail-yw0-x242.google.com [IPv6:2607:f8b0:4002:c05::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5A4FA1594 for ; Fri, 24 Mar 2017 23:07:56 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: by mail-yw0-x242.google.com with SMTP id h11so451832ywa.2 for ; Fri, 24 Mar 2017 16:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=PxeA5+pvsJVUmvT3Lwgf/4pu4a4kx3Be9f7JN+Gwzj0=; b=bsGJTuqb4U9Iu+4epN1OaiSb5iVXMFOBpldo3EVpPPZMVUAGGa90xA83POqy0IxRu1 uNILXVyoBt4Qh/iSbzQaQELuwDEId82ZNLa1NR6lAWd3igMq483as/1jZFAanBTh89Jj TPeG93fXVIuyfZrWpiwF71siNf2O5HBJw86NY96qV15gV8izTrLHJ1NkBKlrNM7c+jHW hPUqLSF7wW8vBGzWGB9GKPVaUUsnzmx5woK+J9vZKa6UYGkvyF7tQjC0mLzdrs9SP+hY 9YqZ1DaTLiZMUTPL1IlFgtftG82QpXA6/hizZeI9ZWFumAmNF35AZRx0fdub8eZWiQW1 bWNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=PxeA5+pvsJVUmvT3Lwgf/4pu4a4kx3Be9f7JN+Gwzj0=; b=GwegkU2QTEJFLhdtYhrbsly5ke/3p7J172R9RfhMTkk6foq+ESheZmx2UOQuGWFdWx 1sQNSw4FUkNR5zmYSGpE/feAYIMPCAwPN/9eN7DUbhIgVq/7uJ9akaJN59yZ48HhoSQy 5FsBpJ3L8n5HWdNMGjV1JvqpscKjaIzN5muWYHEJ2e2FIZzYjFjvU6bR65ANTrOi0K7a VcQY/Bwag9gQEf0N/7XXQ5wbHcIb/8yr/UsWy+hglGd7dJk6U4MOQZ9OI8PKt5TZjsnq tD+mdnLxXdDz7bLgtiLjbanTVfsBu1b0wZEMtxWGrYGNjZrvYj4lYelyW/kyXbGI5kl3 hrUA== X-Gm-Message-State: AFeK/H3HNUuojb5bYqO9PFieVmmlx78Ce2EcJanF+l2HX5yldjExdGc8ffFbZxRlLTcmjA== X-Received: by 10.129.101.7 with SMTP id z7mr712591ywb.45.1490396875126; Fri, 24 Mar 2017 16:07:55 -0700 (PDT) Received: from [10.0.0.67] (adsl-64-237-185-120.prtc.net. [64.237.185.120]) by smtp.gmail.com with ESMTPSA id n83sm1429097ywb.40.2017.03.24.16.07.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Mar 2017 16:07:54 -0700 (PDT) Subject: Re: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 To: Navdeep Parhar References: <20170312231826.GV15630@zxy.spb.ru> <74654520-b8b6-6118-2e46-902a8ea107ac@gmail.com> <173fffac-7ae2-786a-66c0-e9cd7ab78f44@gmail.com> <20170317100814.GN70430@zxy.spb.ru> <9924b2d5-4a72-579c-96c6-4dbdacc07c95@gmail.com> <9694e9f2-daec-924d-e9f6-7b22a634acb5@gmail.com> <20170318052837.GA21730@ox> Cc: slw@zxy.spb.ru, freebsd-net@freebsd.org, John Jasen From: "Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER SCIENCE CORP]" Message-ID: Date: Fri, 24 Mar 2017 19:07:51 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170318052837.GA21730@ox> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2017 23:07:56 -0000 At the time of implementing the vcxl* interfaces we get very bad results. packets errs idrops bytes packets errs bytes colls drops 629k 4.5k 0 66M 629k 0 66M 0 0 701k 5.0k 0 74M 701k 0 74M 0 0 668k 4.8k 0 70M 668k 0 70M 0 0 667k 4.8k 0 70M 667k 0 70M 0 0 645k 4.5k 0 68M 645k 0 68M 0 0 686k 4.9k 0 72M 686k 0 72M 0 0 And by using just the cxl* interfaces we were getting about input (Total) output packets errs idrops bytes packets errs bytes colls drops 2.8M 0 1.2M 294M 1.6M 0 171M 0 0 2.8M 0 1.2M 294M 1.6M 0 171M 0 0 2.8M 0 1.2M 294M 1.6M 0 171M 0 0 2.8M 0 1.2M 295M 1.6M 0 172M 0 0 2.8M 0 1.2M 295M 1.6M 0 171M 0 0 These are our configurations for now. Any advice or suggestion will be appreciated. /etc/rc.conf configurations ifconfig_cxl0="up" ifconfig_cxl1="up" ifconfig_vcxl0="inet 172.16.2.1/24 -tso -lro mtu 9000" ifconfig_vcxl1="inet 172.16.1.1/24 -tso -lro mtu 9000" gateway_enable="YES" /boot/loader.conf configurations # Chelsio Modules t4fw_cfg_load="YES" t5fw_cfg_load="YES" if_cxgbe_load="YES" # rx and tx size dev.cxl.0.qsize_txq=8192 dev.cxl.0.qsize_rxq=8192 dev.cxl.1.qsize_txq=8192 dev.cxl.1.qsize_rxq=8192 # drop toecaps to increase queues dev.t5nex.0.toecaps=0 dev.t5nex.0.rdmacaps=0 dev.t5nex.0.iscsicaps=0 dev.t5nex.0.fcoecaps=0 # Controls the hardware response to congestion. -1 disables # congestion feedback and is not recommended. 0 instructs the # hardware to backpressure its pipeline on congestion. This # usually results in the port emitting PAUSE frames. 1 instructs # the hardware to drop frames destined for congested queues. From cxgbe dev.t5nex.0.cong_drop=1 # Saw these recomendations in Vicenzo email thread hw.cxgbe.num_vis=2 hw.cxgbe.fl_pktshift=0 hw.cxgbe.toecaps_allowed=0 hw.cxgbe.nnmtxq10g=8 hw.cxgbe.nnmrxq10g=8 /etc/sysctl.conf configurations # Turning off pauses dev.cxl.0.pause_settings=0 dev.cxl.1.pause_settings=0 # John Jasen suggestion - March 24, 2017 net.isr.bindthreads=0 net.isr.maxthreads=24 On 3/18/17 1:28 AM, Navdeep Parhar wrote: > On Fri, Mar 17, 2017 at 11:43:32PM -0400, John Jasen wrote: >> On 03/17/2017 03:32 PM, Navdeep Parhar wrote: >> >>> On Fri, Mar 17, 2017 at 12:21 PM, John Jasen wrote: >>>> Yes. >>>> We were hopeful, initially, to be able to achieve higher packet >>>> forwarding rates through either netmap-fwd or due to enhancements based >>>> off https://wiki.freebsd.org/ProjectsRoutingProposal >>> Have you tried netmap-fwd? I'd be interested in how that did in your tests. >> We have. On this particular box, (11-STABLE, netmap-fwd fresh from git) >> it took about 1.7m pps in, dropped 500k, and passed about 800k. >> >> I'm lead to believe that vcxl interfaces may yield better results? > Yes, those are the ones with native netmap support. Any netmap based > application should use the vcxl interfaces. If you used them on the > main cxl interfaces you were running netmap in emulated mode. > > Regards, > Navdeep