From owner-freebsd-net@freebsd.org Sat Jan 23 18:00:50 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93F79A8F0F6 for ; Sat, 23 Jan 2016 18:00:50 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x22a.google.com (mail-lf0-x22a.google.com [IPv6:2a00:1450:4010:c07::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09ECC1D98 for ; Sat, 23 Jan 2016 18:00:50 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lf0-x22a.google.com with SMTP id c192so64013444lfe.2 for ; Sat, 23 Jan 2016 10:00:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=8wI8braWB3xNC9XcjSNGFL+db4thxYNlR5fZRJGxvEc=; b=chlwGFn2bAtbM9FbGJDx6epGxlQG9W6hptKLtWLb/btfRu+arKKR72Zhm4iSydOyj9 vbH8T+hLYqTBLwYZNeiF+Ujh5wYgmSRVzj4D2rK0nyrmJdtZabs5q1cpmHtdaYiSfJiw H3QpAu18rn1fZrwjo2/3QGZafqVndPmpbntlNcCTdRvEneXQpUOq4Ux2+cGbC4QpENIY UX554unETODrievLS6pXZ83/vniHj1vkZh0tSx8fNdLzX4ekq2KZ40TOHbtaKVEUcPxD A0ts8pS7EiZYFbSbDLYkdd8gj//yNqVj4jUkKKupHiqCUBFO2uNXHpQ5YAfqGRt6gPIk cW2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=8wI8braWB3xNC9XcjSNGFL+db4thxYNlR5fZRJGxvEc=; b=QJTxWG9qV763DWRbjO+w7Ko9FopA0ZlsUuO9wUbZdA0T53WEPCtn+FrlC2ufcoadbJ eGf65+wzZEBgGcZidEfTftbU5BZe09AHHBAYReE6mjbvbGn8zbyzy2sG11vuIMs7eZ+b 8PBI8SquBPdu6bYuBvb4lJhLqEXl3iM5VgDpgt4L1cMcKp9vuek1WIlrD3rFUMcQKNSy WU2WKi+WU6LtbT/wbrohXSCgZdO8RageIPElQZQJaj80d8QNk+1OYSL/v/gXX8SN+mqQ 1uOYczlU8N1mCFnmln3fy6dT7YQ0A0z5iOVeVsjUfKTpxo5I+IiKNcJYhE8h7EZYsE4d 4nnw== X-Gm-Message-State: AG10YOTA5mwaHdb93aQ8gA1x6zgapzrJmj8d8E3i5wRx4aNkXjWl0apMyOvapwG+FT6KSGIQJMAN62R8/hLgGg== MIME-Version: 1.0 X-Received: by 10.25.158.136 with SMTP id h130mr3382298lfe.136.1453572048078; Sat, 23 Jan 2016 10:00:48 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 10:00:48 -0800 (PST) In-Reply-To: <20160123174840.32B1DA0121@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> Date: Sat, 23 Jan 2016 10:00:48 -0800 X-Google-Sender-Auth: jQ5WIH6dSN45Lgg4HXYp0yDk2YY Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Marcus Cenzatti Cc: "freebsd-net@freebsd.org" , Navdeep Parhar Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 18:00:50 -0000 On Sat, Jan 23, 2016 at 9:48 AM, Marcus Cenzatti wrote: > > > On 1/23/2016 at 3:35 PM, "Luigi Rizzo" wrote: >> >>On Sat, Jan 23, 2016 at 9:12 AM, Marcus Cenzatti >> wrote: >>> >>> >>> On 1/23/2016 at 1:40 PM, "Navdeep Parhar" >>wrote: >>>> >>>>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: >>>>> hello, >>>>> >>>>> I am testing a chelsio t520-so-cr connected to a Intel card >>with >>>>ix(4) >>>>> driver, I can get the ncxl0 interface to transmit at 14Mpps to >>>>another >>>>> chelsio or to a Intel card. However I can only get 800Kpps- >>1Mpps >>>>for >>>>> RX tests from both chelsio or Intel. >>>>> >>>>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >>>>> >>>>> I tested it untuned first and later I have applied tuning >>>>> recommendations I found on BSDRP[1] website. Results still >>>>ranging >>>>> from 800Kpps to 1Mpps for RX. >>>>> >>>>> Tests are done w/ with pkt-gen in netmap mode on ncxl interface >>>>with >>>>> both IP address and MAC address source/dest. >>>> >>>>The ncxl interfaces have their own MAC addresses. Make sure the >>>>sender >>>>uses the MAC of the receiver's ncxl interface as the destination >>>>MAC. >>>>(netmap's pkt-gen -f tx transmits L2 broadcasts by default). >>>> >>>>Check for PAUSE frames coming out of the receiver (sysctl dev.cxl >>>>| grep >>>>tx_pause). If it's receiving frames on netmap interface the >>>>tx_pause >>>>counter should not move. >>>> >>>>Regards, >>>>Navdeep >>>> >>> >>> hello, >>> >>> yes, MAC addresses are correct, I did the tests again and >>tx_pause won't move, here is the full transcript for the tests: >>> >>> ===> BOX #1 CHELSIO >>> >>> chelsio# ifconfig -v ncxl0 >>> ncxl0: flags=8843 metric >>0 mtu 1500 >>> ether 00:07:43:33:8d:c1 >>> inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 >>> nd6 options=29 >>> media: Ethernet 10Gbase-SR >>> status: active >>> >>> chelsio# ifconfig -v cxl0 >>> cxl0: flags=8843 metric >>0 mtu 1500 >>> >>options=ec00bb>_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> >>> ether 00:07:43:33:8d:c0 >>> nd6 options=29 >>> media: Ethernet 10Gbase-SR >>> status: active >>> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) >>> vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K >>DATE: 2011-06-28 >>> module temperature: 42.79 C Voltage: 3.23 Volts >>> RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) >>> >>> chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s >>00:07:e9:44:d2:ba >>> 311.132189 main [1715] interface is ncxl0 >>> 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to >>0.0.0.0:90 >>> 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 >> >> >>wait, the lower case -s and -d are for IP addresses, >>you need to use -S and -D for the MAC addresses. >>This way you are sending broadcasts, which likely >>means that the chelsio is replicating the packets >>to both the netmap and the regular port and the >>latter (which perhaps comes first) is likely >>dropping packets. >> >>cheers >>luigi > > woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps. > > curiously, I have always used -s/-d with IP addresses on ix-ix testing this is why I never noticed the case, since ix always received 14Mpps, but you probably explained it since ix has one single deviceport per wire, hence the different behavior > > performance stills very low when compared to TX and to what is expected ok so next we can try and see what else is going on. please check the following: a) are you connected through a switch ? if so, try to send out some packets through the ncxl0 port (using pkt-gen and its native MAC address) so the switch can learn the address and does not need to replicate traffic on all ports (which generally is done at a limited rate). b) see if using different packet sizes (say 256, 512, 1024, 1500 passed as the -l option to pkt-gen) affects the rx rate. If the rate does not change (except for 1500 bytes) it may be a problem with interrupt moderation c) use progressively increasing packet rates on the sender, using -R xxxx (start at 500000 packets per second, and then go up until the receiver cannot sustain the tx rate. d) use a smaller batch size on the receiver (-b XXX, use values such as 2, 4, 8, 16...) and see if things improve. Smaller batch sizes make pkt-gen check the NIC more often thus overcoming possible problems with interrupt moderation. Let us know the outcome. Depending on what you see there are several possible explanations. cheers luigi