From owner-freebsd-net@freebsd.org Sat Jan 23 17:35:39 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 437A4A8E349 for ; Sat, 23 Jan 2016 17:35:39 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CAD4C1EDE for ; Sat, 23 Jan 2016 17:35:38 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lb0-x22a.google.com with SMTP id oh2so55910012lbb.3 for ; Sat, 23 Jan 2016 09:35:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=bkiZls4LqG8CK2b7nQoYaGkGoqFTLXFOusicrZEykoQ=; b=UBd8CddEVrPpc88ch45NZ9wZqyxIE/1T3zLVn1YzGJrdfVyvuU80n2KeQ05Ek4I5mM bw3U6h0Yj4rkaC9e1j+YaoJVckAPvypICttS8f5WsWAjl+SfSI0QFji/f9jW3UHCLenZ bM/3whudqEyxaNpQqX0DvoPw6hZuqEzX5yANoOa5es6kkR2HHKLA3tW8JMMKafQaSr2A khH8iN6jAv3L4E/BF6reM4sNmLR89vGN7nHAPHh+1B2zlBmee9hV2gSjc4Y18h7GftpP CLTNicmNnoY7vZ6i6DkPsTyJfC6mE0y4cEfzAAQFQPwLAT8bfRSxUyzRs2eb3gjG2kBc GeZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=bkiZls4LqG8CK2b7nQoYaGkGoqFTLXFOusicrZEykoQ=; b=hXS1ctTzA64SXL12hqY8UKtHWAXIFh5qH7tiV4B448zCX2LqPBSjXQaBiiZs6aL8ib 5xE6X3Vs8DVurZLvb4LTU+/rGKquIybksc3Ba1Ks8ZiOkTFhOIid/rxuxoI7m3qsfcAj 2iZx1vHKayIGysg+bEbytAeHyd+xFTd3Ex05sibSVjLyoBSzPXp2KMsp1+0jRmqQf8kH ZDs4YJcuHTMoGlxXe752nQXzO5jGvnIs2/JOaPC4jtg8BzGbs1oBBUCiMFWmgdDPxGRU q6iCVRdQ2fAZX+Jd4trC/gSMR34hAfQc/boV1YWjhFy/uUjh3e5f1YtrBg7V9dDKHhsp oCrA== X-Gm-Message-State: AG10YOQtfHzdPgAnXAM7owRoWU9boR9rlmoDVZCRMG/INVbgQmiTn3gW8i9H+XmPF1T2uPojupSxWsUP5NBIGw== MIME-Version: 1.0 X-Received: by 10.112.13.99 with SMTP id g3mr3284924lbc.86.1453570535803; Sat, 23 Jan 2016 09:35:35 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 09:35:35 -0800 (PST) In-Reply-To: <20160123171300.0F448A0121@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> Date: Sat, 23 Jan 2016 09:35:35 -0800 X-Google-Sender-Auth: 8YPAKAHihL7_9uVxYBQ7VbG9JJo Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Marcus Cenzatti Cc: Navdeep Parhar , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 17:35:39 -0000 On Sat, Jan 23, 2016 at 9:12 AM, Marcus Cenzatti wrote: > > > On 1/23/2016 at 1:40 PM, "Navdeep Parhar" wrote: >> >>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: >>> hello, >>> >>> I am testing a chelsio t520-so-cr connected to a Intel card with >>ix(4) >>> driver, I can get the ncxl0 interface to transmit at 14Mpps to >>another >>> chelsio or to a Intel card. However I can only get 800Kpps-1Mpps >>for >>> RX tests from both chelsio or Intel. >>> >>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >>> >>> I tested it untuned first and later I have applied tuning >>> recommendations I found on BSDRP[1] website. Results still >>ranging >>> from 800Kpps to 1Mpps for RX. >>> >>> Tests are done w/ with pkt-gen in netmap mode on ncxl interface >>with >>> both IP address and MAC address source/dest. >> >>The ncxl interfaces have their own MAC addresses. Make sure the >>sender >>uses the MAC of the receiver's ncxl interface as the destination >>MAC. >>(netmap's pkt-gen -f tx transmits L2 broadcasts by default). >> >>Check for PAUSE frames coming out of the receiver (sysctl dev.cxl >>| grep >>tx_pause). If it's receiving frames on netmap interface the >>tx_pause >>counter should not move. >> >>Regards, >>Navdeep >> > > hello, > > yes, MAC addresses are correct, I did the tests again and tx_pause won't move, here is the full transcript for the tests: > > ===> BOX #1 CHELSIO > > chelsio# ifconfig -v ncxl0 > ncxl0: flags=8843 metric 0 mtu 1500 > ether 00:07:43:33:8d:c1 > inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 > nd6 options=29 > media: Ethernet 10Gbase-SR > status: active > > chelsio# ifconfig -v cxl0 > cxl0: flags=8843 metric 0 mtu 1500 > options=ec00bb > ether 00:07:43:33:8d:c0 > nd6 options=29 > media: Ethernet 10Gbase-SR > status: active > plugged: SFP/SFP+/SFP28 10G Base-SR (LC) > vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K DATE: 2011-06-28 > module temperature: 42.79 C Voltage: 3.23 Volts > RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) > > chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba > 311.132189 main [1715] interface is ncxl0 > 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90 > 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 wait, the lower case -s and -d are for IP addresses, you need to use -S and -D for the MAC addresses. This way you are sending broadcasts, which likely means that the chelsio is replicating the packets to both the netmap and the regular port and the latter (which perhaps comes first) is likely dropping packets. cheers luigi