From owner-freebsd-net@freebsd.org Sat Jan 23 18:55:15 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A935DA8FFC8 for ; Sat, 23 Jan 2016 18:55:15 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FD7B1B48 for ; Sat, 23 Jan 2016 18:55:15 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lb0-x229.google.com with SMTP id x4so56992059lbm.0 for ; Sat, 23 Jan 2016 10:55:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=yJZxU9s2GkCiP8wP5veDNoNZRJrUpRhJImD8FFvtC28=; b=bim0ZnY8AXyTqU9bIt93wGd8YSHpFgn89ZVYaIjdwiu937KyDlDavACJflH2hwh5Px tt52oVPbDqCK34kZ/ZP9q76K3g8Xx9/1YaR4vJ7k2/muoMLYaAyufy0nKerXu1y3Xc/U MxmYSbNuQQXngaCBpaRpHL9VqNw8tNohsob+1NkQWo9utC7dxlQ6lPO1Ae/+GNpKsm6x EUKjVwyChndioZBwBKbYAqB5t/BPjkpZ8hGQRenlJJcB6DarXG4kLW+j34w4fnAn2f50 Dc5F71PiPoU5A5RyrF0VVF48FhMyafNWdSQT0u5FKIoqhggSUrT+IKX+gQ/L9iFGmF47 cgdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=yJZxU9s2GkCiP8wP5veDNoNZRJrUpRhJImD8FFvtC28=; b=COPNEoKWoU/r1EFeHrV+tCiwLPUf4o7lNe3F8Dk+mpj3Xx4PHpHrDYl/tX73kHKoC3 EDmKUxoiNlwEVI7H7fkpiM/D9d5hxt0Xj4+qFbyQosaxfHC2EsapB7cDaLMFq9BFfwl6 djbbpiez+IfB7/AYmFMKSgNAa5a/nnM3rohLSKeu1f1nO5uICaCFOyaGYM7jVp7mn9+e Vo5EZ2TKAju4AZk5HAZSG7KxuupwP17tHtHeK1gQSiUyNEzkew4QNwLkpLgiCb66SpA8 u2BZJhQORy1PSHAUmJni2FRcPfC/y6HQO0wU3J7CW8IRyX3qbCfstx4j9F+zpqrd9zHJ Cxyg== X-Gm-Message-State: AG10YOTyBSwF17BLMsXvwiGRllc2a71I5b7V3/wyJwHs+n1XjjSo4oW4/2KhyCTbxdefovqfHwJaDRdoPxs0ZQ== MIME-Version: 1.0 X-Received: by 10.112.170.101 with SMTP id al5mr3451625lbc.92.1453575312893; Sat, 23 Jan 2016 10:55:12 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 10:55:12 -0800 (PST) In-Reply-To: <20160123183447.4B150A0126@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <20160123183447.4B150A0126@smtp.hushmail.com> Date: Sat, 23 Jan 2016 10:55:12 -0800 X-Google-Sender-Auth: 65a90C_j9f0Ejqc3ZHa9_d9X_Gk Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Marcus Cenzatti Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 18:55:15 -0000 On Sat, Jan 23, 2016 at 10:34 AM, Marcus Cenzatti wrote: >>> woops, my bad, yes probably we had some drop, with -S and -D now >>I get 1.2Mpps. >>> >>> curiously, I have always used -s/-d with IP addresses on ix-ix >>testing this is why I never noticed the case, since ix always >>received 14Mpps, but you probably explained it since ix has one >>single deviceport per wire, hence the different behavior >>> >>> performance stills very low when compared to TX and to what is >>expected >> >>ok so next we can try and see what else is going on. >>please check the following: >>a) are you connected through a switch ? if so, try to send >> out some packets through the ncxl0 port (using pkt-gen >> and its native MAC address) so the switch can learn the >> address and does not need to replicate traffic on all >> ports (which generally is done at a limited rate). >>b) see if using different packet sizes (say 256, 512, 1024, 1500 >> passed as the -l option to pkt-gen) affects the rx rate. >> If the rate does not change (except for 1500 bytes) >> it may be a problem with interrupt moderation >> >>c) use progressively increasing packet rates on the sender, >> using -R xxxx (start at 500000 packets per second, >> and then go up until the receiver cannot sustain the >> tx rate. >> >>d) use a smaller batch size on the receiver (-b XXX, use >> values such as 2, 4, 8, 16...) and see if things improve. >> Smaller batch sizes make pkt-gen check the NIC more often >> thus overcoming possible problems with interrupt moderation. >> >>Let us know the outcome. Depending on what you see there >>are several possible explanations. >> > > Ok, revisiting the summary > - TX host = Intel ix (host 1) > - RX host = Chelsio T520 (host 2) > - Simple topology host1==host2 directly connected intel port 0 (ix0) w/ chelsio port 0 (ncxl0). > > Tests results: > > => Batch 1 packet len > > TX at 256 bytes = 4.46Mpps/TX and 889Kpps/RX > TX at 256 bytes = 2.33Mpps/TX and 888Kpps/RX, 9.3Gbps on TX side according to pkt-gen > TX at 1024 bytes = 1.19Mpps/TX and 889Kpps/RX, 9.3Bps on TX > TX at 1500 bytes = 816Kpps/TX and 816Kpps/RX, 9.8Gbps on TX > > => Batch 2 rates > > -R 500000 / TX Speed: 499.99 Kpps Bandwidth: 240.00 Mbps (raw 336.00 Mbps) / RX 499Kpps > -R 700000 / TX Speed: 699.96 Kpps Bandwidth: 335.98 Mbps (raw 470.38 Mbps) / RX 699Kpps > -R 900000 / TX Speed: 899.98 Kpps Bandwidth: 431.99 Mbps (raw 604.78 Mbps) / RX 888Kpps > > reached the same limits on batch #1. > > => Batch 3 RX batch sizes, default pkt-gen packet len and fixed 900000 rate > > -r 2 / TX 899.98Kpps / RX 672Kpps > -r 4 / TX 899.98Kpps / RX 713Kpps > -r 8 / TX 899.98Kpps / RX 889Kpps > -r 16 / TX 899.98Kpps / RX 889Kpps > > Results make sense for rates bellow the max, but did not improve... only degraded. I think you are still sending to the broadcast addresss (otherwise i'd expect at least the 1.2 Mpps you were seeing before), and if this is the case the tests are not significant because you are hitting the in-nic duplication. Can you please re-run them sending to the MAC address of ncxl ? cheers luigi