From owner-freebsd-net@freebsd.org Wed Jan 4 15:02:21 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 943C4C9EDD4 for ; Wed, 4 Jan 2017 15:02:21 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: from smtp.smtpout.orange.fr (smtp03.smtpout.orange.fr [80.12.242.125]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "Bizanga Labs SMTP Client Certificate", Issuer "Bizanga Labs CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D210815F0 for ; Wed, 4 Jan 2017 15:02:19 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: from mail-qk0-f171.google.com ([209.85.220.171]) by mwinf5d05 with ME id UF251u00P3iUoqR03F25Qo; Wed, 04 Jan 2017 16:02:11 +0100 X-ME-Helo: mail-qk0-f171.google.com X-ME-Auth: Y29jaGFyZC1sYWJiZS5vbGl2aWVyQG9yYW5nZS5mcg== X-ME-Date: Wed, 04 Jan 2017 16:02:11 +0100 X-ME-IP: 209.85.220.171 Received: by mail-qk0-f171.google.com with SMTP id a20so13490949qkc.1 for ; Wed, 04 Jan 2017 07:02:05 -0800 (PST) X-Gm-Message-State: AIkVDXIUV3TBloawUMIDbMiDkdDrBXZG8AC9olvPzmdm3Jo71saBCkmUWmueLbFeXWAbZeSX7K0lNJ44Urk35w== X-Received: by 10.55.162.194 with SMTP id l185mr60852612qke.4.1483542124863; Wed, 04 Jan 2017 07:02:04 -0800 (PST) MIME-Version: 1.0 Received: by 10.237.34.248 with HTTP; Wed, 4 Jan 2017 07:01:44 -0800 (PST) In-Reply-To: <8f637e2e-cd59-dc65-8476-30989bea516b@gmail.com> References: <8f637e2e-cd59-dc65-8476-30989bea516b@gmail.com> From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Date: Wed, 4 Jan 2017 16:01:44 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Disappointing packets-per-second performance results on a Dell, PE R530 To: Jordan Caraballo Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jan 2017 15:02:21 -0000 On Tue, Jan 3, 2017 at 5:35 PM, Jordan Caraballo < jordancaraballo87@gmail.com> wrote: > We recently tested a Dell R530 with a Chelsio T580 card, under FreeBSD > 10.3, 11.0, -STABLE and -CURRENT, and Centos 7. > > Based on our research, including netmap-fwd and with the routing > improvements project (https://wiki.freebsd.org/ProjectsRoutingProposal), > we hoped for packets-per-second (pps) in the 5+ million range, or even > higher. > > Based on prior testing (http://marc.info/?t=3D140604252400002&r=3D1&w=3D2= ), we > expected 3-4 Million to be easily obtainable. > > Unfortunately, our current results top out at no more than 1.5 M (64 byte= s > length packets) with FreeBSD, and > surprisingly around 3.2 M (128 bytes length packets) with Centos 7, and w= e > are at a loss as to why. > > > =E2=80=8B How do you generate your IP traffic ? You need to take care to generate multiple flows for correctly using all queues of your NIC. If you've got 18 cores per each CPU, cxbge drivers will create 16 NIC queues: Check they are all bound to cores belonging to the the same package= . How did you bind your NIC queues ? Did you disable Ethernet Flow-control and Ethernet entropy harvest ? On my HP ProLiant DL360p Gen8 with 8 cores (Intel Xeon E5-2650 @ 2.60GHz) and Chelsio NIC, I obtain about 5Mpps on a recent head: https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650-8Cores-Chel= sio_T540-CR/fastforwarding-pf-ipfw/results/fbsd12-head.r299288.D8526/README= .md =E2=80=8B =E2=80=8BMy lab setup and tools used is described here: https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp= _proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr For generating my traffic, I'm using a patched version of netmap pkt-gen, patch here: https://github.com/ocochard/BSDRP/blob/master/BSDRPcur/patches/freebsd.pkt-= gen.ae-ipv6.patch Regards, Olivier