Date: Wed, 4 Jan 2017 16:01:44 +0100 From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= <olivier@freebsd.org> To: Jordan Caraballo <jordancaraballo87@gmail.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Disappointing packets-per-second performance results on a Dell, PE R530 Message-ID: <CA%2Bq%2BTcoxK5YBW5WkB8T68uxf2x73kXxPe18hwUU5Y%2BgommuGcw@mail.gmail.com> In-Reply-To: <8f637e2e-cd59-dc65-8476-30989bea516b@gmail.com> References: <8f637e2e-cd59-dc65-8476-30989bea516b@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jan 3, 2017 at 5:35 PM, Jordan Caraballo < jordancaraballo87@gmail.com> wrote: > We recently tested a Dell R530 with a Chelsio T580 card, under FreeBSD > 10.3, 11.0, -STABLE and -CURRENT, and Centos 7. > > Based on our research, including netmap-fwd and with the routing > improvements project (https://wiki.freebsd.org/ProjectsRoutingProposal), > we hoped for packets-per-second (pps) in the 5+ million range, or even > higher. > > Based on prior testing (http://marc.info/?t=3D140604252400002&r=3D1&w=3D2= ), we > expected 3-4 Million to be easily obtainable. > > Unfortunately, our current results top out at no more than 1.5 M (64 byte= s > length packets) with FreeBSD, and > surprisingly around 3.2 M (128 bytes length packets) with Centos 7, and w= e > are at a loss as to why. > > > =E2=80=8B How do you generate your IP traffic ? You need to take care to generate multiple flows for correctly using all queues of your NIC. If you've got 18 cores per each CPU, cxbge drivers will create 16 NIC queues: Check they are all bound to cores belonging to the the same package= . How did you bind your NIC queues ? Did you disable Ethernet Flow-control and Ethernet entropy harvest ? On my HP ProLiant DL360p Gen8 with 8 cores (Intel Xeon E5-2650 @ 2.60GHz) and Chelsio NIC, I obtain about 5Mpps on a recent head: https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650-8Cores-Chel= sio_T540-CR/fastforwarding-pf-ipfw/results/fbsd12-head.r299288.D8526/README= .md =E2=80=8B =E2=80=8BMy lab setup and tools used is described here: https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp= _proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr For generating my traffic, I'm using a patched version of netmap pkt-gen, patch here: https://github.com/ocochard/BSDRP/blob/master/BSDRPcur/patches/freebsd.pkt-= gen.ae-ipv6.patch Regards, Olivier
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bq%2BTcoxK5YBW5WkB8T68uxf2x73kXxPe18hwUU5Y%2BgommuGcw>