Date: Mon, 17 May 2021 09:53:25 +0000 From: Francois ten Krooden <ftk@Nanoteq.com> To: Vincenzo Maffione <vmaffione@freebsd.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Jacques Fourie <jacques.fourie@gmail.com> Subject: RE: Vector Packet Processing (VPP) portability on FreeBSD Message-ID: <AB9BB4D903F59549B2E27CC033B964D6C4F8D74A@NTQ-EXC.nanoteq.co.za> In-Reply-To: <CA%2B_eA9iG=4nemZxM_yETxGTMMC-oXPtMZmWc9DCp%2BqJaCQt4=g@mail.gmail.com> References: <AB9BB4D903F59549B2E27CC033B964D6C4F8BECE@NTQ-EXC.nanoteq.co.za> <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> <CA%2BhQ2%2BjQ2fh4TXz02mTxAHJkHBWzfNhd=yRqPG45E7Z4umAsKA@mail.gmail.com> <e778ca61766741b0950585f6b26d8fff@EX16-05.ad.unipi.it> <CA%2BhQ2%2BhzjT5%2BRXmUUV4PpkXkvgQEJb8JrLPY7LqteV9ixeM7Ew@mail.gmail.com> <AB9BB4D903F59549B2E27CC033B964D6C4F8D386@NTQ-EXC.nanoteq.co.za> <CALX0vxA3_eDRJmEGBak=e99nOrBkFYEmdnBHEY9JLTmT7tQ2vQ@mail.gmail.com> <AB9BB4D903F59549B2E27CC033B964D6C4F8D3BB@NTQ-EXC.nanoteq.co.za> <CA%2B_eA9iG=4nemZxM_yETxGTMMC-oXPtMZmWc9DCp%2BqJaCQt4=g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2021/05/16 09:22, Vincenzo Maffione wrote: > > Hi, > Yes, you are not using emulated netmap mode. > > In the test setup depicted here > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap- > interfaces#test-setup > I think you should really try to replace VPP with the netmap "bridge" > application (tools/tools/netmap/bridge.c), and see what numbers you get. > > You would run the application this way > # bridge -i ix0 -i ix1 > and this will forward any traffic between ix0 and ix1 (in both directions= ). > > These numbers would give you a better idea of where to look next (e.g. VP= P > code improvements or system tuning such as NIC interrupts, CPU binding, > etc.). Thank you for the suggestion. I did run a test with the bridge this morning, and updated the results as w= ell. +-------------+------------------+ | Packet Size | Throughput (pps) | +-------------+------------------+ | 64 bytes | 7.197 Mpps | | 128 bytes | 7.638 Mpps | | 512 bytes | 2.358 Mpps | | 1280 bytes | 964.915 kpps | | 1518 bytes | 815.239 kpps | +-------------+------------------+ Besides for the 64-byte and 128-byte packets the other sizes where matching= the maximum rates possible on 10Gbps. This was when the bridge application was running on a single core, and the = cpu core was maxing out at a 100%. I think there might be a bit of system tuning needed, but I suspect most of= the improvement would be needed in VPP. Regards Francois > > Cheers, > Vincenzo > > Il giorno gio 13 mag 2021 alle ore 15:02 Francois ten Krooden < > ftk@nanoteq.com> ha scritto: > > > On Thursday, 13 May 2021 13:59 Jacques Fourie > > > > > > > > On Thu, May 13, 2021 at 7:27 AM Francois ten Krooden > > > <ftk@nanoteq.com> > > > wrote: > > > > > > On Thursday, 13 May 2021 13:05 Luigi Rizzo wrote: > > > > > > > > On Thu, May 13, 2021 at 10:42 AM Francois ten Krooden > > > > <ftk@nanoteq.com> wrote: > > > > > > > > > > Hi > > > > > > > > > > Just for info I ran a test using TREX > > > > > (https://trex-tgn.cisco.com/) Where I just sent traffic in one > > > > > direction through the box > > > running FreeBSD > > > > with VPP using the netmap interfaces. > > > > > These were the results we found before significant packet loss > > started > > > > occuring. > > > > > +-------------+------------------+ > > > > > | Packet Size | Throughput (pps) | > > > > > +-------------+------------------+ > > > > > | 64 bytes | 1.008 Mpps | > > > > > | 128 bytes | 920.311 kpps | > > > > > | 256 bytes | 797.789 kpps | > > > > > | 512 bytes | 706.338 kpps | > > > > > | 1024 bytes | 621.963 kpps | > > > > > | 1280 bytes | 569.140 kpps | > > > > > | 1440 bytes | 547.139 kpps | > > > > > | 1518 bytes | 524.864 kpps | > > > > > +-------------+------------------+ > > > > > > > > Those numbers are way too low for netmap. > > > > > > > > I believe you are either using the emulated mode, or issuing a > > > > system > > call > > > on > > > > every single packet. > > > > > > > > I am not up to date (Vincenzo may know better) but there used to > > > > be a > > > sysctl > > > > variable to control the operating mode: > > > > > > > > https://www.freebsd.org/cgi/man.cgi?query=3Dnetmap&sektion=3D4 > > > > > > > > SYSCTL VARIABLES AND MODULE PARAMETERS > > > > Some aspects of the operation of netmap and VALE are > > > > controlled through > > > > sysctl variables on FreeBSD (dev.netmap.*) and module > > > > parameters > > on > > > > Linux > > > > (/sys/module/netmap/parameters/*): > > > > > > > > dev.netmap.admode: 0 > > > > Controls the use of native or emulated adapter mode. > > > > > > > > 0 uses the best available option; > > > > > > > > 1 forces native mode and fails if not available; > > > > > > > > 2 forces emulated hence never fails. > > > > > > > > If it still exists, try set it to 1. If the program fails, then > > > > you > > should figure > > > out > > > > why native netmap support is not compiled in. > > > > > > Thank you. I did set this to 1 specifically now and it still works. > > > So > > then it > > > should be running in native mode. > > > > > > I will dig a bit into the function that processes the incoming packet= s. > > > The code I currently use was added to VPP in somewhere before 2016, > > > so it might be that there is a bug in that code. > > > > > > Will try and see if I can find anything interesting there. > > > > > > > > > > > cheers > > > > luigi > > > > > > > A couple of questions / suggestions: > > > > Thank you for the suggestions. > > > > > Will it be possible to test using the netmap bridge app or a vale > > > switch instead of vpp? > > I did perform a test using netmap-fwd ( > > https://github.com/Netgate/netmap-fwd) > > I did look at the code and it appears that the packets are processed > > as a batch in the application. But each packet is passed through the > > complete IP stack in the application, before the next one is processed. > > With this application it was possible to reach about 1.4Mpps for > > 64-byte packets, and 812 kpps for 1518 byte packets I haven't done any > > other tweaking on the FreeBSD box yet. It is running FreeBSD 13.0 > > > > > Did you verify that the TREX setup can perform at line rate when > > connected > > > back to back? > > We did tests with TREX back to back yesterday and we reached the > following. > > +-------------+------------------+ > > | Packet Size | Throughput (pps) | > > +-------------+------------------+ > > | 64 bytes | 14.570 Mpps | > > | 128 bytes | 8.466 kpps | > > | 256 bytes | 4.542 kpps | > > | 512 bytes | 2.354 kpps | > > | 1024 bytes | 1.200 kpps | > > | 1280 bytes | 965.042 kpps | > > | 1440 bytes | 857.795 kpps | > > | 1518 bytes | 814.690 kpps | > > +-------------+------------------+ > > > > > Which NICs are you using? > > We are using Intel X552 10 GbE SFP+ NIC's which is part of the Intel > > Xeon > > D-1537 SoC, on a SuperMicro X10SDV-8C-TLN4F+ Board. > > > > I will also put the results on the github repository > > https://github.com/ftk-ntq/vpp/wiki > > and will update as we get some more information > > > > Kind Regards > > Francois > > > > > > > > > > > Important Notice: > > > > > > This e-mail and its contents are subject to the Nanoteq (Pty) Ltd > > > e-mail > > legal > > > notice available at: > > > http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx > > > _______________________________________________ > > > freebsd-net@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " > > > > > > Important Notice: > > > > This e-mail and its contents are subject to the Nanoteq (Pty) Ltd > > e-mail legal notice available at: > > http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > Important Notice: This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail le= gal notice available at: http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AB9BB4D903F59549B2E27CC033B964D6C4F8D74A>