From owner-freebsd-net@freebsd.org Mon May 17 09:53:44 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7BD176428BB for ; Mon, 17 May 2021 09:53:44 +0000 (UTC) (envelope-from ftk@nanoteq.com) Received: from mailguard.liquidtelecom.co.za (delivery.mailguard.neotel.co.za [41.168.2.25]) by mx1.freebsd.org (Postfix) with ESMTP id 4FkDwl2L1Hz3kPQ; Mon, 17 May 2021 09:53:42 +0000 (UTC) (envelope-from ftk@nanoteq.com) Received: from SEC-NGP-AG02 ([192.168.202.32]) by mailguard.liquidtelecom.co.za with Microsoft SMTPSVC(7.0.6002.18264); Mon, 17 May 2021 11:53:06 +0200 Received: from sec-ngp-spt04.e-purifier.com ([192.168.201.1]) by SEC-NGP-AG02.neotel.e-purifier.co.za with Microsoft SMTPSVC(7.5.7601.17514); Mon, 17 May 2021 11:53:05 +0200 Received: from localhost (localhost [127.0.0.1]) by sec-ngp-spt04.e-purifier.com (Postfix) with ESMTP id 92B2B1012E0B; Mon, 17 May 2021 11:53:38 +0200 (SAST) X-Virus-Scanned: by SpamTitan at e-purifier.com Received: from sec-ngp-spt04.e-purifier.com (localhost [127.0.0.1]) by sec-ngp-spt04.e-purifier.com (Postfix) with ESMTP id E10651012E07; Mon, 17 May 2021 11:53:31 +0200 (SAST) Received: from NTQ-EXC.nanoteq.co.za (unknown [41.170.5.18]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sec-ngp-spt04.e-purifier.com (Postfix) with ESMTPS id CF0651012E0E; Mon, 17 May 2021 11:53:31 +0200 (SAST) Received: from NTQ-EXC.nanoteq.co.za ([fe80::a5b3:4700:5af3:78b2]) by NTQ-EXC.nanoteq.co.za ([fe80::a5b3:4700:5af3:78b2%12]) with mapi id 14.03.0513.000; Mon, 17 May 2021 11:53:29 +0200 From: Francois ten Krooden To: Vincenzo Maffione CC: "freebsd-net@freebsd.org" , Jacques Fourie Subject: RE: Vector Packet Processing (VPP) portability on FreeBSD Thread-Topic: Vector Packet Processing (VPP) portability on FreeBSD Thread-Index: AQHXRapQkx/sKwTM3EOdxg1D+Pz3RqrhREohgAAEi1D//+jugIAAL0oAgAQ6h4CAAdzSEA== Date: Mon, 17 May 2021 09:53:25 +0000 Message-ID: References: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> In-Reply-To: Accept-Language: en-US, en-ZA Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 17 May 2021 09:53:05.0116 (UTC) FILETIME=[6EC549C0:01D74B02] x-archived: yes x-dbused: RGF0YSBTb3VyY2U9MTkyLjE2OC4yMDEuMjc= X-Rspamd-Queue-Id: 4FkDwl2L1Hz3kPQ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of ftk@nanoteq.com has no SPF policy when checking 41.168.2.25) smtp.mailfrom=ftk@nanoteq.com X-Spamd-Result: default: False [-1.63 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[41.168.2.25:from]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[Nanoteq.com]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[41.168.2.25:from]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-0.53)[-0.533]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:36937, ipnet:41.168.0.0/17, country:ZA]; RCVD_COUNT_SEVEN(0.00)[7]; MAILMAN_DEST(0.00)[freebsd-net]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 May 2021 09:53:44 -0000 On 2021/05/16 09:22, Vincenzo Maffione wrote: > > Hi, > Yes, you are not using emulated netmap mode. > > In the test setup depicted here > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap- > interfaces#test-setup > I think you should really try to replace VPP with the netmap "bridge" > application (tools/tools/netmap/bridge.c), and see what numbers you get. > > You would run the application this way > # bridge -i ix0 -i ix1 > and this will forward any traffic between ix0 and ix1 (in both directions= ). > > These numbers would give you a better idea of where to look next (e.g. VP= P > code improvements or system tuning such as NIC interrupts, CPU binding, > etc.). Thank you for the suggestion. I did run a test with the bridge this morning, and updated the results as w= ell. +-------------+------------------+ | Packet Size | Throughput (pps) | +-------------+------------------+ | 64 bytes | 7.197 Mpps | | 128 bytes | 7.638 Mpps | | 512 bytes | 2.358 Mpps | | 1280 bytes | 964.915 kpps | | 1518 bytes | 815.239 kpps | +-------------+------------------+ Besides for the 64-byte and 128-byte packets the other sizes where matching= the maximum rates possible on 10Gbps. This was when the bridge application was running on a single core, and the = cpu core was maxing out at a 100%. I think there might be a bit of system tuning needed, but I suspect most of= the improvement would be needed in VPP. Regards Francois > > Cheers, > Vincenzo > > Il giorno gio 13 mag 2021 alle ore 15:02 Francois ten Krooden < > ftk@nanoteq.com> ha scritto: > > > On Thursday, 13 May 2021 13:59 Jacques Fourie > > > > > > > > On Thu, May 13, 2021 at 7:27 AM Francois ten Krooden > > > > > > wrote: > > > > > > On Thursday, 13 May 2021 13:05 Luigi Rizzo wrote: > > > > > > > > On Thu, May 13, 2021 at 10:42 AM Francois ten Krooden > > > > wrote: > > > > > > > > > > Hi > > > > > > > > > > Just for info I ran a test using TREX > > > > > (https://trex-tgn.cisco.com/) Where I just sent traffic in one > > > > > direction through the box > > > running FreeBSD > > > > with VPP using the netmap interfaces. > > > > > These were the results we found before significant packet loss > > started > > > > occuring. > > > > > +-------------+------------------+ > > > > > | Packet Size | Throughput (pps) | > > > > > +-------------+------------------+ > > > > > | 64 bytes | 1.008 Mpps | > > > > > | 128 bytes | 920.311 kpps | > > > > > | 256 bytes | 797.789 kpps | > > > > > | 512 bytes | 706.338 kpps | > > > > > | 1024 bytes | 621.963 kpps | > > > > > | 1280 bytes | 569.140 kpps | > > > > > | 1440 bytes | 547.139 kpps | > > > > > | 1518 bytes | 524.864 kpps | > > > > > +-------------+------------------+ > > > > > > > > Those numbers are way too low for netmap. > > > > > > > > I believe you are either using the emulated mode, or issuing a > > > > system > > call > > > on > > > > every single packet. > > > > > > > > I am not up to date (Vincenzo may know better) but there used to > > > > be a > > > sysctl > > > > variable to control the operating mode: > > > > > > > > https://www.freebsd.org/cgi/man.cgi?query=3Dnetmap&sektion=3D4 > > > > > > > > SYSCTL VARIABLES AND MODULE PARAMETERS > > > > Some aspects of the operation of netmap and VALE are > > > > controlled through > > > > sysctl variables on FreeBSD (dev.netmap.*) and module > > > > parameters > > on > > > > Linux > > > > (/sys/module/netmap/parameters/*): > > > > > > > > dev.netmap.admode: 0 > > > > Controls the use of native or emulated adapter mode. > > > > > > > > 0 uses the best available option; > > > > > > > > 1 forces native mode and fails if not available; > > > > > > > > 2 forces emulated hence never fails. > > > > > > > > If it still exists, try set it to 1. If the program fails, then > > > > you > > should figure > > > out > > > > why native netmap support is not compiled in. > > > > > > Thank you. I did set this to 1 specifically now and it still works. > > > So > > then it > > > should be running in native mode. > > > > > > I will dig a bit into the function that processes the incoming packet= s. > > > The code I currently use was added to VPP in somewhere before 2016, > > > so it might be that there is a bug in that code. > > > > > > Will try and see if I can find anything interesting there. > > > > > > > > > > > cheers > > > > luigi > > > > > > > A couple of questions / suggestions: > > > > Thank you for the suggestions. > > > > > Will it be possible to test using the netmap bridge app or a vale > > > switch instead of vpp? > > I did perform a test using netmap-fwd ( > > https://github.com/Netgate/netmap-fwd) > > I did look at the code and it appears that the packets are processed > > as a batch in the application. But each packet is passed through the > > complete IP stack in the application, before the next one is processed. > > With this application it was possible to reach about 1.4Mpps for > > 64-byte packets, and 812 kpps for 1518 byte packets I haven't done any > > other tweaking on the FreeBSD box yet. It is running FreeBSD 13.0 > > > > > Did you verify that the TREX setup can perform at line rate when > > connected > > > back to back? > > We did tests with TREX back to back yesterday and we reached the > following. > > +-------------+------------------+ > > | Packet Size | Throughput (pps) | > > +-------------+------------------+ > > | 64 bytes | 14.570 Mpps | > > | 128 bytes | 8.466 kpps | > > | 256 bytes | 4.542 kpps | > > | 512 bytes | 2.354 kpps | > > | 1024 bytes | 1.200 kpps | > > | 1280 bytes | 965.042 kpps | > > | 1440 bytes | 857.795 kpps | > > | 1518 bytes | 814.690 kpps | > > +-------------+------------------+ > > > > > Which NICs are you using? > > We are using Intel X552 10 GbE SFP+ NIC's which is part of the Intel > > Xeon > > D-1537 SoC, on a SuperMicro X10SDV-8C-TLN4F+ Board. > > > > I will also put the results on the github repository > > https://github.com/ftk-ntq/vpp/wiki > > and will update as we get some more information > > > > Kind Regards > > Francois > > > > > > > > > > > Important Notice: > > > > > > This e-mail and its contents are subject to the Nanoteq (Pty) Ltd > > > e-mail > > legal > > > notice available at: > > > http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx > > > _______________________________________________ > > > freebsd-net@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " > > > > > > Important Notice: > > > > This e-mail and its contents are subject to the Nanoteq (Pty) Ltd > > e-mail legal notice available at: > > http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > Important Notice: This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail le= gal notice available at: http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx