From owner-freebsd-net@freebsd.org Mon May 10 13:39:21 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 59BE2637B5E for ; Mon, 10 May 2021 13:39:21 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-vs1-xe36.google.com (mail-vs1-xe36.google.com [IPv6:2607:f8b0:4864:20::e36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ff2GH3xHjz3NgL for ; Mon, 10 May 2021 13:39:19 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-vs1-xe36.google.com with SMTP id l4so1554096vsa.1 for ; Mon, 10 May 2021 06:39:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=QcCdciww+QtR+mIRYAtUYKowqRi4ycQe38rMv1+WG9Y=; b=uYXsFUhOdSywdDFCkOgAt1SlqcZ9ezSOIKA4z8dN1bFxgncOcARDayiU3eoQa09z/3 ZtpkUMRR2VC1H3EdHP4NB7mxLKsY07gfKGApim6hLPOl0pvugVbVP59pSuCF0xxmoInR tEC7I2ZMk1J0/mYTmo9O5MKBViBehihVq2Tt+wy8tkUVvbrTCqWTB1w00i9pVI6gm5jW Xc+OvhYWVDhJLywpAARjLlxAUnxoCzHIb5UU9MJca0V33KLcjHD+57SG7OdUCyyzP8BH YkXi2WoQpCG7VLVM7UPjyav2K4egJTYxJPLbjkOKEq/uhv08SivDzRvcMPGCnBeN6TRQ V+5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=QcCdciww+QtR+mIRYAtUYKowqRi4ycQe38rMv1+WG9Y=; b=iRcnv4188sqaZLhbqdEDLKSNXqHVBN9s1weHfbyCas2F6bDu5Iqq/BmSj49OhsJuhN WVhwvYMibDZVRajGK+3DK6aw5Gkf0KQGRMChLS1LCBxhTR/PJUNp9vodaiA4xM0GAzdI UnRdTT+1gifodIvQSqIsgjb/Ap8d5OiOEeCjT/elVBDSW6lpiNUbR4B8pnidag/fQvKY 1ZGIXYgA8SBgC5lTMOAI0vzBBRUHpbDL0i4kLjLbAvfuXRbW7UPXCEiW7mZGiU4jaBJ2 Qyfk82a2yMsmBAvytNju40hcxcfkjnPsIahOsQkUozqr9wm2bVnrdGf55wsmDpUyciV+ wdFQ== X-Gm-Message-State: AOAM530rfLMq4xMXG5XbO9oDbrClI+ZLHfkrVIoFFfCbQr13FxWtDN4h hLJWnU9sWvRRaPFUzMFUE3Xed4Pr7EjKLOa3Pg32Z0NspIg= X-Google-Smtp-Source: ABdhPJwK4RAtrJrNaRPEd+TyCb52jeYyWiHZIXaVha+gQygtus75ux33eM07cAOEOWXcUWbFT27dKTO0az2phSn0JgI= X-Received: by 2002:a67:5e42:: with SMTP id s63mr18287134vsb.14.1620653958542; Mon, 10 May 2021 06:39:18 -0700 (PDT) MIME-Version: 1.0 Sender: rizzo.unipi@gmail.com Received: by 2002:a67:13c4:0:0:0:0:0 with HTTP; Mon, 10 May 2021 06:39:18 -0700 (PDT) In-Reply-To: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> References: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> From: Luigi Rizzo Date: Mon, 10 May 2021 15:39:18 +0200 X-Google-Sender-Auth: rfE-9xChrsqcMpdf15MGZcdykbI Message-ID: Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD To: Rainer Duffner Cc: Francois ten Krooden , "freebsd-net@freebsd.org" X-Rspamd-Queue-Id: 4Ff2GH3xHjz3NgL X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=uYXsFUhO; dmarc=none; spf=pass (mx1.freebsd.org: domain of rizzounipi@gmail.com designates 2607:f8b0:4864:20::e36 as permitted sender) smtp.mailfrom=rizzounipi@gmail.com X-Spamd-Result: default: False [-3.20 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[rizzo@iet.unipi.it,rizzounipi@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::e36:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_NEQ_ENVFROM(0.00)[rizzo@iet.unipi.it,rizzounipi@gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[unipi.it]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::e36:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::e36:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-Mailman-Approved-At: Mon, 10 May 2021 15:48:30 +0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 May 2021 13:39:21 -0000 On Monday, May 10, 2021, Rainer Duffner wrote: > > > > Am 10.05.2021 um 13:08 schrieb Francois ten Krooden : > > > > Greetings > > > > We have a vested interest in high-speed IPsec VPN on FreeBSD. We have > started with the porting of VPP (https://fd.io/) to FreeBSD. > > > > Currently we have VPP compiled and running with netmap. The speeds we > measure are nowhere near the performance of a 10Gbps link, at around > 350kpps for 1500 byte IPv4 packets. We suspect the biggest issue is relat= ed > to how VPP implements huge pages (Linux) and our modifications to support > super pages on FreeBSD. 350kpps is way too low for being related to tlb and page size issues. I suspect that you are either using the "emulated" netmap mode, which runs on top of mbufs and may involve extra data copies on the receive path, or the test itself is e.g. using TCP and congestion control or limited window size throttle down the rate. I'd retry the test with some open loop traffic source/sink, and using first all possible low level APIs (sockets/mbufs; emulated netmap; native netmap; your vpp port) with small and large packets and increasing packet rates, to see where the limits are with each. That should give you good hints to figure out what is making the performance so bad. Even better if you can run the test between one known-good endpoint so you can test separately the tx and tx sides. Cheers Luigi > > > Apart from the above, there are remaining issues we need to sort out an= d > "Linuxisms" that need porting to FreeBSD, but this is going reasonably > well. We are working in a public Github repository and have started listi= ng > our issues there alongside the code. Our main working branch is "freebsd"= ( > https://github.com/ftk-ntq/vpp/tree/freebsd). > > > > Our aim with this mail is to get the discussion started on porting VPP > to FreeBSD and to invite interested parties to help with the effort. We > intend to upstream the work hoping that the original authors will adopt o= ur > ported code and continue maintaining future compatibility with FreeBSD. > > > > Some of our questions or comments to start the conversation: > > 1. netmap vs. DPDK (VPP relies on DPDK by default with the netmap > integration deprecated). Which will be the best to choose? > > 2. How to correctly implement using super pages / huge pages in FreeBSD > in order to allow VPP to allocate contiguous memory blocks for packet > buffers to process packets from the packet handling framework (netmap/DPD= K)? > > 3. What are suitable alternatives for reading information from procfs > and sysfs on FreeBSD? > > 4. Functionality relying on Linux epoll is currently supported using > epoll-shim. Is this the correct approach? > > > > Any help and input to aid in the effort will be greatly appreciated. > > > > > IIRC, somebody from Netgate (pfSense) had porting this and/or DPDK as a > =E2=80=9Espare time project=E2=80=9C. > > Not sure how far that actually proceeded. Maybe somebody else knows more? > > > Thanks for your efforts, in any case. > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > --=20 -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------