From owner-freebsd-net@freebsd.org Mon May 10 14:39:32 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 245A163A30F for ; Mon, 10 May 2021 14:39:32 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-ua1-f42.google.com (mail-ua1-f42.google.com [209.85.222.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ff3bk51T5z3hn6 for ; Mon, 10 May 2021 14:39:30 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-ua1-f42.google.com with SMTP id g24so5274891uak.11 for ; Mon, 10 May 2021 07:39:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=6ogBuzypv0KutxwTfvDfa0MKvgv/9ywnoUXA+C8SktA=; b=WesELGSi4CPX3GCohZHCdEzrr2WJSwjN4ZfAgTkMoD1Pvn5vk/ChJGGE+SEGjFhMjD 2gmeC9f775/5R2SlYkA/BctlzOaly31c27zCHfD4PmMXOWfEq3A0U9xFltNpo/iYLyP+ Jzq4rSyZjVE5bai9814BdDbXBOEfehozxdXbgaC+M8GJeuTrMWkDf8RnoDMGj/StDuVa o21MNz4gZN8ZJeja8iIfZAhgLIEYFx2Uj9klVPYCdHWC1MPHxprR5MdUWx/C+cuTWUXw Z+vhVp9LJbzu8cAKklkqXyk+Xxg3nRyrenrMyJk+wB00Xs/+wymXl3ulWr10bfUp2drB Z1pA== X-Gm-Message-State: AOAM5322O9UkQdl163+z5W6KP1T+hvf6wADhSaoY75mK9f9NZs0aYL5b Z0biFeJgD/tcUrsJk2juU7i1rYvEIbAnjeU8II6wGf+Gt0PtmQ== X-Google-Smtp-Source: ABdhPJz6WxVdfh/fK2Nbms24z9ygSSipuE0C0/2I6SpInzplvgM1yOzorHGyZ3OXDudUefFBolXd61X1gAfql78HvtM= X-Received: by 2002:ab0:2685:: with SMTP id t5mr19983774uao.140.1620657569599; Mon, 10 May 2021 07:39:29 -0700 (PDT) MIME-Version: 1.0 References: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> In-Reply-To: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> From: Luigi Rizzo Date: Mon, 10 May 2021 16:39:18 +0200 Message-ID: Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD To: Rainer Duffner Cc: Francois ten Krooden , "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4Ff3bk51T5z3hn6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rizzounipi@gmail.com designates 209.85.222.42 as permitted sender) smtp.mailfrom=rizzounipi@gmail.com X-Spamd-Result: default: False [-2.97 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[rizzo@iet.unipi.it,rizzounipi@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_NEQ_ENVFROM(0.00)[rizzo@iet.unipi.it,rizzounipi@gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.222.42:from]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.966]; TAGGED_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[unipi.it]; SPAMHAUS_ZRD(0.00)[209.85.222.42:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.222.42:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.222.42:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-Mailman-Approved-At: Mon, 10 May 2021 15:48:53 +0000 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 May 2021 14:39:32 -0000 [repost since it appears to be blocked] On Mon, May 10, 2021 at 2:42 PM Rainer Duffner wro= te: > > > > > Am 10.05.2021 um 13:08 schrieb Francois ten Krooden : > > > > Greetings > > > > We have a vested interest in high-speed IPsec VPN on FreeBSD. We have s= tarted with the porting of VPP (https://fd.io/) to FreeBSD. > > > > Currently we have VPP compiled and running with netmap. The speeds we m= easure are nowhere near the performance of a 10Gbps link, at around 350kpps= for 1500 byte IPv4 packets. We suspect the biggest issue is related to how= VPP implements huge pages (Linux) and our modifications to support super p= ages on FreeBSD. 350kpps is way too low for being related to tlb and page size issues. I suspect that you are either using the "emulated" netmap mode, which runs on top of mbufs and may involve extra data copies on the receive path, or the test itself is e.g. using TCP and congestion control or limited window size throttle down the rate. I'd retry the test with some open loop traffic source/sink, and using first all possible low level APIs (sockets/mbufs; emulated netmap; native netmap; your vpp port) with small and large packets and increasing packet rates, to see where the limits are with each. That should give you good hints to figure out what is making the performance so bad. Even better if you can run the test between one known-good endpoint so you can test separately the tx and tx sides. Cheers Luigi