From owner-freebsd-net@freebsd.org Tue May 18 07:32:18 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B7ED1642923 for ; Tue, 18 May 2021 07:32:18 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-yb1-xb2e.google.com (mail-yb1-xb2e.google.com [IPv6:2607:f8b0:4864:20::b2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fknl52xy9z3lQT for ; Tue, 18 May 2021 07:32:17 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by mail-yb1-xb2e.google.com with SMTP id d14so4075247ybe.3 for ; Tue, 18 May 2021 00:32:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FLsjbKgcRA1JQOqJgFnDN7qbCHGmygbql4SxVKc49rk=; b=n6VBQTZG2OaZcgdtBKGt5x7Q7SDpW7lvgAPNGwxYvt8zpl2n4OYbYFTtYVdgJ4xack FkKXObrEfSfPhQJXA3J4aWR1luBqKntsZ2uJR1wxnxCWN08hKomSFYt8YwVWMXGzCA7b Jdu2ZWlbEqcncAwJ5Agxus0WMm4LglOhIBoUk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FLsjbKgcRA1JQOqJgFnDN7qbCHGmygbql4SxVKc49rk=; b=KpoB0cHwhQBYIglaCGaiB3xKLxxbF4v652MI2uNu0ib7a9iHQ0qZ5qwE1duphkbAJW RuHInwXI2iTK4oOkkzUaRZTc5th1WCPqWFiZmyMFObn9RXn6I6inw3xiEJ1GiEi7koor 6InOf3JSlPBJOqIwBC1/Gj1GMQbZoxOo22ZxAtPtEtyW+Y/ht4Ev9fPpFh7n1BGk0v23 WMpT/cBXvazVX96qgaK6VFDpQtZJH+LqtvxLBWKsSDguhUSWyJCx1Kj78fIzVovr/JBy 7/6CAIn99q7v5HfA9QGWnLbUWiMGAvo9pP23Gg7lWLbxDDdSx+k+QdCjZ4F5wYlG+5nE pZPQ== X-Gm-Message-State: AOAM532mA0VXrPmAF4XR/QacWftYYJeV9PAXVO7yY+RVUtOuUCTCVJu/ N1Xu8yC95LmDC9mGTgNt0tm0++928r4i4QX+dgmXbQ== X-Google-Smtp-Source: ABdhPJxm4B3PnSleaw7/4YV6qjXVrjj3QQy6k4v7yzRtm6ee0zs1VWS90/QfxmSTAZM5RVEPkzoJ1GcS6CTndqbLGmE= X-Received: by 2002:a25:938a:: with SMTP id a10mr5154739ybm.429.1621323136396; Tue, 18 May 2021 00:32:16 -0700 (PDT) MIME-Version: 1.0 References: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> <20210517192054.0907beea@x23> In-Reply-To: <20210517192054.0907beea@x23> From: Kevin Bowling Date: Tue, 18 May 2021 00:32:05 -0700 Message-ID: Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD To: Marko Zec Cc: Francois ten Krooden , Jacques Fourie , Vincenzo Maffione , "freebsd-net@freebsd.org" X-Rspamd-Queue-Id: 4Fknl52xy9z3lQT X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none (invalid DKIM record) header.d=kev009.com header.s=google header.b=n6VBQTZG; dmarc=none; spf=pass (mx1.freebsd.org: domain of kevin.bowling@kev009.com designates 2607:f8b0:4864:20::b2e as permitted sender) smtp.mailfrom=kevin.bowling@kev009.com X-Spamd-Result: default: False [-1.79 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; FREEMAIL_CC(0.00)[nanoteq.com,gmail.com,freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCPT_COUNT_FIVE(0.00)[5]; DKIM_TRACE(0.00)[kev009.com:~]; NEURAL_HAM_SHORT(-0.99)[-0.993]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::b2e:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[kev009.com]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::b2e:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b2e:from]; R_DKIM_PERMFAIL(0.00)[kev009.com:s=google]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net]; SUSPICIOUS_RECIPS(1.50)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 May 2021 07:32:18 -0000 On Mon, May 17, 2021 at 10:20 AM Marko Zec wrote: > On Mon, 17 May 2021 09:53:25 +0000 > Francois ten Krooden wrote: > > > On 2021/05/16 09:22, Vincenzo Maffione wrote: > > > > > > > > Hi, > > > Yes, you are not using emulated netmap mode. > > > > > > In the test setup depicted here > > > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap- > > > interfaces#test-setup > > > I think you should really try to replace VPP with the netmap > > > "bridge" application (tools/tools/netmap/bridge.c), and see what > > > numbers you get. > > > > > > You would run the application this way > > > # bridge -i ix0 -i ix1 > > > and this will forward any traffic between ix0 and ix1 (in both > > > directions). > > > > > > These numbers would give you a better idea of where to look next > > > (e.g. VPP code improvements or system tuning such as NIC > > > interrupts, CPU binding, etc.). > > > > Thank you for the suggestion. > > I did run a test with the bridge this morning, and updated the > > results as well. +-------------+------------------+ > > | Packet Size | Throughput (pps) | > > +-------------+------------------+ > > | 64 bytes | 7.197 Mpps | > > | 128 bytes | 7.638 Mpps | > > | 512 bytes | 2.358 Mpps | > > | 1280 bytes | 964.915 kpps | > > | 1518 bytes | 815.239 kpps | > > +-------------+------------------+ > > I assume you're on 13.0 where netmap throughput is lower compared to > 11.x due to migration of most drivers to iflib (apparently increased > overhead) and different driver defaults. On 11.x I could move 10G line > rate from one ix to another at low CPU freqs, where on 13.x the CPU > must be set to max speed, and still can't do 14.88 Mpps. > I believe this issue is in the combined txrx interrupt filter. It is causing a bunch of unnecessary tx re-arms. > #1 thing which changed: default # of packets per ring dropped down from > 2048 (11.x) to 1024 (13.x). Try changing this in /boot/loader.conf: > > dev.ixl.0.iflib.override_nrxds=2048 > dev.ixl.0.iflib.override_ntxds=2048 > dev.ixl.1.iflib.override_nrxds=2048 > dev.ixl.1.iflib.override_ntxds=2048 > etc. > > For me this increases the throughput of > bridge -i netmap:ixl0 -i netmap:ixl1 > from 9.3 Mpps to 11.4 Mpps > > #2: default interrupt moderation delays seem to be too long. Combined > with increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62 > (default) to 40 increases the throughput further from 11.4 to 14.5 Mpps > > Hope this helps, > > Marko > > > > Besides for the 64-byte and 128-byte packets the other sizes where > > matching the maximum rates possible on 10Gbps. This was when the > > bridge application was running on a single core, and the cpu core was > > maxing out at a 100%. > > > > I think there might be a bit of system tuning needed, but I suspect > > most of the improvement would be needed in VPP. > > > > Regards > > Francois > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >