From owner-freebsd-net@freebsd.org Thu May 13 13:02:39 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 15984640183 for ; Thu, 13 May 2021 13:02:39 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-vs1-f41.google.com (mail-vs1-f41.google.com [209.85.217.41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FgsJZ2Jmzz4sWQ; Thu, 13 May 2021 13:02:38 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-vs1-f41.google.com with SMTP id o192so13585192vsd.7; Thu, 13 May 2021 06:02:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Zih0Qx+ckPeF9EFgGtcrbZQw+tQOEmUgbf3i1qPHBRA=; b=BxaVeLJ0dHrikXvgIAIslbRoNrSEyaiTxL6e/MPsdBTagu0KDlBmFVOc0Jp9DmHSB2 kTek++Lx18SwfvYULx5yLT4bdjQGvB3UTqFmX9gCXhI4bUMqW16RmYwebatJQGvHiCcv yWAMYtkmkBB2lWlGQ6QwJXWnju3DOnXQoc0JDo6acqk9ZnRNwCOOCT0XD+dspD4WfYSU mFqzBVZ1SqStIoTtbF4IHrlLF0T9fNsiEMX9dJZrMHfazsIlWRmvqjJwzCd/iM7X3Mg3 eX82GbOcInMUWJUWyqnGbZ1scjkX6xGmHgLL6o11omq5lNeT7RpNbXvrj194gT8OExhg gttQ== X-Gm-Message-State: AOAM532x2CoHz9ej3FFkphxJMj2YZk6vQPVwMfMatf1gK3wW2qvEbQSd r9E3NTDJwduulUAiokD9hkHUOZs+Ji44K54bq0rF6f6zPnA= X-Google-Smtp-Source: ABdhPJwTyWaTVD0gI8hqD9QICEe9DtHU5fuRwS8g34rP5lywu+n6iA5iesxpIHVD1jZkCIJ9mlu6O5VGRiZRYRiFYSE= X-Received: by 2002:a67:5e42:: with SMTP id s63mr35564185vsb.14.1620910957346; Thu, 13 May 2021 06:02:37 -0700 (PDT) MIME-Version: 1.0 References: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> <5cdff6ae35a8482a96a4fb40d5bff034@EX16-05.ad.unipi.it> In-Reply-To: From: Luigi Rizzo Date: Thu, 13 May 2021 15:02:26 +0200 Message-ID: Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD To: Francois ten Krooden Cc: Luigi Rizzo , Vincenzo Maffione , "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4FgsJZ2Jmzz4sWQ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rizzounipi@gmail.com designates 209.85.217.41 as permitted sender) smtp.mailfrom=rizzounipi@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[rizzo@iet.unipi.it,rizzounipi@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.217.41:from]; R_DKIM_NA(0.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[rizzo@iet.unipi.it,rizzounipi@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[unipi.it]; SPAMHAUS_ZRD(0.00)[209.85.217.41:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.217.41:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.217.41:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-Mailman-Approved-At: Thu, 13 May 2021 17:39:29 +0000 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 May 2021 13:02:39 -0000 On Thu, May 13, 2021 at 2:57 PM Luigi Rizzo wrote: > > On Thu, May 13, 2021 at 1:27 PM Francois ten Krooden wrote: > > > > > > On Thursday, 13 May 2021 13:05 Luigi Rizzo wrote: > > > > > > On Thu, May 13, 2021 at 10:42 AM Francois ten Krooden > > > wrote: > > > > > > > > Hi > > > > > > > > Just for info I ran a test using TREX (https://trex-tgn.cisco.com/) > > > > Where I just sent traffic in one direction through the box running FreeBSD > > > with VPP using the netmap interfaces. > > > > These were the results we found before significant packet loss started > > > occuring. > > > > +-------------+------------------+ > > > > | Packet Size | Throughput (pps) | > > > > +-------------+------------------+ > > > > | 64 bytes | 1.008 Mpps | > > > > | 128 bytes | 920.311 kpps | > > > > | 256 bytes | 797.789 kpps | > > > > | 512 bytes | 706.338 kpps | > > > > | 1024 bytes | 621.963 kpps | > > > > | 1280 bytes | 569.140 kpps | > > > > | 1440 bytes | 547.139 kpps | > > > > | 1518 bytes | 524.864 kpps | > > > > +-------------+------------------+ > > > > > > Those numbers are way too low for netmap. > > > > > > I believe you are either using the emulated mode, or issuing a system call on > > > every single packet. > > > > > > I am not up to date (Vincenzo may know better) but there used to be a sysctl > > > variable to control the operating mode: > > > > > > https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4 > > > > > > SYSCTL VARIABLES AND MODULE PARAMETERS > > > Some aspects of the operation of netmap and VALE are controlled > > > through > > > sysctl variables on FreeBSD (dev.netmap.*) and module parameters on > > > Linux > > > (/sys/module/netmap/parameters/*): > > > > > > dev.netmap.admode: 0 > > > Controls the use of native or emulated adapter mode. > > > > > > 0 uses the best available option; > > > > > > 1 forces native mode and fails if not available; > > > > > > 2 forces emulated hence never fails. > > > > > > If it still exists, try set it to 1. If the program fails, then you should figure out > > > why native netmap support is not compiled in. > > > > Thank you. I did set this to 1 specifically now and it still works. So then it should be running in native mode. > > > > I will dig a bit into the function that processes the incoming packets. > > The code I currently use was added to VPP in somewhere before 2016, so it might be that there is a bug in that code. > > Then try to instrument the code and see how many packets > you are getting on every RXSYNC system call. > > If the value is mostly/always 0-1 then there is some bug > with the (user) code that frees slots in the queue. Or another issue could be that your application spends too much time to process packets, so the bottleneck is user processing. The thing to monitor would be the time between system calls, divided by the number of packets processed in between .. ioctl(RXSYNC); t1 = get_nanoseconds(); n = t2 = get_nanoseconds() time_per_packet = (t2 - t1) / n; ioctl(RXSYNC); ... cheers luigi