From owner-freebsd-net@freebsd.org Mon May 17 17:20:28 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1226D64E15E for ; Mon, 17 May 2021 17:20:28 +0000 (UTC) (envelope-from zec@fer.hr) Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-db5eur03on0609.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe0a::609]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FkQrB4MKCz4fn9; Mon, 17 May 2021 17:20:26 +0000 (UTC) (envelope-from zec@fer.hr) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=W3T/TJEL6vwJpHFYaOX/Enk9rZHEez3+769JGe+JyU9IrLhpf1um6BTISS0iyaj25e8Urtv+N+esx0mylRlZ/+TzPv6DWGkWD1yCz2+t5E7ISLHH6RUQyS8imWZ5SdGsUL9kBkarfYbq9lCiVX5coaOPbey7u/KFWi/HVdyxOTG+Q7Zq4HQJHTvBP8XS2hIYBQ6oVYHkB7DoJhRGt9lSSiWUusYkZ/ILFO18YPtQLiq1BP9/xDvqyWd4uMIPn/XPyNi0fTZ2yfyfllsrJXqRS1izojrZ+XY/sPtrjGHC2rzeCQkB398TOaWg+M9PEc7+rsfK0pbC++4kQt+9+GMkhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=p8rASC8ZC0Jy5z3adOIQJ9PpF5mUrmdT/h1ALV4PSPE=; b=BttZx2opnKV9nhBpV1ZFaCe3rqKWeN4E5XvicDCuXlwlZQ66YwXzhLuugmRf635SqMRMT7v72L54NmOkFzkJN2bXe9nDbvTmViYdB3h/uHWHG7S4+hrZVmnr6ZetQ1miYg5lHFaZTDow94TX8UKbiJ4KxeURR74LPZvvIG3NBl347xI0mNuODEFkejmy9f7zU4ttw9+PuN3PHQbSiH5hDmYk13L/uIr/Amiv63rcvotkEmawUyqnBCxMGmGXi3UfVRo3Xw36VXaaFphQcu1z19atKDSn0447nvu0mafiFiIX+T0Xho3uNQXNBVSkX0GuPiyRTu8bJaXJBcm6mSeceg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fer.hr; dmarc=pass action=none header.from=fer.hr; dkim=pass header.d=fer.hr; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ferhr.onmicrosoft.com; s=selector2-ferhr-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=p8rASC8ZC0Jy5z3adOIQJ9PpF5mUrmdT/h1ALV4PSPE=; b=mN75ioRtSpmbWNFPLucoaSeguE8H37G/GTJytNrCcYO/OQFsBLRp+o4A5TeUudQdUHLKNd65mBBXXnJXHcRgaKovVhfQPRb/9OJ4LjVE6yqdC+iFGNQ95TUDRE+a5WYitTLYbPiHX1e0sLjIZXM1RSlZje2gemkEoJa3eb3lKws= Received: from VE1PR08MB4783.eurprd08.prod.outlook.com (2603:10a6:802:a9::16) by VI1PR0801MB1630.eurprd08.prod.outlook.com (2603:10a6:800:59::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4129.25; Mon, 17 May 2021 17:20:23 +0000 Received: from VE1PR08MB4783.eurprd08.prod.outlook.com ([fe80::99b1:b848:51cb:27e8]) by VE1PR08MB4783.eurprd08.prod.outlook.com ([fe80::99b1:b848:51cb:27e8%4]) with mapi id 15.20.4129.031; Mon, 17 May 2021 17:20:23 +0000 Date: Mon, 17 May 2021 19:20:54 +0200 From: Marko Zec To: Francois ten Krooden Cc: Vincenzo Maffione , "freebsd-net@freebsd.org" , Jacques Fourie Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD Message-ID: <20210517192054.0907beea@x23> In-Reply-To: References: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; amd64-portbld-freebsd11.4) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Originating-IP: [31.147.121.219] X-ClientProxiedBy: VI1PR07CA0286.eurprd07.prod.outlook.com (2603:10a6:800:130::14) To VE1PR08MB4783.eurprd08.prod.outlook.com (2603:10a6:802:a9::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from x23 (31.147.121.219) by VI1PR07CA0286.eurprd07.prod.outlook.com (2603:10a6:800:130::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4150.11 via Frontend Transport; Mon, 17 May 2021 17:20:23 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3306e117-5fb3-4e33-8686-08d919580e01 X-MS-TrafficTypeDiagnostic: VI1PR0801MB1630: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oma6ukMMng9GHa/sO4hBcwHQO9s9Rc/MCZ3PsI+LJWKihUE0Ms3yFxgD/mfwdcu4LKXQSY3I3xrlCJhRrawwCxXEMBOcMVKI3XxzBW6Fpwwd4Fje011dg2IucGUp99sqJ/WD7hvl7+kfJN4/t32YQJWw04kmHgm0stIDq8pZU6okhWym3g1bRQdUOsLb+e/wVOWnKKZ4SpuHJ+Q86hV0p7ZPuK/7xQQOxSmvzkSMET3RfEmpL7RIgqmQlN9DKBOF2tJCJEJQQ6gJQBUolOclrEcO3m8pkjK9LYDl2069b9MVGeGLx15E4bV9zxrNiH9BKGelqUaatli5juWsZdKRXtpA/Sg8++swGoBDVvPNLaZhHPcQDsnijQwj1uyhMTdmLX7io/5ru947Xlo+sJ2RB/1NHTANzGN5n43tVdRvqtNdsYhwDqo+8861gK4hTQBKVbCxvt7zXNNzx74xx5BhV+rt3qc9/q5kWeQKDRlPKc42bbb7n7DllzAZoCZ0aWE7KqVFDMpp8FlOeZQkkHrnQhqibojEbNLP0UoPaHE1rAej2GydAhIsrPPab3xIPFYWvcqaC3SGlsUVcYmXg+xtgUQterohrP9Cz1VPUz8NrbvZpMRzKl/CkDWQEOA1kQVJQsN/xLP5HrAWMptVkT+NdCb0TfkrQDYfo7HjClAIKnNxMLbnV/BGsD4t730cTeyi4icBEJ9XsfgVMPfx4v0PUFeKkjzc1Rduh1IuBrEW6R0= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VE1PR08MB4783.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(39830400003)(376002)(346002)(396003)(366004)(136003)(6496006)(86362001)(6666004)(4326008)(52116002)(53546011)(66946007)(6916009)(83380400001)(16526019)(33716001)(2906002)(8936002)(316002)(55016002)(786003)(9686003)(8676002)(956004)(54906003)(186003)(1076003)(966005)(26005)(66556008)(66476007)(9576002)(5660300002)(478600001)(38350700002)(38100700002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?JWwrUo/S2Yx7AsaEQ4kr4BdVbRtOgjS7So25jaJWMAuj5yNjXwVMYDEnjCln?= =?us-ascii?Q?u+YrgnYXC0ueaeidAkBeatzBlOXbNs4+WdblspKoL41OdHUrjfwfFsABAlm8?= =?us-ascii?Q?5D1Iib4neeuAZu52C2u3vLCz44e+ae9vvqlNt8O1oUwjYP8lmyiYfGp2JWXC?= =?us-ascii?Q?HrL7bVt9rXRWQ9nWpwGRUCSYukPePZMDxLJuhPupyeETjXPAZ6vkrdIBBNV4?= =?us-ascii?Q?9I9V6ZSMX4MYm6nhBhETsNtMHQoUmrOHEkAcyIu1LaDL/aJMUXSzXu/IF3ZB?= =?us-ascii?Q?oUyFgFZjoi7n0iL2eReWXsxf8QKVvA/FfATvGbCnggq8COupqIwSxFRSCCdB?= =?us-ascii?Q?sizWs2P7XiOrSzjP6PLrZF5fAV0kFO19Hv5CGmrswhBnD+gGi3m0xFM0SF3n?= =?us-ascii?Q?ZpeZcCEirU3eUL40xbDhXhPPv7bXWxBpRCcOemXVkhH7BXTIjxKnyJpMPwXg?= =?us-ascii?Q?oDAWR/+mIbX2hEzkeqTNQILhm8DZznxuIf/yYbYMa9kgPDiv5nFOPJVVNvR2?= =?us-ascii?Q?XLH1OdbSkwgCzoAUFL0ZUpE1cpOnweCdLegnm0lUAcWlp7uE4pK/qGMkH2RO?= =?us-ascii?Q?yqxP1hdi9hBsMhxUcHk07z6zMbPcgZPR1wYe6cFnH0+E3Lppul58zLDH6/ih?= =?us-ascii?Q?cu/ZrhRjTlQyCA1I2CkXcYDTUntev1SFL0fgybRnNv36luqzqhmYXkc1SO84?= =?us-ascii?Q?JXuFJH9NAlJz3vR3bqNNoASiQUsiguUiXTIXxcP+XcyNWZXZcnMVfU0yTWpp?= =?us-ascii?Q?Tmm8Qk7I/GmjBGakhrq9IFZAtrN4qs6k5jtwdVqLLgUp151xWysjmMZg4KoF?= =?us-ascii?Q?23y7RzameQlGdf1o4sIeC1eAN/kl7GHIzuObZmcSZgfGETiYnvp9m+/gLZfI?= =?us-ascii?Q?sYphKSWYm8Ujma3DAZhPuH3QAbUPEhi0xfNhmrt6uEuJSGLT1kxzl84P5IQS?= =?us-ascii?Q?g7lFXCisQTVbRgRXmWpWfU8fSPBzfU4sij3KqoYD1V1+MkiRqA0Bn0+JkAjc?= =?us-ascii?Q?zn4XR+LDXzuc6UkoLO1B+hcaqHU1x/9/5k4s2Fhgl8+gJ7RwWatlWWviXtft?= =?us-ascii?Q?dT8bA6OYn32PHa3u6vRFw+1f/Q/VOANUPpqiCB9FSEH2I7doJ5ZxnLcyK8hb?= =?us-ascii?Q?eOETJ6wOI2Dag0bHWNYwvxjC9unMlV51ORzVo3qf7Mfu4qEGuVjZ1OKLafeM?= =?us-ascii?Q?inatbpLG/PlQKNWc2bJljCueHHqBaGDF22MLYaUeOVfmxRuGZ9CtVRQaPiPL?= =?us-ascii?Q?HpT3B/dgZSMUtyozUNDUzPxFgqjC5wxbB9MVSNKOzdwaNH3iZqY8Ps65PrYa?= =?us-ascii?Q?uMxdXacbApsi3mUBShCufl5T?= X-OriginatorOrg: fer.hr X-MS-Exchange-CrossTenant-Network-Message-Id: 3306e117-5fb3-4e33-8686-08d919580e01 X-MS-Exchange-CrossTenant-AuthSource: VE1PR08MB4783.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2021 17:20:23.5211 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: ca71eddc-cc7b-4e5b-95bd-55b658e696be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cPvyvWvDLa3dy2J7Uuz61l6PKY1jEcw1kke6u8zR52F5iK0wjHgppdTKE5qKCrwN X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0801MB1630 X-Rspamd-Queue-Id: 4FkQrB4MKCz4fn9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ferhr.onmicrosoft.com header.s=selector2-ferhr-onmicrosoft-com header.b=mN75ioRt; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=none; spf=pass (mx1.freebsd.org: domain of zec@fer.hr designates 2a01:111:f400:fe0a::609 as permitted sender) smtp.mailfrom=zec@fer.hr X-Spamd-Result: default: False [-2.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[ferhr.onmicrosoft.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RECEIVED_SPAMHAUS_PBL(0.00)[31.147.121.219:received]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe0a::609:from]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[ferhr.onmicrosoft.com:s=selector2-ferhr-onmicrosoft-com]; FREEFALL_USER(0.00)[zec]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[fer.hr]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe0a::609:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a01:111:f400:fe0a::609:from]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; SUSPICIOUS_RECIPS(1.50)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 May 2021 17:20:28 -0000 On Mon, 17 May 2021 09:53:25 +0000 Francois ten Krooden wrote: > On 2021/05/16 09:22, Vincenzo Maffione wrote: > > > > > Hi, > > Yes, you are not using emulated netmap mode. > > > > In the test setup depicted here > > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap- > > interfaces#test-setup > > I think you should really try to replace VPP with the netmap > > "bridge" application (tools/tools/netmap/bridge.c), and see what > > numbers you get. > > > > You would run the application this way > > # bridge -i ix0 -i ix1 > > and this will forward any traffic between ix0 and ix1 (in both > > directions). > > > > These numbers would give you a better idea of where to look next > > (e.g. VPP code improvements or system tuning such as NIC > > interrupts, CPU binding, etc.). > > Thank you for the suggestion. > I did run a test with the bridge this morning, and updated the > results as well. +-------------+------------------+ > | Packet Size | Throughput (pps) | > +-------------+------------------+ > | 64 bytes | 7.197 Mpps | > | 128 bytes | 7.638 Mpps | > | 512 bytes | 2.358 Mpps | > | 1280 bytes | 964.915 kpps | > | 1518 bytes | 815.239 kpps | > +-------------+------------------+ I assume you're on 13.0 where netmap throughput is lower compared to 11.x due to migration of most drivers to iflib (apparently increased overhead) and different driver defaults. On 11.x I could move 10G line rate from one ix to another at low CPU freqs, where on 13.x the CPU must be set to max speed, and still can't do 14.88 Mpps. #1 thing which changed: default # of packets per ring dropped down from 2048 (11.x) to 1024 (13.x). Try changing this in /boot/loader.conf: dev.ixl.0.iflib.override_nrxds=2048 dev.ixl.0.iflib.override_ntxds=2048 dev.ixl.1.iflib.override_nrxds=2048 dev.ixl.1.iflib.override_ntxds=2048 etc. For me this increases the throughput of bridge -i netmap:ixl0 -i netmap:ixl1 from 9.3 Mpps to 11.4 Mpps #2: default interrupt moderation delays seem to be too long. Combined with increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62 (default) to 40 increases the throughput further from 11.4 to 14.5 Mpps Hope this helps, Marko > Besides for the 64-byte and 128-byte packets the other sizes where > matching the maximum rates possible on 10Gbps. This was when the > bridge application was running on a single core, and the cpu core was > maxing out at a 100%. > > I think there might be a bit of system tuning needed, but I suspect > most of the improvement would be needed in VPP. > > Regards > Francois