From nobody Tue Mar 5 07:31:45 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TpnM45X90z5C4Ft for ; Tue, 5 Mar 2024 07:32:00 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TpnM367lMz4FLF for ; Tue, 5 Mar 2024 07:31:59 +0000 (UTC) (envelope-from kib@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kib@freebsd.org) smtp.mailfrom=kib@freebsd.org Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 4257Vjud076621; Tue, 5 Mar 2024 09:31:48 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 4257Vjud076621 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 4257VjVi076620; Tue, 5 Mar 2024 09:31:45 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Tue, 5 Mar 2024 09:31:45 +0200 From: Konstantin Belousov To: net@freebsd.org Cc: yohadt@nvidia.com, ddaniel@nvidia.com Subject: IPSEC inline offload Message-ID: List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[kib]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[net@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_NONE(0.00)[]; RCPT_COUNT_THREE(0.00)[3] X-Rspamd-Queue-Id: 4TpnM367lMz4FLF NVIDIA networking developed support for inline IPSEC offload on FreeBSD. It consists of the common infrastructure part, and changes for mlx5en(4) driver fitting the kernel counterpart. Inline offload moves (almost) all IPSEC processing of the packet from CPU and crypto accelerators, to the network card. Put it differently, on Tx card is fed with plain-text packets and sends out encapsulated and encrypted/authenticated packet on wire. On Rx, card verifies authentification and then does decryption/decapsulation, pushing the plain text packet to the host' IP stack. Some additions were made to the PF_KEY socket interface to allow more fine control of the offload, and to get some visibility into the offload operation. Otherwise, kernel with offload support is compatible with the existing interfaces, in particular, existing IKE daemons work with it as is. Both SPD and SAD are pushed to the card. Right now, driver supports only IPv4 AES-GCM transport mode, UDP ESP encapsulation (NAT-T), and several stateless offloads, most significant of which is TSO. RoCE over offloaded IPSEC works seamlessly. No other modes are supported. This is not a fundamental restriction of the design, only a current state in the development efforts. Architectural limitation of the hardware plus current infrastructure design is lack of support for fragmentation. The customer which driven the efforts does not need it, and for most interesting workloads it is not needed anyway. Also see above about TSO support. In some not scientific comparisions we made, between two FreeBSD hosts, single-stream TCP performance measured with iperf3: non-offloaded IPSEC vs non-offloaded IPSEC 3Mbit/sec offloaded IPSEC (traffic src) vs non-offloaded IPSEC 3.2Gbit/sec Patches are currently split into two jumbo reviews: infra https://reviews.freebsd.org/D44219 mlx5 driver https://reviews.freebsd.org/D44220