Date: Fri, 19 Jul 2024 16:26:43 +0200 From: Kristof Provost <kp@FreeBSD.org> To: Konstantin Belousov <kib@FreeBSD.org> Cc: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: Re: git: ef2a572bf6bd - main - ipsec_offload: kernel infrastructure Message-ID: <E73D7994-C3C4-4823-BBD5-DEB2AEB6663E@FreeBSD.org> In-Reply-To: <202407121125.46CBP8eo093121@gitrepo.freebsd.org> References: <202407121125.46CBP8eo093121@gitrepo.freebsd.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On 12 Jul 2024, at 13:25, Konstantin Belousov wrote: > The branch main has been updated by kib: > > URL: > https://cgit.FreeBSD.org/src/commit/?id=ef2a572bf6bdcac97ef29ce631d2f50f938e1ec8 > > commit ef2a572bf6bdcac97ef29ce631d2f50f938e1ec8 > Author: Konstantin Belousov <kib@FreeBSD.org> > AuthorDate: 2021-08-22 19:38:04 +0000 > Commit: Konstantin Belousov <kib@FreeBSD.org> > CommitDate: 2024-07-12 04:27:58 +0000 > > ipsec_offload: kernel infrastructure > > Inline IPSEC offload moves almost whole IPSEC processing from the > CPU/MCU and possibly crypto accelerator, to the network card. > > The transmitted packet content is not touched by CPU during TX > operations, kernel only does the required policy and security > association lookups to find out that given flow is offloaded, and > then > packet is transmitted as plain text to the card. For driver > convenience, > a metadata is attached to the packet identifying SA which must > process > the packet. Card does encryption of the payload, padding, > calculates > authentication, and does the reformat according to the policy. > > Similarly, on receive, card does the decapsulation, decryption, > and > authentification. Kernel receives the identifier of SA that was > used to process the packet, together with the plain-text packet. > > Overall, payload octets are only read or written by card DMA > engine, > removing a lot of memory subsystem overhead, and saving CPU time > because > IPSEC algos calculations are avoided. > > If driver declares support for inline IPSEC offload (with the > IFCAP2_IPSEC_OFFLOAD capability set and registering method table > struct > if_ipsec_accel_methods), kernel offers the SPD and SAD to driver. > Driver decides which policies and SAs can be offloaded based on > hardware capacity, and acks/nacks each SA for given interface to > kernel. Kernel needs to keep this information to make a decision > to > skip software processing on TX, and to assume processing already > done > on RX. This shadow SPD/SAD database of offloads is rooted from > policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and > SAs > (struct secasvar accel_ipfs, struct ifp_handle_sav). > > Some extensions to the PF_KEY socket allow to limit interfaces for > which given SP/SA could be offloaded (proposed for offload). > Also, > additional statistics extensions allow to observe > allocation/octet/use > counters for specific SA. > > Since SPs and SAs are typically instantiated in non-sleepable > context, > while offloading them into card is expected to require costly > async > manipulations of the card state, calls to the driver for offload > and > termination are executed in the threaded taskqueue. It also > solves > the issue of allocating resources needed for the offload database. > Neither ipf_handle_sp nor ipf_handle_sav do not add reference to > the > owning SP/SA, the offload must be terminated before last reference > is > dropped. ipsec_accel only adds transient references to ensure > safe > pointer ownership by taskqueue. > > Maintaining the SA counters for hardware-accelerated packets is > the > duty of the driver. The helper > ipsec_accel_drv_sa_lifetime_update() > is provided to hide accel infrastructure from drivers which would > use > expected callout to query hardware periodically for updates. > > Reviewed by: rscheff (transport, stack integration), np > Sponsored by: NVIDIA networking > Differential revision: https://reviews.freebsd.org/D44219 > --- > sys/conf/files | 2 + > sys/conf/options | 1 + > sys/modules/ipsec/Makefile | 5 +- > sys/netipsec/ipsec.c | 17 + > sys/netipsec/ipsec.h | 11 + > sys/netipsec/ipsec_input.c | 11 + > sys/netipsec/ipsec_offload.c | 1061 > ++++++++++++++++++++++++++++++++++++++++++ > sys/netipsec/ipsec_offload.h | 191 ++++++++ > sys/netipsec/ipsec_output.c | 15 + > sys/netipsec/ipsec_pcb.c | 38 +- > sys/netipsec/key.c | 270 ++++++++++- > sys/netipsec/key.h | 6 + > sys/netipsec/key_debug.c | 5 + > sys/netipsec/keydb.h | 14 + > 14 files changed, 1628 insertions(+), 19 deletions(-) > I’m seeing messages like `ipsec_accel_sa_install_newkey: spi 0x1001 flags 0x40 seq 0` running the test suite now. Are those stray debug printfs? Best regards, Kristof [-- Attachment #2 --] <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8"> </head> <body><div style="font-family: sans-serif;"><div class="markdown" style="white-space: normal;"> <p dir="auto">On 12 Jul 2024, at 13:25, Konstantin Belousov wrote:</p> </div><div class="plaintext" style="white-space: normal;"><blockquote style="margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #136BCE; color: #136BCE;"><p dir="auto">The branch main has been updated by kib:</p> <p dir="auto">URL: <a href="https://cgit.FreeBSD.org/src/commit/?id=ef2a572bf6bdcac97ef29ce631d2f50f938e1ec8">https://cgit.FreeBSD.org/src/commit/?id=ef2a572bf6bdcac97ef29ce631d2f50f938e1ec8</a></p>; <p dir="auto">commit ef2a572bf6bdcac97ef29ce631d2f50f938e1ec8 <br> Author: Konstantin Belousov <kib@FreeBSD.org> <br> AuthorDate: 2021-08-22 19:38:04 +0000 <br> Commit: Konstantin Belousov <kib@FreeBSD.org> <br> CommitDate: 2024-07-12 04:27:58 +0000</p> <p dir="auto"> ipsec_offload: kernel infrastructure</p> <p dir="auto"> Inline IPSEC offload moves almost whole IPSEC processing from the <br> CPU/MCU and possibly crypto accelerator, to the network card.</p> <p dir="auto"> The transmitted packet content is not touched by CPU during TX <br> operations, kernel only does the required policy and security <br> association lookups to find out that given flow is offloaded, and then <br> packet is transmitted as plain text to the card. For driver convenience, <br> a metadata is attached to the packet identifying SA which must process <br> the packet. Card does encryption of the payload, padding, calculates <br> authentication, and does the reformat according to the policy.</p> <p dir="auto"> Similarly, on receive, card does the decapsulation, decryption, and <br> authentification. Kernel receives the identifier of SA that was <br> used to process the packet, together with the plain-text packet.</p> <p dir="auto"> Overall, payload octets are only read or written by card DMA engine, <br> removing a lot of memory subsystem overhead, and saving CPU time because <br> IPSEC algos calculations are avoided.</p> <p dir="auto"> If driver declares support for inline IPSEC offload (with the <br> IFCAP2_IPSEC_OFFLOAD capability set and registering method table struct <br> if_ipsec_accel_methods), kernel offers the SPD and SAD to driver. <br> Driver decides which policies and SAs can be offloaded based on <br> hardware capacity, and acks/nacks each SA for given interface to <br> kernel. Kernel needs to keep this information to make a decision to <br> skip software processing on TX, and to assume processing already done <br> on RX. This shadow SPD/SAD database of offloads is rooted from <br> policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs <br> (struct secasvar accel_ipfs, struct ifp_handle_sav).</p> <p dir="auto"> Some extensions to the PF_KEY socket allow to limit interfaces for <br> which given SP/SA could be offloaded (proposed for offload). Also, <br> additional statistics extensions allow to observe allocation/octet/use <br> counters for specific SA.</p> <p dir="auto"> Since SPs and SAs are typically instantiated in non-sleepable context, <br> while offloading them into card is expected to require costly async <br> manipulations of the card state, calls to the driver for offload and <br> termination are executed in the threaded taskqueue. It also solves <br> the issue of allocating resources needed for the offload database. <br> Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the <br> owning SP/SA, the offload must be terminated before last reference is <br> dropped. ipsec_accel only adds transient references to ensure safe <br> pointer ownership by taskqueue.</p> <p dir="auto"> Maintaining the SA counters for hardware-accelerated packets is the <br> duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update() <br> is provided to hide accel infrastructure from drivers which would use <br> expected callout to query hardware periodically for updates.</p> <p dir="auto"> Reviewed by: rscheff (transport, stack integration), np <br> Sponsored by: NVIDIA networking <br> Differential revision: <a href="https://reviews.freebsd.org/D44219">https://reviews.freebsd.org/D44219</a>; <br> --- <br> sys/conf/files | 2 + <br> sys/conf/options | 1 + <br> sys/modules/ipsec/Makefile | 5 +- <br> sys/netipsec/ipsec.c | 17 + <br> sys/netipsec/ipsec.h | 11 + <br> sys/netipsec/ipsec_input.c | 11 + <br> sys/netipsec/ipsec_offload.c | 1061 ++++++++++++++++++++++++++++++++++++++++++ <br> sys/netipsec/ipsec_offload.h | 191 ++++++++ <br> sys/netipsec/ipsec_output.c | 15 + <br> sys/netipsec/ipsec_pcb.c | 38 +- <br> sys/netipsec/key.c | 270 ++++++++++- <br> sys/netipsec/key.h | 6 + <br> sys/netipsec/key_debug.c | 5 + <br> sys/netipsec/keydb.h | 14 + <br> 14 files changed, 1628 insertions(+), 19 deletions(-)</p> <br></blockquote></div> <div class="markdown" style="white-space: normal;"> <p dir="auto">I’m seeing messages like <code style="padding: 0 0.25em; background-color: #E4E4E4;">ipsec_accel_sa_install_newkey: spi 0x1001 flags 0x40 seq 0</code> running the test suite now.<br> Are those stray debug printfs?</p> <p dir="auto">Best regards,<br> Kristof</p> </div> </div> </body> </html>help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E73D7994-C3C4-4823-BBD5-DEB2AEB6663E>