From owner-freebsd-hackers@freebsd.org Tue Oct 27 18:47:10 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 73843457AA4 for ; Tue, 27 Oct 2020 18:47:10 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CLLKT2Py6z4273; Tue, 27 Oct 2020 18:47:08 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 09RIl629013552 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 27 Oct 2020 11:47:06 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 09RIl67m013551; Tue, 27 Oct 2020 11:47:06 -0700 (PDT) (envelope-from jmg) Date: Tue, 27 Oct 2020 11:47:06 -0700 From: John-Mark Gurney To: Benjamin Kaduk Cc: Mark Johnston , Neel Chauhan , freebsd-hackers@freebsd.org Subject: Re: QAT driver Message-ID: <20201027184706.GF31099@funkthat.com> Mail-Followup-To: Benjamin Kaduk , Mark Johnston , Neel Chauhan , freebsd-hackers@freebsd.org References: <20201026200059.GA66299@raichu> <723fbd7326df42ce30cd5e361db9c736@neelc.org> <20201027032720.GB31663@raichu> <20201027045735.GJ39170@kduck.mit.edu> <20201027130754.GE31663@raichu> <20201027162445.GN39170@kduck.mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201027162445.GN39170@kduck.mit.edu> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Tue, 27 Oct 2020 11:47:06 -0700 (PDT) X-Rspamd-Queue-Id: 4CLLKT2Py6z4273 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-0.88 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-0.85)[-0.851]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.41)[-0.413]; NEURAL_HAM_MEDIUM(-0.82)[-0.816]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2020 18:47:10 -0000 Benjamin Kaduk wrote this message on Tue, Oct 27, 2020 at 09:24 -0700: > On Tue, Oct 27, 2020 at 09:07:54AM -0400, Mark Johnston wrote: > > On Mon, Oct 26, 2020 at 09:57:35PM -0700, Benjamin Kaduk wrote: > > > On Mon, Oct 26, 2020 at 11:27:20PM -0400, Mark Johnston wrote: > > > > On Mon, Oct 26, 2020 at 08:00:08PM -0700, Neel Chauhan wrote: > > > > > Hi, > > > > > > > > > > This is great news for me with my home HPE ML110 G10/Xeon 4108 server. > > > > > > > > > > However, I will not be able to test this patch unless it can get > > > > > backported to 12.1 or 12.2 once it's out, and I don't expect backporting > > > > > to happen. > > > > > > > > Indeed, it wouldn't appear before 12.3. > > > > > > > > > I have one question about this: will I be able to use this to accelerate > > > > > OpenSSL? Is additional code needed? > > > > > > > > In principle OpenSSL can make use of cryptodev(4) using the cryptodev > > > > engine, which would allow requests to be handled by qat(4) (or any other > > > > hardware crypto driver loaded in the kernel). I don't know that the > > > > cryptodev engine is really maintained these days though. More > > > > > > The openssl cryptodev engine was rewritten in > > > https://github.com/openssl/openssl/pull/3744 , but engines are going to be > > > deprecated in openssl 3.0. > > > > Is this the devcrypto engine? It appears to be broken on FreeBSD: it > > Yes, the devcrypto engine. > > > tries to invoke CIOCGSESSION on a /dev/crypto descriptor, but one is > > supposed to first use CRIOGET to get a separate descriptor with which > > sessions are associated. > > As the linked page says, "implemented based on cryptodev-linux and then > adjusted to work on FreeBSD 8.4". I don't know of anyone testing it on a > recent FreeBSD prior to your report. Well, It does work for FreeBSD 11.3 and OpenSSL 1.0.2h-freebsd 3 May 2016, so it's clear that OpenSSL broken their code... I just ran some tests and w/ cryptodev loaded, and it gives wildly fast speeds, because it only times the user cpu time, but since the work is done in the kernel, it doesn't get accounted: gold,pts,/tmp/so,507$ktrace openssl speed -evp aes-128-cbc Doing aes-128-cbc for 3s on 16 size blocks: 127989 aes-128-cbc's in 0.13s Doing aes-128-cbc for 3s on 64 size blocks: 126928 aes-128-cbc's in 0.08s Doing aes-128-cbc for 3s on 256 size blocks: 135133 aes-128-cbc's in 0.09s Doing aes-128-cbc for 3s on 1024 size blocks: 121699 aes-128-cbc's in 0.05s Doing aes-128-cbc for 3s on 8192 size blocks: 74813 aes-128-cbc's in 0.02s OpenSSL 1.0.2h-freebsd 3 May 2016 built on: date not available options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) compiler: clang The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128-cbc 15418.91k 103979.42k 402548.92k 2658555.22k 39223558.14k If you "fix" the number (multiply by .02 and divid by 3), you get the more reasonable 261MB/sec... And then when I unload cryptodev, I get: gold,pts,/tmp/so,518$openssl speed -evp aes-128-cbc Doing aes-128-cbc for 3s on 16 size blocks: 66217110 aes-128-cbc's in 2.98s Doing aes-128-cbc for 3s on 64 size blocks: 20354897 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 256 size blocks: 5899224 aes-128-cbc's in 2.97s Doing aes-128-cbc for 3s on 1024 size blocks: 1512931 aes-128-cbc's in 2.94s Doing aes-128-cbc for 3s on 8192 size blocks: 191150 aes-128-cbc's in 2.95s OpenSSL 1.0.2h-freebsd 3 May 2016 built on: date not available options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) compiler: clang The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128-cbc 355938.69k 434237.80k 508699.40k 527401.31k 531658.63k Which is reasonable for not going to the kernel everytime... (CBC encrypt is a slow mode).. So, sounds like the OpenSSL broke /dev/crypto on FreeBSD. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."