From owner-svn-src-all@freebsd.org Tue Oct 20 17:50:21 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3089F4370A8; Tue, 20 Oct 2020 17:50:21 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CG1P90pzMz4Z10; Tue, 20 Oct 2020 17:50:21 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F320918AA5; Tue, 20 Oct 2020 17:50:20 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 09KHoKru063805; Tue, 20 Oct 2020 17:50:20 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 09KHoIS9063793; Tue, 20 Oct 2020 17:50:18 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <202010201750.09KHoIS9063793@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Tue, 20 Oct 2020 17:50:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r366901 - in head: share/man/man4 sys/amd64/conf sys/conf sys/crypto/openssl sys/i386/conf sys/modules sys/modules/ossl X-SVN-Group: head X-SVN-Commit-Author: jhb X-SVN-Commit-Paths: in head: share/man/man4 sys/amd64/conf sys/conf sys/crypto/openssl sys/i386/conf sys/modules sys/modules/ossl X-SVN-Commit-Revision: 366901 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2020 17:50:21 -0000 Author: jhb Date: Tue Oct 20 17:50:18 2020 New Revision: 366901 URL: https://svnweb.freebsd.org/changeset/base/366901 Log: Add a kernel crypto driver using assembly routines from OpenSSL. Currently, this supports SHA1 and SHA2-{224,256,384,512} both as plain hashes and in HMAC mode on both amd64 and i386. It uses the SHA intrinsics when present similar to aesni(4), but uses SSE/AVX instructions when they are not. Note that some files from OpenSSL that normally wrap the assembly routines have been adapted to export methods usable by 'struct auth_xform' as is used by existing software crypto routines. Reviewed by: gallatin, jkim, delphij, gnn Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26821 Added: head/share/man/man4/ossl.4 (contents, props changed) head/sys/crypto/openssl/ossl.c (contents, props changed) head/sys/crypto/openssl/ossl.h (contents, props changed) head/sys/crypto/openssl/ossl_hash.h (contents, props changed) - copied, changed from r366898, head/crypto/openssl/include/crypto/md32_common.h head/sys/crypto/openssl/ossl_sha.h - copied, changed from r366898, head/crypto/openssl/include/openssl/sha.h head/sys/crypto/openssl/ossl_sha1.c (contents, props changed) - copied, changed from r366898, head/crypto/openssl/crypto/sha/sha_local.h head/sys/crypto/openssl/ossl_sha256.c (contents, props changed) - copied, changed from r366898, head/crypto/openssl/crypto/sha/sha256.c head/sys/crypto/openssl/ossl_sha512.c (contents, props changed) - copied, changed from r366898, head/crypto/openssl/crypto/sha/sha512.c head/sys/modules/ossl/ head/sys/modules/ossl/Makefile (contents, props changed) Modified: head/share/man/man4/Makefile head/sys/amd64/conf/NOTES head/sys/conf/files.amd64 head/sys/conf/files.i386 head/sys/conf/files.x86 head/sys/i386/conf/NOTES head/sys/modules/Makefile Modified: head/share/man/man4/Makefile ============================================================================== --- head/share/man/man4/Makefile Tue Oct 20 17:24:29 2020 (r366900) +++ head/share/man/man4/Makefile Tue Oct 20 17:50:18 2020 (r366901) @@ -398,6 +398,7 @@ MAN= aac.4 \ ocs_fc.4\ ohci.4 \ orm.4 \ + ${_ossl.4} \ ow.4 \ ow_temp.4 \ owc.4 \ @@ -819,6 +820,7 @@ _ntb_transport.4=ntb_transport.4 _nvd.4= nvd.4 _nvme.4= nvme.4 _nvram.4= nvram.4 +_ossl.4= ossl.4 _padlock.4= padlock.4 _pchtherm.4= pchtherm.4 _rr232x.4= rr232x.4 Added: head/share/man/man4/ossl.4 ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/share/man/man4/ossl.4 Tue Oct 20 17:50:18 2020 (r366901) @@ -0,0 +1,105 @@ +.\" Copyright (c) 2020 Netflix, Inc +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer, +.\" without modification. +.\" 2. Redistributions in binary form must reproduce at minimum a disclaimer +.\" similar to the "NO WARRANTY" disclaimer below ("Disclaimer") and any +.\" redistribution must be conditioned upon including a substantially +.\" similar Disclaimer requirement for further binary redistribution. +.\" +.\" NO WARRANTY +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTIBILITY +.\" AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL +.\" THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, +.\" OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF +.\" THE POSSIBILITY OF SUCH DAMAGES. +.\" +.\" $FreeBSD$ +.\" +.Dd October 19, 2020 +.Dt OSSL 4 +.Os +.Sh NAME +.Nm ossl +.Nd "driver using OpenSSL assembly routines on x86 CPUs" +.Sh SYNOPSIS +To compile this driver into the kernel, +place the following lines in your +kernel configuration file: +.Bd -ragged -offset indent +.Cd "device crypto" +.Cd "device cryptodev" +.Cd "device ossl" +.Ed +.Pp +Alternatively, to load the driver as a +module at boot time, place the following line in +.Xr loader.conf 5 : +.Bd -literal -offset indent +ossl_load="YES" +.Ed +.Sh DESCRIPTION +The OpenSSL distribution includes architecture-specific +implementations for some commonly used cryptographic algorithms. +This driver adds a wrapper around these routines permitting them to be +used by in-kernel cryptography consumers such as kernel TLS and IPsec. +.Pp +The +.Nm +driver includes architecture-specific implementations for the following +architectures: +.Pp +.Bl -bullet -compact +.It +amd64 +.It +i386 +.El +.Pp +The +.Nm +driver includes support for the following algorithms: +.Pp +.Bl -bullet -compact +.It +SHA1 +.It +SHA1-HMAC +.It +SHA2-224 +.It +SHA2-224-HMAC +.It +SHA2-256 +.It +SHA2-256-HMAC +.It +SHA2-384 +.It +SHA2-384-HMAC +.It +SHA2-512 +.It +SHA2-512-HMAC +.El +.Sh SEE ALSO +.Xr crypto 4 , +.Xr intro 4 , +.Xr ipsec 4 , +.Xr crypto 7 , +.Xr crypto 9 +.Sh HISTORY +The +.Nm +driver first appeared in +.Fx 13.0 . Modified: head/sys/amd64/conf/NOTES ============================================================================== --- head/sys/amd64/conf/NOTES Tue Oct 20 17:24:29 2020 (r366900) +++ head/sys/amd64/conf/NOTES Tue Oct 20 17:50:18 2020 (r366901) @@ -533,6 +533,7 @@ device tpm device padlock_rng # VIA Padlock RNG device rdrand_rng # Intel Bull Mountain RNG device aesni # AES-NI OpenCrypto module +device ossl # OpenSSL OpenCrypto module device ioat # Intel I/OAT DMA engine # Modified: head/sys/conf/files.amd64 ============================================================================== --- head/sys/conf/files.amd64 Tue Oct 20 17:24:29 2020 (r366900) +++ head/sys/conf/files.amd64 Tue Oct 20 17:50:18 2020 (r366901) @@ -137,6 +137,9 @@ cddl/dev/dtrace/amd64/dtrace_asm.S optional dtrace c cddl/dev/dtrace/amd64/dtrace_subr.c optional dtrace compile-with "${DTRACE_C}" crypto/aesni/aeskeys_amd64.S optional aesni crypto/des/des_enc.c optional netsmb +crypto/openssl/amd64/sha1-x86_64.S optional ossl +crypto/openssl/amd64/sha256-x86_64.S optional ossl +crypto/openssl/amd64/sha512-x86_64.S optional ossl dev/acpi_support/acpi_wmi_if.m standard dev/agp/agp_amd64.c optional agp dev/agp/agp_i810.c optional agp Modified: head/sys/conf/files.i386 ============================================================================== --- head/sys/conf/files.i386 Tue Oct 20 17:24:29 2020 (r366900) +++ head/sys/conf/files.i386 Tue Oct 20 17:50:18 2020 (r366901) @@ -77,6 +77,9 @@ compat/linux/linux.c optional compat_linux compat/ndis/winx32_wrap.S optional ndisapi pci crypto/aesni/aeskeys_i386.S optional aesni crypto/des/arch/i386/des_enc.S optional netsmb +crypto/openssl/i386/sha1-586.S optional ossl +crypto/openssl/i386/sha256-586.S optional ossl +crypto/openssl/i386/sha512-586.S optional ossl dev/agp/agp_ali.c optional agp dev/agp/agp_amd.c optional agp dev/agp/agp_amd64.c optional agp Modified: head/sys/conf/files.x86 ============================================================================== --- head/sys/conf/files.x86 Tue Oct 20 17:24:29 2020 (r366900) +++ head/sys/conf/files.x86 Tue Oct 20 17:50:18 2020 (r366901) @@ -53,6 +53,10 @@ intel_sha256.o optional aesni \ compile-with "${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} -mmmx -msse -msse4 -msha ${.IMPSRC}" \ no-implicit-rule \ clean "intel_sha256.o" +crypto/openssl/ossl.c optional ossl +crypto/openssl/ossl_sha1.c optional ossl +crypto/openssl/ossl_sha256.c optional ossl +crypto/openssl/ossl_sha512.c optional ossl crypto/via/padlock.c optional padlock crypto/via/padlock_cipher.c optional padlock crypto/via/padlock_hash.c optional padlock Added: head/sys/crypto/openssl/ossl.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/crypto/openssl/ossl.c Tue Oct 20 17:50:18 2020 (r366901) @@ -0,0 +1,369 @@ +/* + * Copyright (c) 2020 Netflix, Inc + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer, + * without modification. + * 2. Redistributions in binary form must reproduce at minimum a disclaimer + * similar to the "NO WARRANTY" disclaimer below ("Disclaimer") and any + * redistribution must be conditioned upon including a substantially + * similar Disclaimer requirement for further binary redistribution. + * + * NO WARRANTY + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTIBILITY + * AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, + * OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER + * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF + * THE POSSIBILITY OF SUCH DAMAGES. + */ + +/* + * A driver for the OpenCrypto framework which uses assembly routines + * from OpenSSL. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include + +#include "cryptodev_if.h" + +struct ossl_softc { + int32_t sc_cid; +}; + +struct ossl_session_hash { + struct ossl_hash_context ictx; + struct ossl_hash_context octx; + struct auth_hash *axf; + u_int mlen; +}; + +struct ossl_session { + struct ossl_session_hash hash; +}; + +/* + * See OPENSSL_ia32cap(3). + * + * [0] = cpu_feature but with a few custom bits + * [1] = cpu_feature2 but with AMD XOP in bit 11 + * [2] = cpu_stdext_feature + * [3] = 0 + */ +unsigned int OPENSSL_ia32cap_P[4]; + +static MALLOC_DEFINE(M_OSSL, "ossl", "OpenSSL crypto"); + +static void +ossl_cpuid(void) +{ + uint64_t xcr0; + u_int regs[4]; + u_int max_cores; + + /* Derived from OpenSSL_ia32_cpuid. */ + + OPENSSL_ia32cap_P[0] = cpu_feature & ~(CPUID_B20 | CPUID_IA64); + if (cpu_vendor_id == CPU_VENDOR_INTEL) { + OPENSSL_ia32cap_P[0] |= CPUID_IA64; + if ((cpu_id & 0xf00) != 0xf00) + OPENSSL_ia32cap_P[0] |= CPUID_B20; + } + + /* Only leave CPUID_HTT on if HTT is present. */ + if (cpu_vendor_id == CPU_VENDOR_AMD && cpu_exthigh >= 0x80000008) { + max_cores = (cpu_procinfo2 & AMDID_CMP_CORES) + 1; + if (cpu_feature & CPUID_HTT) { + if ((cpu_procinfo & CPUID_HTT_CORES) >> 16 <= max_cores) + OPENSSL_ia32cap_P[0] &= ~CPUID_HTT; + } + } else { + if (cpu_high >= 4) { + cpuid_count(4, 0, regs); + max_cores = (regs[0] >> 26) & 0xfff; + } else + max_cores = -1; + } + if (max_cores == 0) + OPENSSL_ia32cap_P[0] &= ~CPUID_HTT; + else if ((cpu_procinfo & CPUID_HTT_CORES) >> 16 == 0) + OPENSSL_ia32cap_P[0] &= ~CPUID_HTT; + + OPENSSL_ia32cap_P[1] = cpu_feature2 & ~AMDID2_XOP; + if (cpu_vendor_id == CPU_VENDOR_AMD) + OPENSSL_ia32cap_P[1] |= amd_feature2 & AMDID2_XOP; + + OPENSSL_ia32cap_P[2] = cpu_stdext_feature; + if ((OPENSSL_ia32cap_P[1] & CPUID2_XSAVE) == 0) + OPENSSL_ia32cap_P[2] &= ~(CPUID_STDEXT_AVX512F | + CPUID_STDEXT_AVX512DQ); + + /* Disable AVX512F on Skylake-X. */ + if ((cpu_id & 0x0fff0ff0) == 0x00050650) + OPENSSL_ia32cap_P[2] &= ~(CPUID_STDEXT_AVX512F); + + if (cpu_feature2 & CPUID2_OSXSAVE) + xcr0 = rxcr(0); + else + xcr0 = 0; + + if ((xcr0 & (XFEATURE_AVX512 | XFEATURE_AVX)) != + (XFEATURE_AVX512 | XFEATURE_AVX)) + OPENSSL_ia32cap_P[2] &= ~(CPUID_STDEXT_AVX512VL | + CPUID_STDEXT_AVX512BW | CPUID_STDEXT_AVX512IFMA | + CPUID_STDEXT_AVX512F); + if ((xcr0 & XFEATURE_AVX) != XFEATURE_AVX) { + OPENSSL_ia32cap_P[1] &= ~(CPUID2_AVX | AMDID2_XOP | CPUID2_FMA); + OPENSSL_ia32cap_P[2] &= ~CPUID_STDEXT_AVX2; + } +} + +static void +ossl_identify(driver_t *driver, device_t parent) +{ + + if (device_find_child(parent, "ossl", -1) == NULL) + BUS_ADD_CHILD(parent, 10, "ossl", -1); +} + +static int +ossl_probe(device_t dev) +{ + + device_set_desc(dev, "OpenSSL crypto"); + return (BUS_PROBE_DEFAULT); +} + +static int +ossl_attach(device_t dev) +{ + struct ossl_softc *sc; + + sc = device_get_softc(dev); + + ossl_cpuid(); + sc->sc_cid = crypto_get_driverid(dev, sizeof(struct ossl_session), + CRYPTOCAP_F_SOFTWARE | CRYPTOCAP_F_SYNC | + CRYPTOCAP_F_ACCEL_SOFTWARE); + if (sc->sc_cid < 0) { + device_printf(dev, "failed to allocate crypto driver id\n"); + return (ENXIO); + } + + return (0); +} + +static int +ossl_detach(device_t dev) +{ + struct ossl_softc *sc; + + sc = device_get_softc(dev); + + crypto_unregister_all(sc->sc_cid); + + return (0); +} + +static struct auth_hash * +ossl_lookup_hash(const struct crypto_session_params *csp) +{ + + switch (csp->csp_auth_alg) { + case CRYPTO_SHA1: + case CRYPTO_SHA1_HMAC: + return (&ossl_hash_sha1); + case CRYPTO_SHA2_224: + case CRYPTO_SHA2_224_HMAC: + return (&ossl_hash_sha224); + case CRYPTO_SHA2_256: + case CRYPTO_SHA2_256_HMAC: + return (&ossl_hash_sha256); + case CRYPTO_SHA2_384: + case CRYPTO_SHA2_384_HMAC: + return (&ossl_hash_sha384); + case CRYPTO_SHA2_512: + case CRYPTO_SHA2_512_HMAC: + return (&ossl_hash_sha512); + default: + return (NULL); + } +} + +static int +ossl_probesession(device_t dev, const struct crypto_session_params *csp) +{ + + if ((csp->csp_flags & ~(CSP_F_SEPARATE_OUTPUT | CSP_F_SEPARATE_AAD)) != + 0) + return (EINVAL); + switch (csp->csp_mode) { + case CSP_MODE_DIGEST: + if (ossl_lookup_hash(csp) == NULL) + return (EINVAL); + break; + default: + return (EINVAL); + } + + return (CRYPTODEV_PROBE_ACCEL_SOFTWARE); +} + +static void +ossl_setkey_hmac(struct ossl_session *s, const void *key, int klen) +{ + + hmac_init_ipad(s->hash.axf, key, klen, &s->hash.ictx); + hmac_init_opad(s->hash.axf, key, klen, &s->hash.octx); +} + +static int +ossl_newsession(device_t dev, crypto_session_t cses, + const struct crypto_session_params *csp) +{ + struct ossl_session *s; + struct auth_hash *axf; + + s = crypto_get_driver_session(cses); + + axf = ossl_lookup_hash(csp); + s->hash.axf = axf; + if (csp->csp_auth_mlen == 0) + s->hash.mlen = axf->hashsize; + else + s->hash.mlen = csp->csp_auth_mlen; + + if (csp->csp_auth_klen == 0) { + axf->Init(&s->hash.ictx); + } else { + if (csp->csp_auth_key != NULL) { + fpu_kern_enter(curthread, NULL, FPU_KERN_NOCTX); + ossl_setkey_hmac(s, csp->csp_auth_key, + csp->csp_auth_klen); + fpu_kern_leave(curthread, NULL); + } + } + return (0); +} + +static int +ossl_process(device_t dev, struct cryptop *crp, int hint) +{ + struct ossl_hash_context ctx; + char digest[HASH_MAX_LEN]; + const struct crypto_session_params *csp; + struct ossl_session *s; + struct auth_hash *axf; + int error; + bool fpu_entered; + + s = crypto_get_driver_session(crp->crp_session); + csp = crypto_get_params(crp->crp_session); + axf = s->hash.axf; + + if (is_fpu_kern_thread(0)) { + fpu_entered = false; + } else { + fpu_kern_enter(curthread, NULL, FPU_KERN_NOCTX); + fpu_entered = true; + } + + if (crp->crp_auth_key != NULL) + ossl_setkey_hmac(s, crp->crp_auth_key, csp->csp_auth_klen); + + ctx = s->hash.ictx; + + if (crp->crp_aad != NULL) + error = axf->Update(&ctx, crp->crp_aad, crp->crp_aad_length); + else + error = crypto_apply(crp, crp->crp_aad_start, + crp->crp_aad_length, axf->Update, &ctx); + if (error) + goto out; + + error = crypto_apply(crp, crp->crp_payload_start, + crp->crp_payload_length, axf->Update, &ctx); + if (error) + goto out; + + axf->Final(digest, &ctx); + + if (csp->csp_auth_klen != 0) { + ctx = s->hash.octx; + axf->Update(&ctx, digest, axf->hashsize); + axf->Final(digest, &ctx); + } + + if (crp->crp_op & CRYPTO_OP_VERIFY_DIGEST) { + char digest2[HASH_MAX_LEN]; + + crypto_copydata(crp, crp->crp_digest_start, s->hash.mlen, + digest2); + if (timingsafe_bcmp(digest, digest2, s->hash.mlen) != 0) + error = EBADMSG; + explicit_bzero(digest2, sizeof(digest2)); + } else { + crypto_copyback(crp, crp->crp_digest_start, s->hash.mlen, + digest); + } + explicit_bzero(digest, sizeof(digest)); + +out: + if (fpu_entered) + fpu_kern_leave(curthread, NULL); + + crp->crp_etype = error; + crypto_done(crp); + + explicit_bzero(&ctx, sizeof(ctx)); + return (0); +} + +static device_method_t ossl_methods[] = { + DEVMETHOD(device_identify, ossl_identify), + DEVMETHOD(device_probe, ossl_probe), + DEVMETHOD(device_attach, ossl_attach), + DEVMETHOD(device_detach, ossl_detach), + + DEVMETHOD(cryptodev_probesession, ossl_probesession), + DEVMETHOD(cryptodev_newsession, ossl_newsession), + DEVMETHOD(cryptodev_process, ossl_process), + + DEVMETHOD_END +}; + +static driver_t ossl_driver = { + "ossl", + ossl_methods, + sizeof(struct ossl_softc) +}; + +static devclass_t ossl_devclass; + +DRIVER_MODULE(ossl, nexus, ossl_driver, ossl_devclass, NULL, NULL); +MODULE_VERSION(ossl, 1); +MODULE_DEPEND(ossl, crypto, 1, 1, 1); Added: head/sys/crypto/openssl/ossl.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/crypto/openssl/ossl.h Tue Oct 20 17:50:18 2020 (r366901) @@ -0,0 +1,51 @@ +/* + * Copyright (c) 2020 Netflix, Inc + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer, + * without modification. + * 2. Redistributions in binary form must reproduce at minimum a disclaimer + * similar to the "NO WARRANTY" disclaimer below ("Disclaimer") and any + * redistribution must be conditioned upon including a substantially + * similar Disclaimer requirement for further binary redistribution. + * + * NO WARRANTY + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTIBILITY + * AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, + * OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER + * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF + * THE POSSIBILITY OF SUCH DAMAGES. + * + * $FreeBSD$ + */ + +#ifndef __OSSL_H__ +#define __OSSL_H__ + +/* Compatibility shims. */ +#define OPENSSL_cleanse explicit_bzero + +/* Used by assembly routines to select CPU-specific variants. */ +extern unsigned int OPENSSL_ia32cap_P[4]; + +/* Needs to be big enough to hold any hash context. */ +struct ossl_hash_context { + uint32_t dummy[54]; +} __aligned(32); + +extern struct auth_hash ossl_hash_sha1; +extern struct auth_hash ossl_hash_sha224; +extern struct auth_hash ossl_hash_sha256; +extern struct auth_hash ossl_hash_sha384; +extern struct auth_hash ossl_hash_sha512; + +#endif /* !__OSSL_H__ */ Copied and modified: head/sys/crypto/openssl/ossl_hash.h (from r366898, head/crypto/openssl/include/crypto/md32_common.h) ============================================================================== --- head/crypto/openssl/include/crypto/md32_common.h Tue Oct 20 17:00:43 2020 (r366898, copy source) +++ head/sys/crypto/openssl/ossl_hash.h Tue Oct 20 17:50:18 2020 (r366901) @@ -5,96 +5,17 @@ * this file except in compliance with the License. You can obtain a copy * in the file LICENSE in the source distribution or at * https://www.openssl.org/source/license.html + * + * $FreeBSD$ */ -/*- - * This is a generic 32 bit "collector" for message digest algorithms. - * Whenever needed it collects input character stream into chunks of - * 32 bit values and invokes a block function that performs actual hash - * calculations. +/* + * Derived from include/crypto/md32_common.h * - * Porting guide. - * - * Obligatory macros: - * - * DATA_ORDER_IS_BIG_ENDIAN or DATA_ORDER_IS_LITTLE_ENDIAN - * this macro defines byte order of input stream. - * HASH_CBLOCK - * size of a unit chunk HASH_BLOCK operates on. - * HASH_LONG - * has to be at least 32 bit wide. - * HASH_CTX - * context structure that at least contains following - * members: - * typedef struct { - * ... - * HASH_LONG Nl,Nh; - * either { - * HASH_LONG data[HASH_LBLOCK]; - * unsigned char data[HASH_CBLOCK]; - * }; - * unsigned int num; - * ... - * } HASH_CTX; - * data[] vector is expected to be zeroed upon first call to - * HASH_UPDATE. - * HASH_UPDATE - * name of "Update" function, implemented here. - * HASH_TRANSFORM - * name of "Transform" function, implemented here. - * HASH_FINAL - * name of "Final" function, implemented here. - * HASH_BLOCK_DATA_ORDER - * name of "block" function capable of treating *unaligned* input - * message in original (data) byte order, implemented externally. - * HASH_MAKE_STRING - * macro converting context variables to an ASCII hash string. - * - * MD5 example: - * - * #define DATA_ORDER_IS_LITTLE_ENDIAN - * - * #define HASH_LONG MD5_LONG - * #define HASH_CTX MD5_CTX - * #define HASH_CBLOCK MD5_CBLOCK - * #define HASH_UPDATE MD5_Update - * #define HASH_TRANSFORM MD5_Transform - * #define HASH_FINAL MD5_Final - * #define HASH_BLOCK_DATA_ORDER md5_block_data_order + * HASH_UPDATE and HASH_FINAL have been updated to work with the + * auth_hash interface. */ -#include - -#if !defined(DATA_ORDER_IS_BIG_ENDIAN) && !defined(DATA_ORDER_IS_LITTLE_ENDIAN) -# error "DATA_ORDER must be defined!" -#endif - -#ifndef HASH_CBLOCK -# error "HASH_CBLOCK must be defined!" -#endif -#ifndef HASH_LONG -# error "HASH_LONG must be defined!" -#endif -#ifndef HASH_CTX -# error "HASH_CTX must be defined!" -#endif - -#ifndef HASH_UPDATE -# error "HASH_UPDATE must be defined!" -#endif -#ifndef HASH_TRANSFORM -# error "HASH_TRANSFORM must be defined!" -#endif -#ifndef HASH_FINAL -# error "HASH_FINAL must be defined!" -#endif - -#ifndef HASH_BLOCK_DATA_ORDER -# error "HASH_BLOCK_DATA_ORDER must be defined!" -#endif - -#define ROTATE(a,n) (((a)<<(n))|(((a)&0xffffffff)>>(32-(n)))) - #if defined(DATA_ORDER_IS_BIG_ENDIAN) # define HOST_c2l(c,l) (l =(((unsigned long)(*((c)++)))<<24), \ @@ -125,15 +46,17 @@ * Time for some action :-) */ -int HASH_UPDATE(HASH_CTX *c, const void *data_, size_t len) +static int +HASH_UPDATE(void *c_, const void *data_, unsigned int len) { + HASH_CTX *c = c_; const unsigned char *data = data_; unsigned char *p; HASH_LONG l; size_t n; if (len == 0) - return 1; + return 0; l = (c->Nl + (((HASH_LONG) len) << 3)) & 0xffffffffUL; if (l < c->Nl) /* overflow */ @@ -163,7 +86,7 @@ int HASH_UPDATE(HASH_CTX *c, const void *data_, size_t } else { memcpy(p + n, data, len); c->num += (unsigned int)len; - return 1; + return 0; } } @@ -180,16 +103,13 @@ int HASH_UPDATE(HASH_CTX *c, const void *data_, size_t c->num = (unsigned int)len; memcpy(p, data, len); } - return 1; + return 0; } -void HASH_TRANSFORM(HASH_CTX *c, const unsigned char *data) +static void +HASH_FINAL(uint8_t *md, void *c_) { - HASH_BLOCK_DATA_ORDER(c, data, 1); -} - -int HASH_FINAL(unsigned char *md, HASH_CTX *c) -{ + HASH_CTX *c = c_; unsigned char *p = (unsigned char *)c->data; size_t n = c->num; @@ -222,35 +142,5 @@ int HASH_FINAL(unsigned char *md, HASH_CTX *c) HASH_MAKE_STRING(c, md); #endif - return 1; + return; } - -#ifndef MD32_REG_T -# if defined(__alpha) || defined(__sparcv9) || defined(__mips) -# define MD32_REG_T long -/* - * This comment was originally written for MD5, which is why it - * discusses A-D. But it basically applies to all 32-bit digests, - * which is why it was moved to common header file. - * - * In case you wonder why A-D are declared as long and not - * as MD5_LONG. Doing so results in slight performance - * boost on LP64 architectures. The catch is we don't - * really care if 32 MSBs of a 64-bit register get polluted - * with eventual overflows as we *save* only 32 LSBs in - * *either* case. Now declaring 'em long excuses the compiler - * from keeping 32 MSBs zeroed resulting in 13% performance - * improvement under SPARC Solaris7/64 and 5% under AlphaLinux. - * Well, to be honest it should say that this *prevents* - * performance degradation. - */ -# else -/* - * Above is not absolute and there are LP64 compilers that - * generate better code if MD32_REG_T is defined int. The above - * pre-processor condition reflects the circumstances under which - * the conclusion was made and is subject to further extension. - */ -# define MD32_REG_T int -# endif -#endif Copied and modified: head/sys/crypto/openssl/ossl_sha.h (from r366898, head/crypto/openssl/include/openssl/sha.h) ============================================================================== --- head/crypto/openssl/include/openssl/sha.h Tue Oct 20 17:00:43 2020 (r366898, copy source) +++ head/sys/crypto/openssl/ossl_sha.h Tue Oct 20 17:50:18 2020 (r366901) @@ -5,31 +5,27 @@ * this file except in compliance with the License. You can obtain a copy * in the file LICENSE in the source distribution or at * https://www.openssl.org/source/license.html + * + * $FreeBSD$ */ -#ifndef HEADER_SHA_H -# define HEADER_SHA_H +#ifndef __OSSL_SHA_H__ +#define __OSSL_SHA_H__ -# include -# include - -#ifdef __cplusplus -extern "C" { -#endif - -/*- - * !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - * ! SHA_LONG has to be at least 32 bits wide. ! - * !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +/* + * This is always included last which permits the namespace hacks below + * to work. */ +#define SHA256_CTX OSSL_SHA256_CTX +#define SHA512_CTX OSSL_SHA512_CTX + +/* From include/openssl/sha.h */ # define SHA_LONG unsigned int # define SHA_LBLOCK 16 # define SHA_CBLOCK (SHA_LBLOCK*4)/* SHA treats input data as a * contiguous array of 32 bit wide * big-endian values. */ -# define SHA_LAST_BLOCK (SHA_CBLOCK-8) -# define SHA_DIGEST_LENGTH 20 typedef struct SHAstate_st { SHA_LONG h0, h1, h2, h3, h4; @@ -38,12 +34,6 @@ typedef struct SHAstate_st { unsigned int num; } SHA_CTX; -int SHA1_Init(SHA_CTX *c); -int SHA1_Update(SHA_CTX *c, const void *data, size_t len); -int SHA1_Final(unsigned char *md, SHA_CTX *c); -unsigned char *SHA1(const unsigned char *d, size_t n, unsigned char *md); -void SHA1_Transform(SHA_CTX *c, const unsigned char *data); - # define SHA256_CBLOCK (SHA_LBLOCK*4)/* SHA-256 treats input data as a * contiguous array of 32 bit wide * big-endian values. */ @@ -55,42 +45,15 @@ typedef struct SHA256state_st { unsigned int num, md_len; } SHA256_CTX; -int SHA224_Init(SHA256_CTX *c); -int SHA224_Update(SHA256_CTX *c, const void *data, size_t len); -int SHA224_Final(unsigned char *md, SHA256_CTX *c); -unsigned char *SHA224(const unsigned char *d, size_t n, unsigned char *md); -int SHA256_Init(SHA256_CTX *c); -int SHA256_Update(SHA256_CTX *c, const void *data, size_t len); -int SHA256_Final(unsigned char *md, SHA256_CTX *c); -unsigned char *SHA256(const unsigned char *d, size_t n, unsigned char *md); -void SHA256_Transform(SHA256_CTX *c, const unsigned char *data); - -# define SHA224_DIGEST_LENGTH 28 -# define SHA256_DIGEST_LENGTH 32 -# define SHA384_DIGEST_LENGTH 48 -# define SHA512_DIGEST_LENGTH 64 - /* - * Unlike 32-bit digest algorithms, SHA-512 *relies* on SHA_LONG64 - * being exactly 64-bit wide. See Implementation Notes in sha512.c - * for further details. - */ -/* * SHA-512 treats input data as a * contiguous array of 64 bit * wide big-endian values. */ # define SHA512_CBLOCK (SHA_LBLOCK*8) -# if (defined(_WIN32) || defined(_WIN64)) && !defined(__MINGW32__) -# define SHA_LONG64 unsigned __int64 -# define U64(C) C##UI64 -# elif defined(__arch64__) -# define SHA_LONG64 unsigned long -# define U64(C) C##UL -# else + # define SHA_LONG64 unsigned long long # define U64(C) C##ULL -# endif typedef struct SHA512state_st { SHA_LONG64 h[8]; @@ -102,18 +65,4 @@ typedef struct SHA512state_st { unsigned int num, md_len; } SHA512_CTX; -int SHA384_Init(SHA512_CTX *c); -int SHA384_Update(SHA512_CTX *c, const void *data, size_t len); -int SHA384_Final(unsigned char *md, SHA512_CTX *c); -unsigned char *SHA384(const unsigned char *d, size_t n, unsigned char *md); -int SHA512_Init(SHA512_CTX *c); -int SHA512_Update(SHA512_CTX *c, const void *data, size_t len); -int SHA512_Final(unsigned char *md, SHA512_CTX *c); -unsigned char *SHA512(const unsigned char *d, size_t n, unsigned char *md); -void SHA512_Transform(SHA512_CTX *c, const unsigned char *data); - -#ifdef __cplusplus -} -#endif - -#endif +#endif /* !__OSSL_SHA_H__ */ Copied and modified: head/sys/crypto/openssl/ossl_sha1.c (from r366898, head/crypto/openssl/crypto/sha/sha_local.h) ============================================================================== --- head/crypto/openssl/crypto/sha/sha_local.h Tue Oct 20 17:00:43 2020 (r366898, copy source) +++ head/sys/crypto/openssl/ossl_sha1.c Tue Oct 20 17:50:18 2020 (r366901) @@ -7,12 +7,22 @@ * https://www.openssl.org/source/license.html */ -#include -#include +#include +__FBSDID("$FreeBSD$"); -#include -#include +#include +#include +#include +#include + +#include +#include + +/* sha1-x86_64.S */ +void sha1_block_data_order(SHA_CTX *c, const void *p, size_t len); + +/* From crypto/sha/sha_local.h */ #define DATA_ORDER_IS_BIG_ENDIAN #define HASH_LONG SHA_LONG @@ -27,398 +37,41 @@ ll=(c)->h4; (void)HOST_l2c(ll,(s)); \ } while (0) -#define HASH_UPDATE SHA1_Update -#define HASH_TRANSFORM SHA1_Transform -#define HASH_FINAL SHA1_Final -#define HASH_INIT SHA1_Init +#define HASH_UPDATE ossl_sha1_update +#define HASH_FINAL ossl_sha1_final +#define HASH_INIT ossl_sha1_init #define HASH_BLOCK_DATA_ORDER sha1_block_data_order -#define Xupdate(a,ix,ia,ib,ic,id) ( (a)=(ia^ib^ic^id), \ - ix=(a)=ROTATE((a),1) \ - ) -#ifndef SHA1_ASM -static void sha1_block_data_order(SHA_CTX *c, const void *p, size_t num); -#else -void sha1_block_data_order(SHA_CTX *c, const void *p, size_t num); -#endif - -#include "crypto/md32_common.h" - #define INIT_DATA_h0 0x67452301UL #define INIT_DATA_h1 0xefcdab89UL #define INIT_DATA_h2 0x98badcfeUL #define INIT_DATA_h3 0x10325476UL #define INIT_DATA_h4 0xc3d2e1f0UL -int HASH_INIT(SHA_CTX *c) +static void +HASH_INIT(void *c_) { + SHA_CTX *c = c_; memset(c, 0, sizeof(*c)); c->h0 = INIT_DATA_h0; c->h1 = INIT_DATA_h1; c->h2 = INIT_DATA_h2; c->h3 = INIT_DATA_h3; c->h4 = INIT_DATA_h4; - return 1; } -#define K_00_19 0x5a827999UL -#define K_20_39 0x6ed9eba1UL -#define K_40_59 0x8f1bbcdcUL -#define K_60_79 0xca62c1d6UL +#include "ossl_hash.h" -/* - * As pointed out by Wei Dai, F() below can be simplified to the code in - * F_00_19. Wei attributes these optimizations to Peter Gutmann's SHS code, - * and he attributes it to Rich Schroeppel. - * #define F(x,y,z) (((x) & (y)) | ((~(x)) & (z))) - * I've just become aware of another tweak to be made, again from Wei Dai, - * in F_40_59, (x&a)|(y&a) -> (x|y)&a - */ *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***